The Computation Limits of Deep Learning

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This project catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.

Paper
Neil Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel Manso
Wired, VentureBeat, Discover, The Next Web, Interesting Engineering, Tech Gig

Benchmarks

Image classification
ImageNet

Object Detection
MS COCO

Question Answering
SQuAD 1.1

Named Entity Recognition
Conll 2003

Machine Translation on WMT2014 English-French

vs.


Transformer+BT (ADMIN init)	2020	46.40	-
Noisy back-translation	2018	45.60	180 EFLOPs
mRASP+Fine-Tune	2020	44.30	-
Transformer + R-Drop	2021	43.95	8.54 EFLOPs
Admin	2020	43.80	48.8 EFLOPs
BERT-fused NMT	2020	43.78	114 EFLOPs
MUSE(Paralllel Multi-scale Attention)	2019	43.50	-
T5	2019	43.40	-
Local Joint Self-attention	2019	43.30	-
Depth Growing	2019	43.27	28.4 EFLOPs

Want to contribute?

You have access to our database where you can point out any errors or suggest changes

Go to database