This website is based on a rumoured list of papers that Ilya Sutskever gave to John Carmack. We currently only have a list of 27. If you or anyone you know has the full, canonical list please feel free to reach out (https://michaelcrowley.dev/). https://30papers.com/papers/cs231n CS231n: Convolutional Neural Networks for Visual Recognition The course notes that teach convolutional networks from first principles, from linear classifiers up to deep architectures for images. Andrej Karpathy Fei-Fei Li Plus many more https://30papers.com/papers/unreasonable-effectiveness-of-rnns The Unreasonable Effectiveness of Recurrent Neural Networks A hands on blog post that trains character level RNNs to generate text and shows, with vivid examples, how much structure they capture. Andrej Karpathy https://30papers.com/papers/understanding-lstms Understanding LSTM Networks The clearest visual explanation of how LSTM gates carry information across long sequences, widely used as a first introduction. Christopher Olah https://30papers.com/papers/alexnet ImageNet Classification with Deep Convolutional Neural Networks AlexNet. The convolutional network that won ImageNet by a wide margin and set off the modern deep learning era. Alex Krizhevsky Ilya Sutskever https://30papers.com/papers/deep-residual-learning Deep Residual Learning for Image Recognition ResNet. Introduces residual connections that let networks grow to hundreds of layers by learning changes to the input rather than full transformations. Kaiming He Xiangyu Zhang https://30papers.com/papers/dilated-convolutions Multi-Scale Context Aggregation by Dilated Convolutions Shows how dilated convolutions expand the receptive field without losing resolution, which sharpened dense prediction tasks like segmentation. FY Fisher Yu Vladlen Koltun https://30papers.com/papers/identity-mappings-resnets Identity Mappings in Deep Residual Networks A follow up to ResNet that studies why identity shortcuts work so well and proposes a cleaner pre-activation residual block. Kaiming He Xiangyu Zhang https://30papers.com/papers/rnn-regularization Recurrent Neural Network Regularization Shows how to apply dropout to LSTMs correctly, on the non-recurrent connections, so large recurrent models stop overfitting. Wojciech Zaremba Ilya Sutskever https://30papers.com/papers/deep-speech-2 Deep Speech 2: End-to-End Speech Recognition in English and Mandarin An end to end speech recognition system trained with connectionist temporal classification that worked across two very different languages. Dario Amodei RA Rishita Anubhai Plus many more https://30papers.com/papers/order-matters Order Matters: Sequence to Sequence for Sets Examines how the order of inputs and outputs affects sequence to sequence models, and how to handle data that is really a set. Oriol Vinyals Samy Bengio https://30papers.com/papers/neural-machine-translation Neural Machine Translation by Jointly Learning to Align and Translate Introduces the attention mechanism, letting a translation model look back at the relevant source words instead of a single fixed summary. Dzmitry Bahdanau Kyunghyun Cho https://30papers.com/papers/pointer-networks Pointer Networks A sequence model whose outputs point back at positions in the input, which suits problems whose answer is a selection or ordering of the inputs. Oriol Vinyals Meire Fortunato https://30papers.com/papers/attention-is-all-you-need Attention Is All You Need The Transformer. Replaces recurrence entirely with self attention, the architecture that underpins almost every modern large language model. Ashish Vaswani Noam Shazeer Plus many more https://30papers.com/papers/annotated-transformer The Annotated Transformer A line by line, runnable reimplementation of the Transformer that turns the original paper into working, readable code. Sasha Rush Plus many more https://30papers.com/papers/neural-turing-machines Neural Turing Machines Couples a neural network to an external memory it can read and write with differentiable attention, learning simple algorithms from examples. Alex Graves GW Greg Wayne https://30papers.com/papers/relational-reasoning A Simple Neural Network Module for Relational Reasoning Introduces the relation network, a small plug in module that lets a network reason about how pairs of objects relate to each other. Adam Santoro DR David Raposo Plus many more https://30papers.com/papers/relational-recurrent-networks Relational Recurrent Neural Networks Adds a self attention based memory to recurrent networks so that stored memories can interact, improving tasks that need relational reasoning over time. Adam Santoro RF Ryan Faulkner Plus many more https://30papers.com/papers/neural-message-passing Neural Message Passing for Quantum Chemistry Unifies many graph neural networks under a message passing framework and applies it to predicting molecular properties. Justin Gilmer SS Samuel S. Schoenholz https://30papers.com/papers/scaling-laws Scaling Laws for Neural Language Models Measures how language model loss falls as a smooth power law in model size, data, and compute, the empirical basis for building ever larger models. Jared Kaplan SM Sam McCandlish Plus many more https://30papers.com/papers/gpipe GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism A pipeline parallelism library that splits a giant model across devices and keeps them busy, making it practical to train very large networks. YH Yanping Huang YC Youlong Cheng Plus many more https://30papers.com/papers/keeping-neural-networks-simple Keeping Neural Networks Simple by Minimizing the Description Length of the Weights An early information-theoretic argument that good networks are ones whose weights can be described with few bits, linking generalization to compression. Geoffrey Hinton DV Drew van Camp https://30papers.com/papers/mdl-principle-tutorial A Tutorial Introduction to the Minimum Description Length Principle A readable introduction to choosing models by how well they compress the data, treating learning as finding the shortest description. Peter Grunwald https://30papers.com/papers/first-law-of-complexodynamics The First Law of Complexodynamics A blog essay asking for a formal law that explains why the complexity of a closed system rises, peaks, and falls, rather than simply tracking entropy. Scott Aaronson https://30papers.com/papers/coffee-automaton Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton Uses a simple cellular automaton model of coffee mixing with cream to ask why complexity rises and then falls as a system moves toward equilibrium. Scott Aaronson Sean M. Carroll https://30papers.com/papers/kolmogorov-complexity Kolmogorov Complexity The textbook treatment of the shortest program that produces a string, the formal backbone behind description length and algorithmic randomness. Thomas M. Cover JA Joy A. Thomas https://30papers.com/papers/variational-lossy-autoencoder Variational Lossy Autoencoder Combines variational autoencoders with autoregressive decoders, and shows how to control which information the latent code is forced to keep. XC Xi Chen Diederik P. Kingma Plus 6 more https://30papers.com/papers/machine-super-intelligence Machine Super Intelligence A doctoral thesis that proposes a formal, universal measure of machine intelligence and explores its consequences for very capable agents. Shane Legg