Why Stacking Layers Works
One neuron is almost useless. A stack of them recognises a cat. The difference is an abstraction ladder — the single most important reason deep learning works at all.
If you stop a child in the middle of learning to read and ask what she's doing, you'll get different answers depending on the week. Early on: "looking at shapes." Then: "putting the sounds together." Then: "reading words." A month later: "reading sentences." A few years later, if you ask the same kid what she's doing while reading a chapter book, she'll look at you funny and say, "um, reading." The shapes don't register anymore. The sounds don't register. The words barely register. She's just… following the story.
Nothing happened to her eyes. What happened is that her brain built a ladder. Shapes became letters. Letters became syllables. Syllables became words. Words became meaning. And each step of the ladder hid the details of the step below it so that the next step could do something more interesting. By the time she's chasing a plot, she can't even see the shapes anymore — they've been abstracted away.
That ladder is the single most important idea in all of deep learning. Not calculus. Not fancy architectures. A ladder of abstractions, stacked high enough that something that started as raw pixels ends up as "that's a cat." Everything else is supporting cast.
In the previous post you met one artificial neuron — a tiny voting committee that multiplies a few numbers, adds them, and squishes the total. Individually, these things are almost pointless. In this post, we stack them, and watch "almost pointless" turn into "quietly runs the modern world." No calculus. No code. One ladder.
A pile of committees, arranged in rows
Here's what "stacking layers" actually means.
Every box with "Layer" in it is just a row of neurons — the same tiny voting committees from the last post — arranged side by side. Maybe a hundred of them. Maybe ten thousand. The key move is this: the outputs of one row of neurons become the inputs of the next row.
That's it. That's the whole architectural idea. Arrange neurons in rows, feed the outputs of one row into the next, and call it "deep" if there are a lot of rows. The "depth" in "deep learning" is not a metaphor. It's a literal count of how many rows of committees the signal passes through before an answer falls out the far end.
A network with three rows is a deep network, technically. A network with a hundred rows is very deep. The biggest modern language models are hundreds of layers deep, with hundreds of billions of weights scattered across them. But the move — feed outputs forward, one layer at a time — is the same whether you have three layers or three hundred.
The reading ladder, in slow motion
Okay, so we have rows of committees passing votes forward. Why should that be powerful? A row of committees just feeding into another row of committees sounds like… more committees. Where's the magic?
The magic is that each layer can summarize the previous layer at a higher level of abstraction, and the network, through training, figures out what to summarize at each level. The reading ladder makes this concrete.
Imagine you're teaching a network to read an English sentence from pixels on a page. Here's what the layers learn, roughly, if training goes well:
Five layers. Each one more abstract than the last.
Layer 1 · line segments. The first row of neurons doesn't see "words." It sees raw pixel brightnesses, and each committee learns to get excited about a very specific low-level feature — a short horizontal stroke here, a diagonal there, a little curve. A thousand committees in this layer will end up specialising on a thousand different tiny strokes.
Layer 2 · letters. The second row doesn't see pixels at all. It sees the pattern of which strokes layer 1 got excited about. And it turns out that "this combination of strokes" is exactly what a letter is. The letter A is three strokes in a particular configuration. The letter O is a big curve and not much else. A committee in layer 2 can learn "look for an A" just by checking whether the right strokes in layer 1 are firing.
Layer 3 · words. The third row doesn't see letters as shapes. It sees the pattern of which letters layer 2 got excited about and it learns to recognise common clusters. "The cat" is a particular pattern of letters in a particular order. Each word is, from this layer's point of view, a recognisable fingerprint across layer 2.
Layer 4 · meaning. Now we're somewhere interesting. This row doesn't see words. It sees patterns of words together, and it learns to care about things like "this sentence is about an animal, that sentence is about food, this one is a question."
Layer 5 · output. A final row that takes the abstract "what is this sentence about" signal and turns it into whatever the network was trained to produce — a category, a translation, a next word, a yes-or-no.
At no point did any human tell the network look for letters here, look for words there. The network was only given pairs of inputs and correct answers, and it figured out on its own that letters are useful because they help predict words, and words are useful because they help predict meaning, and meaning is useful because it helps predict the correct answer. The whole ladder falls out of the training process. That is wild, and that is the reason any of this works.
Why one layer can't do this
Here's a question worth chewing on: why do we need the ladder at all? Why can't a single layer, with enough neurons, just learn "cat" directly from pixels?
The honest answer — and this was a real controversy in machine learning for decades — is: theoretically it can, practically it's a disaster. A single very wide layer with a huge number of neurons is technically powerful enough to learn almost anything, in the same way that a single very long list is technically powerful enough to store almost any information. But "technically possible" and "actually works" are different questions.
The shallow version has to learn a direct mapping from every possible combination of pixels to every possible meaning, all in one step. There are astronomically many possible combinations. Most pictures of cats share almost no pixels with other pictures of cats — lighting, angle, breed, pose, background, everything varies. A shallow network has to either memorise every exact pattern it's seen or somehow magic out the deep commonalities in a single step, and neither works in practice.
A deep network cheats. It doesn't learn the cat-to-pixel mapping in one go. It learns edges, which are much easier because edges show up in almost every image. Then it learns textures and small shapes from edges, which is easier because textures repeat. Then it learns parts (eyes, ears, whiskers) from textures, which is easier because parts repeat. Then it learns cats from parts, which is easier because parts compose predictably. Each step of the ladder is a small, local generalization, and the whole ladder ends up doing something no single step could have done.
In short: depth lets the network build up complicated concepts from simple ones, instead of learning the complicated concept from scratch. That turns an impossible problem into a sequence of easy ones.
There's even a name for this in the literature: the composability of features. The world around us, it turns out, is conveniently composable. Objects are made of parts, parts are made of shapes, shapes are made of edges. Sentences are made of words, words are made of letters, letters are made of strokes. A network that mirrors this composition structure — with layers of abstraction — fits the world far better than a network that tries to skip to the end. We didn't build the world this way. We got lucky that it is this way, and deep networks are the shape of model that exploits the luck.
The one thing that breaks the spell
For decades, people knew stacking layers should work but couldn't get it to. You'd build a deep network, hit "train," and the thing would fall apart. The signal from the far end of the network couldn't travel backwards through all those layers to adjust the early weights, and training would stall. People blamed the math. The math was fine; the training setup was subtle.
A few tricks, developed slowly between roughly 2006 and 2015, broke the dam. Better activation functions (that squish step from the last post). Better ways to initialise the weights at the start. Something called residual connections — basically cheating-but-legal shortcuts that let information skip ahead a few layers so early layers don't get starved. More data, more compute, more patience. Each trick was small. Together they turned "deep networks don't really train" into "deep networks are the only thing that trains well." The 2012 moment we've been referring to since M1.4 was partly about data, partly about hardware — but it was also about this: the tricks finally clicked, and the ladder started working.
You don't need to remember any of those tricks by name. You just need to know that the reason the world suddenly noticed deep learning around 2012 is that people finally figured out how to actually train a deep network. Before that, "stack more layers" was a theoretical suggestion. After that, it was the entire industry.
Why this is the punchline of all modern AI
Every impressive AI result of the last fifteen years — image recognition, speech recognition, machine translation, chatbots, image generation, voice cloning, AlphaGo — has the same structural secret at its core: a deep network with a useful abstraction ladder. The details of the ladder change. For images, early layers see edges. For text, early layers see character patterns. For audio, early layers see local waveforms. The higher you climb, the more abstract things get, until the top of the ladder is whatever the task needs.
When people say "deep learning," this is what the "deep" refers to — not that the system understands things more deeply (it doesn't), but that there are a lot of layers of abstraction between the input and the output, and those layers were built automatically by training. Every time you hear "the model learned a rich representation," someone is describing the middle of the ladder. Every time you hear "the model generalises well," someone is saying "its ladder isn't just memorising, it's building features that transfer."
Hold onto this sentence, because we'll come back to it in every remaining post of this module:
Depth is how a neural network trades "learn everything in one step" for "learn a little at each step, in a useful order." That trade is the whole trick.
That's the sentence you want ringing in your head when we hit convolutions in M3.4 and recurrent networks in M3.5 and transformers in M4. Those architectures are all different ways of building the ladder for different kinds of data. The ladder is the constant. The rungs change.
What just changed in your head
You started this post thinking of a neural network as a black box with some fuzzy brain-ish stuff inside. You're ending it with a picture of a ladder. At the bottom, raw numbers from the world — pixels, sound samples, word scores. At the top, whatever the task needs. In between, layers of committees, each one summarising the previous layer in slightly more abstract terms, until the thing at the top has a shape simple enough to vote on.
The ladder is automatic. Nobody told the network to learn letters. Training figured out that letters were useful. That is the part that most people don't see, and it's the part worth keeping. Depth doesn't give the network understanding. It gives it room to build useful intermediate ideas, and useful intermediate ideas turn out to be nine-tenths of being good at a task.
One sentence to carry forward: if a network is doing something impressive, there is a ladder inside it, and each rung of that ladder is worth being curious about.
In the next post, we take the last thing we haven't explained: how does any of this actually get learned? How do all those millions of weights find values that make the ladder work? The answer involves rolling downhill with your eyes closed, and it's simpler than it sounds. No calculus, I promise.
Course navigation
| ⬅️ Previous | 📍 You are here | Next ➡️ |
| ⬅️ Previous M3.1 · The Neuron Is a Lie | M3.2 | Next ➡️ M3.3 · Training, Downhill Blindfolded |
📚 AI Zero to Hero · Course Home — all 33 posts, six modules.
Cover photo via Unsplash. This post is part of the AI Zero to Hero series.
