How neural networks are built

Stacking neurons into layers is easy. Finding the right weights — that’s the engineering.

In this section we look at the two pieces that turn the basic neuron from the last section into a working deep-learning system. First: how networks are organised into layers. Second: how those layers are trained using a beautiful algorithm called back-propagation, which made all of modern deep learning possible.

Layers, in plain words

A neural network is just neurons arranged in layers. Each layer has many neurons; each neuron takes inputs from the previous layer.

• Input layer. The features themselves. For our Khalti example, the three numbers (amount, time_of_day, new_device).
• Hidden layers. One or more layers of neurons. Each neuron in a hidden layer looks at all of the previous layer’s outputs and produces a new number.
• Output layer. The final layer that produces what we actually want — a probability for binary classification, a number for regression, a vector of word probabilities for a language model.

“Deep” learning simply means many hidden layers. ChatGPT-class models have around 100 hidden layers. A small Nepali Devanagari classifier might have 3 or 4. The basic shape is the same.

A worked picture: Devanagari digit classifier

Imagine a network that recognises a handwritten Devanagari digit ० through ९ from a 28×28 pixel image. The input layer has 784 numbers — one per pixel. The output layer has 10 numbers — one probability per digit.

In between, we use two hidden layers, each with say 64 neurons.

• The 64 neurons in the first hidden layer each look at all 784 pixels. After training, each one becomes a detector for some simple pattern — a curve in the top-right, an edge in the middle, a circle.
• The 64 neurons in the second hidden layer each look at the 64 outputs of the first layer. After training, these have learned higher-level patterns — “two stacked circles,” “left-hanging curve with right-side vertical bar.”
• The 10 output neurons each look at the 64 outputs of the second layer and produce a probability for one digit.

The whole network has roughly 784 × 64 + 64 × 64 + 64 × 10 ≈ 55,000 weights. Modest by 2026 standards. Trainable in a few minutes on a laptop.

Activation functions: the squishes

We mentioned the sigmoid in the last section. There are a few common activation functions, each with its niche.

• Sigmoid. Maps any real number to (0, 1). Useful for the output layer of a binary classifier — it produces something interpretable as a probability.
• ReLU (Rectified Linear Unit). Returns the input if it is positive; returns 0 otherwise. Why this? Because it makes deep networks much easier to train. This single innovation in 2010 is one of the reasons deep learning works at scale.
• Softmax. Used in the output layer of a multi-class classifier (like our 10-digit example). It converts 10 raw numbers into 10 probabilities that sum to 1.
• Tanh. Like sigmoid, but maps to (−1, 1). Sometimes useful in hidden layers.

Choosing activation functions is a craft skill. The default in 2026 is “ReLU in hidden layers, softmax in classification outputs.” That choice is right far more often than it is wrong.

Backpropagation, the algorithm

We have a network. We initialise its 55,000 weights to small random numbers. We show it a Devanagari digit image. Of course it produces nonsense — the weights are random.

We compute how wrong the network was: how far the output probabilities are from a one-hot encoding of the true label. Call this the loss. The training problem is to change the weights so the loss is smaller on the next image. But there are 55,000 weights, all interacting through layers. How can we adjust each one?

Backpropagation answers this. It says: starting from the loss at the output, propagate the error backwards through the network, layer by layer, computing for each weight how much it contributed to the loss. Each weight then gets a tiny nudge in the direction that would have reduced the loss.

Repeat across the entire training set, dozens or hundreds of times. The weights gradually settle into values that make the network’s outputs match the labels.

That is the entire algorithm. Backpropagation is essentially the chain rule from calculus, applied systematically to a stack of weighted sums and activation functions. The mathematical beauty is that it costs roughly the same compute as running the network forward — without that property, training a 100-layer network would be hopelessly slow.

What can go wrong in training

Three common failure modes, all worth recognising:

1. Vanishing gradients. In very deep networks with sigmoid activations, the error signal shrinks toward zero as it propagates backwards. Early layers stop learning. The cure: ReLU activations, careful initialisation, and architectural choices (residual connections).

2. Overfitting. The network memorises the training data and fails to generalise to new examples. Cures: more data, smaller networks, regularisation tricks like dropout, and crucially — holding out validation data and watching loss on it.

3. Bad learning rate. Each backprop step adjusts weights by a small amount controlled by the learning rate. Too high, and training diverges. Too low, and it never finishes. Modern optimisers (Adam, AdamW) adjust the learning rate automatically and have made this much less painful than it used to be.

What you should remember

Two ideas:

• A neural network is layers of neurons. Each neuron is a weighted sum plus activation. The output of one layer is the input of the next.
• Training is backpropagation: compute loss, push the error backwards, nudge each weight, repeat.

That description fits both your two-layer Devanagari classifier and OpenAI’s 100-layer GPT. The difference is scale, architecture, and compute — not the underlying algorithm.

Check your understanding

Quick check

—

What does the backpropagation algorithm do during the training of a neural network?

1. AGenerates random weights at the start of training
2. BComputes how much each weight contributed to the network's error and adjusts each one slightly to reduce that error
3. CSorts the training data by class label
4. DLoads the input images from disk

Check answer→

What comes next

We have basic networks and the training algorithm. The next section is the two big architectural ideas of modern deep learning — convolutions for images and transformers for language — both of which started as small twists on the basic layered network and ended up powering almost every famous AI system in 2026.