Chris Akampa Sirmuel
Technology Researcher, Tecrozon
In our first article, we examined how an artificial neuron transforms numerical inputs into an output. We followed the calculation through weights, bias, and an activation function, discovering how several numbers can be combined into a single result. But calculating an output does not automatically make a neural network intelligent or useful.
A neuron can produce a number without knowing whether that number is correct. A network can process thousands of examples and still make poor predictions if its parameters are unsuitable for the task. The central question, therefore, is not simply how a neural network calculates. It is how the network's parameters are adjusted so that its calculations become more useful.
This is the mathematics of learning. To understand it, we must examine what happens after a prediction has been made: how the system measures its error, how calculus identifies the effect of its parameters, and how an optimization algorithm uses that information to update the network.
1. The Learning Problem: Comparing a Prediction with a Target
Imagine a simple model designed to predict a numerical value. It receives an input x, multiplies it by a weight w, and produces a prediction:
ŷ = wx
The symbol ŷ, pronounced y-hat, represents the predicted value. The actual ground-truth target is written as y.
Suppose the input is x = 2, and the current weight setting is w = 0.2. The model computes:
x = 2, w = 0.2
ŷ = 0.2 × 2 = 0.4
Now suppose the correct target is y = 1. The model has predicted 0.4 when the target is 1. The raw difference between the prediction and the target is:
ŷ − y = 0.4 − 1 = −0.6
The negative sign tells us that the prediction is below the target. However, the raw difference is not always sufficient to measure performance across a dataset. Training therefore commonly uses a mathematical function called a loss function. The loss function converts the prediction and target into a numerical measure of how poorly the model performed according to a chosen objective.
2. Loss Functions: Turning Prediction Errors into Numbers
One common loss function for numerical prediction is squared error:
L = (ŷ − y)²
Here, L represents the loss for this example. Substituting our values:
L = (0.4 − 1)²
L = (−0.6)² = 0.36
The error was negative, but squaring it produces a positive loss. Why square the difference? Squaring penalizes larger errors more heavily than smaller ones. An error of 0.6 produces a squared loss of 0.36, while an error of 0.1 produces a loss of only 0.01. This quadratic penalty is one reason squared error is fundamental in regression and numerical prediction.
Other tasks use different loss functions. Classification systems, for example, typically use cross-entropy loss to evaluate predicted probability distributions. The appropriate choice depends entirely on what the model is being trained to do. A loss function is not simply a universal measurement of intelligence; it measures performance according to a specified mathematical objective.
The training process now has a quantity it can attempt to reduce. But there is an immediate problem: knowing that the loss is 0.36 does not, by itself, tell us exactly how to change the weight. For that, we need calculus.
Figure 1. The loss surface L(w) = (2w − 1)². At the current weight w = 0.20, the derivative is −2.4. Because the slope is negative, increasing the weight moves downhill toward the minimum at w = 0.50.
3. Derivatives: Discovering How a Parameter Affects Loss
A derivative describes how a function changes when one of its variables changes. In neural-network training, we are particularly interested in how the loss changes when a weight or bias changes.
Return to our model:
ŷ = wx
L = (wx − y)²
The weight w influences the prediction, and the prediction influences the loss. Therefore, changing w changes L. We want to calculate ∂L/∂w, which expresses the derivative of the loss with respect to the weight.
Using differentiation and the chain rule:
∂L/∂w = 2(wx − y) · x
We already know our concrete starting values: x = 2, w = 0.2, and y = 1. Substituting these into the derivative formula:
∂L/∂w = 2(0.2 × 2 − 1)(2)
= 2(0.4 − 1)(2)
= 2(−0.6)(2)
= −2.4
This derivative is called a gradient component. Because our model in this example has only one adjustable weight, the gradient consists of this single number.
What does −2.4 tell us? At the current parameter value, the loss decreases locally if we increase the weight by a sufficiently small amount. Conversely, decreasing the weight slightly would increase the loss. The derivative supplies the precise local direction in which the parameter should move to reduce the loss. It does not provide the perfect weight immediately; instead, it gives us a local direction for improvement.
4. Gradient Descent: Updating the Weight
Once the gradient has been calculated, an optimization algorithm uses it to update the parameter. One of the simplest and most fundamental methods is gradient descent. Its update rule is:
w_new = w_old − η · (∂L/∂w)
The symbol η (eta), called the learning rate, controls the size of the update step. In our example:
w_old = 0.2
∂L/∂w = −2.4
Suppose we choose a learning rate of η = 0.1. Then:
w_new = 0.2 − (0.1 × −2.4)
w_new = 0.2 + 0.24 = 0.44
Because subtracting a negative number is equivalent to adding a positive number, the weight has changed from 0.20 to 0.44.
Notice the sequence: the model did not guess a replacement weight at random. The gradient supplied exact information about how the loss responds to the parameter, and the learning rate determined the size of the update. The result is a new parameter value that we can evaluate by running the model again.
Figure 2. One complete gradient descent step. The weight adjusts from 0.20 to 0.44 (left), which pushes the prediction from 0.40 to 0.88 (right), reducing the squared loss from 0.36 to 0.0144.
5. Did the Update Actually Improve the Prediction?
We should never assume that an update worked merely because the weight changed. We must calculate the new prediction and evaluate its loss on the objective function.
Our original prediction was:
ŷ = 0.2 × 2 = 0.4
L = (0.4 − 1)² = 0.36
After the update, the weight is 0.44. The new forward pass produces:
ŷ_new = 0.44 × 2 = 0.88
The new loss becomes:
L_new = (0.88 − 1)²
L_new = (−0.12)² = 0.0144
The loss has fallen from 0.36 to 0.0144. This is a substantial improvement:
| Measurement |
Before update |
After update |
| Weight (w) |
0.20 |
0.44 |
| Prediction (ŷ) |
0.40 |
0.88 |
| Target (y) |
1.00 |
1.00 |
| Squared loss (L) |
0.3600 |
0.0144 |
The model is now predicting a value much closer to the target. But it has not reached the target exactly. The weight can be adjusted again, and another prediction can be evaluated. This repeated cycle of prediction, measurement, gradient evaluation, and update is the foundation of gradient-based learning.
One crucial qualification matters: an update does not always reduce the loss. An excessively large learning rate can overshoot a useful parameter value or make training completely unstable. The outcome depends intimately on the loss function, the parameter values, and the optimization method.
6. The Learning Rate: Why the Size of an Update Matters
The learning rate determines how far a parameter moves in response to its gradient. We can see its dramatic influence using our exact example, where the gradient is −2.4:
If η = 0.01: w_new = 0.2 − (0.01 × −2.4) = 0.224 (too small, barely changes)
If η = 0.10: w_new = 0.2 − (0.10 × −2.4) = 0.440 (effective convergence step)
If η = 0.50: w_new = 0.2 − (0.50 × −2.4) = 1.400 (severe overshoot)
For this last update with η = 0.5, we can calculate the resulting prediction and squared loss:
ŷ = 1.4 × 2 = 2.8
L = (2.8 − 1)² = (1.8)² = 3.24
The loss has exploded from 0.36 to 3.24. The learning rate was far too large for this update step.
Figure 3. The impact of learning rate on a loss surface. A rate that is too small makes almost no progress (left); a balanced rate moves directly to the valley floor (center); an excessively large rate overshoots to the opposite wall and causes divergence (right).
This demonstrates why successful training requires more than knowing the gradient. The optimization process must also carefully govern how aggressively parameters respond to that information. In practical neural networks, suitable learning rates depend on the model architecture, dataset, loss function, and optimizer. There is no single universal learning rate that works for every problem.
7. But a Neural Network Has Many Weights
Our introductory example has a single weight. Real neural networks possess thousands, millions, or even hundreds of billions of adjustable parameters. Each weight can influence the network's final output and, consequently, its loss.
For a model with parameters collected into a vector θ = (θ₁, θ₂, …, θₙ), the gradient is a collection of partial derivatives:
∇_θ L = ( ∂L/∂θ₁, ∂L/∂θ₂, …, ∂L/∂θₙ )
Each component indicates how the loss changes locally with respect to one parameter, holding all other parameters fixed for that partial derivative. In basic gradient descent, all parameters can be updated simultaneously:
θ_new = θ_old − η · ∇_θ L
The network does not need to change parameters one at a time. It calculates gradients for all parameters and updates them together. However, those parameters are deeply interconnected through the network's computational graph. Changing one weight alters downstream activations across multiple layers, which cascades into the loss. This is why computing gradients efficiently becomes the paramount computational challenge. The algorithm that solves this in multilayer architectures is backpropagation.
8. Backpropagation: Tracing the Error Through the Network
Consider a multilayer network. During the forward pass, input values propagate forward through successive layers. Each neuron calculates a weighted sum, adds its bias, applies an activation function, and transmits its output onward. Eventually, the network produces a prediction, and the loss function evaluates that prediction against the target.
Backpropagation then calculates how the loss depends on parameters throughout the network. The key mathematical tool is the chain rule of calculus.
Suppose an early parameter affects a hidden-layer value, which affects another layer, which eventually affects the final prediction and its loss. The chain rule allows us to multiply the derivatives along this sequence of dependencies:
w → z → a → L
∂L/∂w = (∂L/∂a) · (∂a/∂z) · (∂z/∂w)
Each factor describes one specific link in the computational chain:
∂z/∂w describes how the pre-activation changes with the weight.
∂a/∂z describes how the activation changes with the pre-activation value.
∂L/∂a describes how the loss changes with the activation.
Multiplying these derivatives connects the parameter directly to the final loss. Backpropagation applies this principle systematically across the entire network, working backward through the computational graph to obtain gradients for every weight and bias.
Figure 4. The computational graph and chain rule decomposition. Forward pass operations compute predictions and loss (solid lines); backpropagation flows in reverse (dashed lines), multiplying local sensitivities to compute ∂L/∂w.
It is essential to distinguish two separate steps: backpropagation computes gradients; an optimizer uses those gradients to update parameters. They work in close concert, but they are distinct operations. In software frameworks such as PyTorch, automatic differentiation (autograd) tracks operations during the forward pass and executes backpropagation automatically.
9. What Happens When the Network Learns from Many Examples?
So far, we have examined a single training example. Real-world machine learning involves datasets with thousands or millions of samples.
Suppose a model is learning to predict house prices. Its training data contains numerous property features and corresponding market prices. For each sample, the model produces a prediction and the loss measures the discrepancy. The optimization process aggregates information across examples to compute parameter updates.
A standard approach is mini-batch training. Instead of updating parameters after every single example (which can be noisy) or waiting to process the entire massive dataset (which is computationally slow), the model processes a balanced batch of examples and updates weights based on the batch gradient:
1. Select a mini-batch of training examples.
2. Run them through the forward pass to generate predictions.
3. Calculate the batch loss.
4. Compute gradients through backpropagation.
5. Update parameters using the optimizer.
6. Repeat with subsequent batches until an epoch completes.
Figure 5. The mini-batch training loop. The model repeatedly samples batches, calculates forward predictions and loss, backpropagates gradients, and updates weights until convergence.
An epoch represents one complete traversal through the entire training dataset. The process repeats across multiple batches and epochs, progressively reshaping how the network responds to input patterns.
However, a steadily declining training loss does not guarantee that the model will perform well on unfamiliar data. A model can overfit its training set, memorizing sample idiosyncrasies rather than learning true underlying regularities. Models must therefore be evaluated against a separate validation dataset that was never used during training.
10. Are All Neural Networks Trained Using the Same Method?
No. While gradient descent and backpropagation provide the mathematical foundation for modern deep learning, concrete implementations vary widely:
| Optimizer |
Mechanism |
Primary Use Case |
| SGD (Stochastic) |
Updates weights using gradients from individual samples or small mini-batches. |
Foundational optimization, convex objectives, fine-tuning. |
| Adam |
Maintains per-parameter adaptive learning rates based on running first and second moments. |
Transformers, large language models, deep convolutional networks. |
| AdamW |
Decouples L2 weight decay regularization from gradient moment updates. |
State-of-the-art LLM training and visual transformers. |
Furthermore, other machine learning paradigms introduce entirely distinct training mechanics. Reinforcement learning relies on policy gradients and reward signals rather than supervised targets. Self-supervised learning formulates surrogate objectives, such as predicting masked tokens in text. Nevertheless, the unifying principle across all these methods remains unchanged: define an objective, calculate its sensitivities via calculus, and systematically update parameters.
11. What Learning Does—and Does Not—Mean
When a neural network learns, its numerical parameters shift in response to an optimization objective. The resulting transformations often become remarkably effective at classifying inputs, synthesizing text, or recognizing sensory patterns.
However, this does not mean the network possesses cognitive understanding in the human sense. It does not comprehend why its predictions work. Nor does training guarantee that every update is beneficial, that the network discovers a globally optimal rule, or that it will generalize reliably outside its training distribution.
The system is being minimized against a mathematical loss function. How faithfully that loss captures the real-world task, how representative the data is, and how rigorously the model is evaluated govern whether the resulting system is genuinely useful.
Conclusion: Learning is a Process of Parameter Adjustment
An artificial neuron calculates an output from its inputs and parameters. Learning begins when predictions are judged against an objective.
A loss function quantifies the discrepancy. Calculus determines how that loss responds to model parameters. Optimization algorithms use those gradient directions to update weights. In multilayer networks, backpropagation applies the chain rule backward through the computational graph.
The essential architecture is consistent: the model generates a prediction, measures performance, identifies parameter sensitivities, and executes gradient-based updates.
The next question is what happens when thousands or millions of these parameter-dependent calculations are connected into multi-layer architectures, where each layer shapes the representations of the next. That is the subject of our third research article: how individual neurons assemble into deep neural networks, and how representations and gradients flow through the complete system.
The Mathematics of Weight Adjustment in Summary
1. Forward Pass: ŷ = wx
2. Loss Evaluation: L = (ŷ − y)²
3. Gradient Calculus: ∂L/∂w = 2(wx − y) · x
4. Parameter Update: w ← w − η · (∂L/∂w)
5. Deep Chain Rule: ∂L/∂w = (∂L/∂a) · (∂a/∂z) · (∂z/∂w)
Learning is not magic; it is the systematic, calculus-driven adjustment of parameters until predictions align with truth.
References
PyTorch Documentation — A Gentle Introduction to torch.autograd. Explains forward propagation, loss, backpropagation, gradient calculation, and optimizer updates.
PyTorch Documentation — Automatic Differentiation with torch.autograd. Covers computational graphs, derivatives, and the chain rule used to calculate gradients.
Google for Developers — Machine Learning Crash Course. Provides structured learning material on loss, gradient descent, and neural-network training.
Google for Developers — Neural Networks. Explores neural-network layers, nonlinear patterns, and backpropagation.