Introduction#
Learning goals and core concepts#
Before diving deep into computational graphs, backpropagation, and gradient descent, let us clarify the learning goals for this chapter:
Learning goals
Computational graph intuition: View mathematical operations as a plumbing system of “data flow”
Activation function intuition: Understand how non-linearity “folds” linearly non-separable data into separable distributions
Loss function intuition: Understand how different objectives shape the geometry of optimization problems
Backpropagation intuition: “Attribute” the final error to each parameter (the credit assignment mechanism)
Gradient descent intuition: Understand how the geometry of loss surfaces impacts optimization paths
Code connection: Observe the actual behavior of these mechanisms in PyTorch
The basic problem of machine learning#
The goal of machine learning is to let computers discover patterns from data autonomously, rather than writing rules line by line ourselves. The core challenge is automatically adjusting model parameters so the model better fits data and makes accurate predictions.
Core challenges in machine learning
Model complexity: Modern deep learning models can have millions or even billions of parameters
Optimization difficulty: Finding the optimal solution in such high-dimensional spaces is like finding an exit in a maze
Computational efficiency: Fast and resource-efficient algorithms are required to handle massive datasets and complex models
Generalization capability: Models must perform well not only on training data, but also reliably on unseen test data
Why do we need these core mechanisms?#
Neural network training involves several key steps, each requiring specific mathematical tools:
Computational graph: Describes how data flows through the network, laying the structural foundation for automatic differentiation
Activation Functions: Introduces non-linearity, enabling the network to partition complex decision boundaries
Loss Functions: Defines a metric for “prediction quality,” converting training into an optimization problem
Backpropagation Algorithm: Efficiently computes gradients and performs error “credit assignment”
Gradient Descent and Optimization Algorithms: Leverages gradient information to iteratively update parameters
These concepts work together to form the core mechanisms of modern deep learning frameworks [GBC16].
Chapter preview#
In this chapter, we will start from Computational graph to establish an intuitive perspective on “data flow.” We will then see how Activation Functions partition decision boundaries in feature space via non-linear transformations. Next, we will explore how Loss Functions define optimization objectives and shape the geometry of the loss surface. After that, we will understand how Backpropagation Algorithm efficiently performs “credit assignment”—attributing the final error back to every parameter. Finally, we will explore how Gradient Descent and Optimization Algorithms navigates loss surfaces to locate optimal parameters.
After mastering these core mechanisms, Summary and Outlook will review the key points of this chapter. In the next chapter, we will build practical neural networks using PyTorch and observe how these theories translate into training dynamics on the MNIST task.
References
Contributors and Revision History
View detailed revision history
-
e2bc8802026-06-13 - Heyan Zhu: fix: missing bib keys in pdf output -
d7c9da72026-05-12 - Heyan Zhu: docs: update sphinx only filter from not pdf to not latex -
a954ca52026-05-11 - Heyan Zhu: docs: revise and restructure documentation, fix formatting, and move env setup and sphinx guide to a separate appendix section -
f159d562026-05-08 - Heyan Zhu: fix: address complaints from the linter -
59126f42026-04-26 - Heyan Zhu: docs(math-fundamentals): update content structure and add citations -
756a7932026-04-25 - Heyan Zhu: docs(math-fundamentals): update content structure and improve explanations -
0c291d72025-12-10 - Heyan Zhu: docs: restructure course materials and add new content