Summary and Outlook#

Core Chapter Review#

In this chapter, we established the mathematical foundations of deep learning and built a complete training workflow:

  1. Computational graph: Describes “how data flows”

    • Intuition: Like a factory assembly line, tracing processing steps from input to output.

    • Role: Visualizes complex computations and provides structure for Backpropagation Algorithm.

  2. Learning Task Formulations: Defines “what patterns to look for”

    • Intuition: Classification finds decision boundaries, regression finds mappings, and autoregression finds sequence dependencies.

    • Role: Different task types correspond to different output formats and loss functions.

  3. Activation Functions: Introduces non-linear expressiveness

    • Intuition: Draws decision boundaries in feature space, “folding” linearly inseparable data into separable regions.

    • Key: ReLU is the default choice for hidden layers, mitigating the vanishing gradient problem.

  4. Loss Functions: Defines “what makes a good prediction”

    • Intuition: Measures the gap between predictions and targets, shaping the optimization landscape for Gradient Descent and Optimization Algorithms.

    • Key: Cross-entropy provides strong gradients for classification, while MSE suits regression.

  5. Backpropagation Algorithm: “Credit assignment” mechanism

    • Intuition: Tracing back responsibility after a project failure, assigning blame proportionally to contributions.

    • Core: The chain rule propagates gradients efficiently back through the Computational graph.

  6. Gradient Descent and Optimization Algorithms: Walking downhill in the loss landscape

    • Intuition: Step by step toward the valley floor along the steepest direction computed by Backpropagation Algorithm.

    • Key Point: The learning rate is crucial; Adam is the default go-to.

Training Workflow Overview#

Interconnections among sections in this chapter:

        graph LR
    A[数据] --> B[计算图前向传播]
    B --> C[确定任务类型]
    C --> D[经过激活函数]
    D --> E[损失函数计算误差]
    E --> F[反向传播计算梯度]
    F --> G[梯度下降更新参数]
    G --> B

This is the core deep learning training loop:

  1. Data flows into the computational graph to generate predictions based on the task type (classification/regression/autoregression).

  2. The loss function evaluates prediction quality.

  3. Backpropagation computes gradients for every parameter.

  4. Gradient descent adjusts parameters to decrease the loss.

  5. Repeat until convergence.

Quick Reference of Key Points#

Concept

Core Intuition

Practical Advice

Computational Graph

Data pipeline

Built automatically by PyTorch; can be visualized for debugging

Task Formulation

Classification finds decision boundaries, regression finds mappings, autoregression finds dependencies

Determine the task type first, then choose the loss function and output layer

Activation Functions

Divides decision boundaries in feature space

Use ReLU for hidden layers; select output layer based on task

Loss Function

Shapes optimization landscape

Use cross-entropy for classification, MSE/MAE for regression

Backpropagation

Credit assignment

Handled automatically by frameworks; conceptual understanding suffices

Gradient Descent

Downhill strategy

Default to Adam; use Warmup + Cosine decay for large models

Learning Rate

Step size

Most critical hyperparameter; commonly 0.001–0.1

Frequently Asked Questions#

Q: Why are activation functions needed? A: Without activation functions, a multi-layer network is equivalent to a single linear transformation and cannot learn complex patterns.

Q: What is the relationship between backpropagation and gradient descent? A: Backpropagation computes gradients, while gradient descent uses gradients to update parameters. Together they complete training.

Q: What is the relationship between loss functions and optimization algorithms? A: The loss function defines “where to go” (the goal), while the optimization algorithm determines “how to get there” (the path).

Q: Are local minima really a big problem? A: In high-dimensional deep networks, saddle points are much more common than local minima. Noise from stochastic gradients usually helps escape them.

Moving to the Next Chapter#

The mathematical foundations in this chapter will be put into practice in 上篇：练习峰:

  • Perceptrons and MLPs: Building actual networks using computational graphs

  • Convolutional Neural Networks: Processing spatial features of images

  • Recurrent Neural Networks: Processing temporal features of sequences (see 序列建模：从 RNN 到 Transformer 再到 Mamba for details)

  • Training Techniques: Batch normalization, Dropout, early stopping, etc.

Are you ready?

Now you understand “why deep learning works”. In the next chapter, we will learn “how to make it work better”—moving from theory to practice to build and train real neural networks.

Remember: Math is merely a tool, intuition is your guide, and practice is the sole criterion for testing truth.

Recommended Resources#

English Resources#

Quick Review:

In-Depth Understanding:

  • Deep Learning (Goodfellow et al.), Chapters 4–6

  • Stanford CS231n Course (Fei-Fei Li et al., Computer Vision + Deep Learning)

Hands-On Practice:

Chinese Resources#

Video Courses:

Documentation & Tutorials:

End of chapter. Next step: 上篇：练习峰

Contributors and Revision History

View detailed revision history
  • dc35df2 2026-06-14 - Heyan Zhu: refactor: unify into two-chapter mountain-climbing narrative
  • 8eb3275 2026-05-09 - Heyan Zhu: feat(sequence-modeling): add new chapter on sequence modeling from RNN to Mamba
  • f159d56 2026-05-08 - Heyan Zhu: fix: address complaints from the linter
  • 59126f4 2026-04-26 - Heyan Zhu: docs(math-fundamentals): update content structure and add citations
  • ae2053f 2026-04-26 - Heyan Zhu: docs(math-fundamentals): add task-formulations and update related content
  • 756a793 2026-04-25 - Heyan Zhu: docs(math-fundamentals): update content structure and improve explanations
  • 0c291d7 2025-12-10 - Heyan Zhu: docs: restructure course materials and add new content