This project explores gradient descent as a geometric optimization process. The goal is to build intuition for derivatives, gradients, and optimization by visualizing how gradient descent behaves on simple functions.
The project is intentionally not focused on machine learning performance. Instead, it treats gradient descent as a calculus concept and studies its behavior step by step using analytical functions.
This project was created while studying calculus, with a focus on understanding derivatives and gradients beyond symbolic computation.
Gradient descent is often introduced as a standard algorithm, but its geometric meaning is rarely emphasized. Here, the algorithm is examined as a process of moving along a surface using only local slope information.
The goal is to connect calculus concepts to optimization in a transparent and interpretable way.
- A function defines a surface (or curve).
- The derivative or gradient encodes local directional information.
- Gradient descent moves in the direction of the negative gradient.
- The trajectory depends on the learning rate and initialization.
- The method has inherent limitations that can be visualized.
No datasets, no models, no black-box optimization.
Introduces the derivative as the slope of a function. Tangent lines are used to show how derivatives encode direction.
Implements gradient descent for a simple one-dimensional function and visualizes the optimization trajectory step by step.
Demonstrates how different learning rates lead to slow convergence, stable convergence, oscillation, or divergence.
Extends gradient descent to two dimensions. The gradient is visualized as a vector field, and trajectories are shown on contour plots.
Explores non-convex functions and saddle points. Shows how gradient descent can converge to local minima or stall near saddle points.
- Gradient descent is a local method.
- The learning rate strongly affects behavior.
- Initialization matters in non-convex settings.
- A zero gradient does not guarantee a minimum.
- Visualization reveals limitations that formulas alone hide.
The project uses only standard scientific Python libraries:
- numpy
- matplotlib
- jupyter
Gradient descent is simple, powerful, and widely used. However, it is not a universal solution.
By visualizing its behavior on simple functions, this project highlights both the strengths and the fundamental limitations of gradient-based optimization.
Understanding these geometric properties is essential before applying gradient descent to complex models.