Descent method — Steepest descent and conjugate gradient in Python

Python implementation

Descent method — Steepest descent and conjugate gradient (math explained)

Let’s start with this equation and we want to solve for x:

The solution x the minimize the function below when A is symmetric positive definite (otherwise, x could be the maximum). It is because the gradient of f(x), ∇f(x) = Ax- b. And when Ax=b, ∇f(x)=0 and thus x is the minimum of the function.

In this article, I am going to show you two ways to find the solution x — method of Steepest Descent and method of Conjugate Gradient.

Method of Steepest Descent in Python

Now let’s use this steepest_descent function to calculate

With the steepest_descent method, we get a value of (-4,5) and a wall time 2.01ms.