algorithms by showing their motivation to resolve these challenges and how this leads to
the derivation of their update rules. We will also take a short look at algorithms and
architectures to optimize gradient descent in a parallel and distributed setting. Finally, we
will consider additional strategies that are helpful for optimizing gradient descent.
Gradient descent is a way to minimize an objective function parameterized by a
model's parameters by updating the parameters in the opposite direction of the
gradient of the objective function w.r.t. to the parameters. The learning rate
determines the size of the steps we take to reach a (local) minimum. In other words, we
follow the direction of the slope of the surface created by the objective function downhill
until we reach a valley. If you are unfamiliar with gradient descent, you can find a good
introduction on optimizing neural networks .
Gradient descent variants
There are three variants of gradient descent, which differ in how much data we use to
compute the gradient of the objective function. Depending on the amount of data, we
make a trade-off between the accuracy of the parameter update and the time it takes to
perform an update.
Batch gradient descent
Vanilla gradient descent, aka batch gradient descent, computes the gradient of the cost
function w.r.t. to the parameters for the entire training dataset:
.
As we need to calculate the gradients for the whole dataset to perform just
one
update,
batch gradient descent can be very slow and is intractable for datasets that don't fit in
memory. Batch gradient descent also doesn't allow us to update our model
online
, i.e. with
new examples on-the-fly.
In code, batch gradient descent looks something like this:
