Deriving the Learning Correction

Deriving the Learning Correction#

For gradient descent, we need to derive the update to the matrix $A$ based on training on a set of our data, $(x^{k}, y^{k})$ .

Let’s start with our cost function:

f (A_{i j}) = \sum_{i = 1}^{N_{out}} (z_{i} - y_{i}^{k})^{2} = \sum_{i = 1}^{N_{out}} [g (\underset{\equiv α_{i}}{\underset{⏟}{\sum_{j = 1}^{N_{in}} A_{i j} x_{j}^{k}}}) - y_{i}^{k}]^{2}

where we’ll refer to the product $α \equiv A x$ to help simplify notation.

We can compute the derivative with respect to a single matrix element, $A_{p q}$ by applying the chain rule:

\frac{\partial f}{\partial A_{p q}} = 2 \sum_{i = 1}^{N_{out}} (z_{i} - y_{i}^{k}) {\frac{\partial g}{\partial ξ} |}_{ξ = α_{i}} \frac{\partial α_{i}}{\partial A_{p q}}

with

\frac{\partial α_{i}}{\partial A_{p q}} = \sum_{j = 1}^{N_{in}} \frac{\partial A_{i j}}{\partial A_{p q}} x_{j}^{k} = \sum_{j = 1}^{N_{in}} δ_{i p} δ_{j q} x_{j}^{k} = δ_{i p} x_{q}^{k}

and for $g (ξ)$ , we will assume the sigmoid function,so

\frac{\partial g}{\partial ξ} = \frac{\partial}{\partial ξ} \frac{1}{1 + e^{- ξ}} = - (1 + e^{- ξ})^{- 2} (- e^{- ξ}) = g (ξ) \frac{e^{- ξ}}{1 + e^{- ξ}} = g (ξ) (1 - g (ξ))

which gives us:

\begin{aligned} \frac{\partial f}{\partial A_{p q}} & = 2 \sum_{i = 1}^{N_{out}} (z_{i} - y_{i}^{k}) z_{i} (1 - z_{i}) δ_{i p} x_{q}^{k} \\ = 2 (z_{p} - y_{p}^{k}) z_{p} (1 - z_{p}) x_{q}^{k} \end{aligned}

where we used the fact that the $δ_{i p}$ means that only a single term contributes to the sum.

Note that:

$e_{p}^{k} \equiv (z_{p} - y_{p}^{k})$ is the error on the output layer, and the correction is proportional to the error (as we would expect).
The $k$ superscripts here remind us that this is the result of only a single pair of data from the training set.

Now $z$ and $y^{k}$ are all vectors of size $N_{out} \times 1$ and $x^{k}$ is a vector of size $N_{in} \times 1$ , so we can write this expression for the matrix as a whole as:

\frac{\partial f}{\partial A} = 2 (z - y^{k}) \circ z \circ (1 - z) \cdot (x^{k})^{⊺}

where the operator $\circ$ represents element-by-element multiplication (the Hadamard product).

Performing the update#

We could do the update like we just saw with our gradient descent example: take a single data point, $(x^{k}, y^{k})$ and do the full minimization, continually estimating the correction, $\partial f / \partial A$ and updating $A$ until we reach a minimum. The problem with this is that $(x^{k}, y^{k})$ is only one point in our training data, and there is no guarantee that if we minimize completely with point $k$ that we will also be a minimum with point $k + 1$ .

Instead we take multiple passes through the training data (called epochs) and apply only a single push in the direction that gradient descent suggests, scaled by a learning rate $η$ .

The overall minimization appears as:

* Loop over epochs

Loop over the training data, ${(x^{0}, y^{0}), (x^{1}, y^{1}), \dots}$ . We’ll refer to the current training pair as $(x^{k}, y^{k})$
- Propagate $x^{k}$ through the network, getting the output $z = g ({A x}^{k})$
- Compute the error on the output layer, $e^{k} = z - y^{k}$
- Update the matrix $A$ according to:
  
  $A \leftarrow A - 2 η e^{k} \circ z \circ (1 - z) \cdot (x^{k})^{⊺}$

Deriving the Learning Correction

Contents

Deriving the Learning Correction#

Performing the update#