\documentclass[11pt]{article}
%\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
%\geometry{landscape}                % Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{epstopdf}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{hyperref}
\usepackage{color}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

% margin notes
\usepackage{marginnote}
\usepackage[top=2cm, bottom=2cm, outer=7cm, inner=2cm, heightrounded, marginparwidth=5cm, marginparsep=1cm]{geometry}
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
\renewcommand*{\marginfont}{\footnotesize}

\input{../../misc/macros.tex}

\newcommand{\lossReg}{\loss_{\rm reg}}



\title{Lecture 2, Part 2: Backpropagation}
\author{Roger Grosse}
\date{}

\begin{document}
\maketitle

\section{Introduction}

So far, we've seen how to train ``shallow'' models, where the predictions are computed as a linear function of the inputs. We've also observed that deeper models are much more powerful than linear ones, in that they can compute a broader set of functions. Let's put these two together, and see how to train a multilayer neural network. We will do this using backpropagation, the central algorithm of this course. Backpropagation (``backprop'' for short) is a way of computing the partial derivatives of a loss function with respect to the parameters of a network; we use these derivatives in gradient descent, exactly the way we did with linear regression and logistic regression. 

If you've taken a multivariate calculus class, you've probably encountered the Chain Rule for partial derivatives, a generalization of the Chain Rule from univariate calculus. In a sense, backprop is ``just'' the Chain Rule --- but with some interesting twists and potential gotchas. This lecture and Lecture 6 focus on backprop. (In between, we'll see a cool example of how to use it.) This lecture covers the mathematical justification and shows how to implement a backprop routine by hand. Implementing backprop can get tedious if you do it too often. In Lecture 6, we'll see how to implement an \emph{automatic differentiation} engine, so that derivatives even of rather complicated cost functions can be computed automatically. (And just as efficiently as if you'd done it carefully by hand!)

This will be your least favorite lecture, since it requires the most tedious derivations of the whole course.

\subsection{Learning Goals}

\begin{itemize}
\item Be able to compute the derivatives of a cost function using backprop.
\end{itemize}

\subsection{Background}

I would highly recommend reviewing and practicing the Chain Rule for partial derivatives. I'd suggest Khan Academy\footnote{\url{https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/multivariable-chain-rule/v/multivariable-chain-rule}}, but you can also find lots of resources on Metacademy\footnote{\url{https://metacademy.org/graphs/concepts/chain_rule}}.


\section{The Chain Rule revisited}

Before we get to neural networks, let's start by looking more closely at an example we've already covered: a linear classification model. For simplicity, let's assume we have univariate inputs and a single training example $(\inputUni, \target)$. The predictions are a linear function followed by a sigmoidal nonlinearity. Finally, we use the squared error loss function. The model and loss function are as follows:
\begin{align}
\intermediate &= \weightUni \inputUni + \bias \\
\prediction &= \sigmoid(\intermediate) \\
\loss &= \frac{1}{2} (\prediction - \target)^2
\end{align}
Now, to change things up a bit, let's add a \emph{regularizer} to the cost function. We'll cover regularizers properly in a later lecture, but intuitively, they try to encourage ``simpler'' explanations. In this example, we'll use the regularizer $\frac{\weightCost}{2} \weightUni^2$, which encourages $\weightUni$ to be close to zero. ($\weightCost$ is a hyperparameter; the larger it is, the more strongly the weights prefer to be close to zero.) The cost function, then, is:
\begin{align}
\regularizer &= \frac{1}{2} \weightUni^2 \\
\lossReg &= \loss + \weightCost \regularizer. \label{eqn:cost}
\end{align}
In order to perform gradient descent, we wish to compute the partial derivatives $\partial \cost / \partial \weightUni$ and $\partial \cost / \partial \bias$.

This example will cover all the important ideas behind backprop; the only thing harder about the case of multilayer neural nets will be the cruftier notation.

\subsection{How you would have done it in calculus class}
\label{sec:naive}

Recall that you can calculate partial derivatives the same way you would calculate univariate derivatives. In particular, we can expand out the cost function in terms of $\weightUni$ and $\bias$, and then compute the derivatives using repeated applications of the univariate Chain Rule.
\begin{align*}
\lossReg &= \frac{1}{2} ( \sigmoid(\weightUni \inputUni + \bias) - \target)^2 + \frac{\weightCost}{2} \weightUni^2 \\
\frac{\partial \lossReg}{\partial \weightUni} &= \frac{\partial}{\partial \weightUni} \left[ \frac{1}{2} ( \sigmoid(\weightUni \inputUni + \bias) - \target)^2 + \frac{\weightCost}{2} \weightUni^2 \right] \\
&= \frac{1}{2} \frac{\partial}{\partial \weightUni} ( \sigmoid(\weightUni \inputUni + \bias) - \target)^2 + \frac{\weightCost}{2} \frac{\partial}{\partial \weightUni} \weightUni^2 \\
&= (\sigma(\weightUni \inputUni + \bias) - \target) \frac{\partial}{\partial \weightUni} (\sigma(\weightUni \inputUni + \bias) - \target) + \weightCost \weightUni \\
&= (\sigma(\weightUni \inputUni + \bias) - \target) \sigma^\prime (\weightUni \inputUni + \bias) \frac{\partial}{\partial \weightUni} (\weightUni \inputUni + \bias) + \weightCost \weightUni \\
&= (\sigma(\weightUni \inputUni + \bias) - \target) \sigma^\prime (\weightUni \inputUni + \bias) \inputUni + \weightCost \weightUni \\
\frac{\partial \lossReg}{\partial \bias} &= \frac{\partial}{\partial \bias} \left[ \frac{1}{2} ( \sigmoid(\weightUni \inputUni + \bias) - \target)^2 + \frac{\weightCost}{2} \weightUni^2 \right] \\
&= \frac{1}{2} \frac{\partial}{\partial \bias} ( \sigmoid(\weightUni \inputUni + \bias) - \target)^2 + \frac{\weightCost}{2} \frac{\partial}{\partial \bias} \weightUni^2 \\
&= ( \sigmoid(\weightUni \inputUni + \bias) - \target) \frac{\partial}{\partial \bias} ( \sigmoid(\weightUni \inputUni + \bias) - \target) + 0 \\
&= (\sigma(\weightUni \inputUni + \bias) - \target) \sigma^\prime (\weightUni \inputUni + \bias) \frac{\partial}{\partial \bias} (\weightUni \inputUni + \bias) \\
&= (\sigma(\weightUni \inputUni + \bias) - \target) \sigma^\prime (\weightUni \inputUni + \bias) 
\end{align*}

This gives us the correct answer, but hopefully it's apparent from this example that this method has several drawbacks:
\begin{enumerate}
\item The calculations are very cumbersome. In this derivation, we had to copy lots of terms from one line to the next, and it's easy to accidentally drop something. (In fact, I made such a mistake while writing these notes!) While the calculations are doable in this simple example, they become impossibly cumbersome for a realistic neural net.
\item The calculations involve lots of redundant work. For instance, the first three steps in the two derivations above are nearly identical. 
\item Similarly, the final expressions have lots of repeated terms, which means lots of redundant work if we implement these expressions directly.\marginnote{Actually, even in this derivation, I used the ``efficiency trick'' of not expanding out $\sigmoid^\prime$. If I had expanded it out, the expressions would be even more hideous, and would involve \emph{six} copies of $\weightUni \inputUni + \bias$.} For instance, $\weightUni \inputUni + \bias$ is computed a total of four times between $\partial \cost / \partial \weightUni$ and $\partial \cost / \partial \bias$. The larger expression $(\sigma(\weightUni \inputUni + \bias) - \target) \sigma^\prime (\weightUni \inputUni + \bias)$ is computed twice. If you happen to notice these things, then perhaps you can be clever in your implementation and factor out the repeated expressions. But, as you can imagine, such efficiency improvements might not always jump out at you when you're implementing an algorithm. 
\end{enumerate}
The idea behind backpropagation is to share the repeated computations wherever possible. We'll see that the backprop calculations, if done properly, are very clean and modular.

\subsection{Multivariable chain rule: the easy case}

We've already used the univariate Chain Rule a bunch of times, but it's worth remembering the formal definition:
\begin{equation}
\frac{\deriv}{\deriv t} f(g(t)) = f^\prime(g(t)) g^\prime(t).
\end{equation}
Roughly speaking, increasing $t$ by some infinitesimal quantity $h_1$ ``causes'' $g$ to change by the infinitesimal $h_2 = g^\prime(t) h_1$. This in turn causes $f$ to change by $f^\prime(g(t)) h_2 = f^\prime(g(t)) g^\prime(t) h_1$.

The multivariable Chain Rule is a generalization of the univariate one. Let's say we have a function $f$ in two variables, and we want to compute $\frac{d}{dt} f(x(t), y(t))$. Changing $t$ slightly has two effects: it changes $x$ slightly, and it changes $y$ slightly. Each of these effects causes a slight change to $f$. For infinitesimal changes, these effects combine additively. The Chain Rule, therefore, is given by:
\begin{equation}
\frac{\deriv}{\deriv t} f(x(t), y(t)) = \frac{\partial f}{\partial x} \frac{\deriv x}{\deriv t} + \frac{\partial f}{\partial y} \frac{\deriv y}{\deriv t}. \label{eqn:chain-rule}
\end{equation}


\subsection{An alternative notation}

It will be convenient for us to introduce an alternative notation for the derivatives we compute. In particular, notice that the left-hand side in all of our derivative calculations is $\deriv \loss / \deriv v$, where $v$ is some quantity we compute in order to compute $\loss$. (Or substitute for $\loss$ whichever variable we're trying to compute derivatives of.) We'll use the notation
\begin{equation}
\costDeriv{v} \triangleq \frac{\partial \loss}{\partial v}.
\end{equation}
This notation is less crufty, and also emphasizes that $\costDeriv{v}$ is a value we compute, rather than a mathematical expression to be evaluated. This notation is nonstandard; see the appendix if you want more justification for it.

We can rewrite the multivariable Chain rule (Eqn.~\ref{eqn:chain-rule}) using this notation:
\begin{equation}
\costDeriv{t} = \costDeriv{x} \frac{\deriv x}{\deriv t} + \costDeriv{y} \frac{\deriv y}{\deriv t}. \label{eqn:chain-rule-rewritten}
\end{equation}
Here, we use $\deriv x / \deriv t$ to mean we should actually evaluate the derivative algebraically in order to determine the formula for $\costDeriv{t}$, whereas $\costDeriv{x}$ and $\costDeriv{y}$ are values previously computed by the algorithm.



\subsection{Using the computation graph}
\label{sec:computation-graph}

In this section, we finally introduce the main algorithm for this course, which is known as {\bf backpropagation}, or {\bf reverse mode automatic differentiation (autodiff)}.\footnote{Automatic differentiation was invented in 1970, and backprop in the late 80s. Originally, backprop referred to the special case of reverse mode autodiff applied to neural nets, although the derivatives were typically written out by hand (rather than using an autodiff package). But in the last few years, neural nets have gotten so diverse that we basically think of them as compositions of functions. Also, very often, backprop is now implemented using an autodiff software package. For these reasons, the distinction between autodiff and backprop has gotten blurred, and we will use the terms interchangeably in this course. Note that there is also a forward mode autodiff, but it's rarely used in neural nets, and we won't cover it in this course.}

Now let's return to our running example, written again for convenience:
\begin{align*}
\intermediate &= \weightUni \inputUni + \bias \\
\prediction &= \sigmoid(\intermediate) \\
\loss &= \frac{1}{2} (\prediction - \target)^2 \\
\regularizer &= \frac{1}{2} \weightUni^2 \\
\lossReg &= \loss + \weightCost \regularizer. \label{eqn:cost}
\end{align*}
Let's introduce the {\bf computation graph}. The nodes in the graph correspond to all the values that are computed, with edges to indicate which values are computed from which other values. The computation graph for our running example is shown in Figure \ref{fig:cg-simple}.\marginnote{Note that the computation graph is \emph{not} the network architecture. The nodes correspond to values that are computed, rather than to units in the network.}

\begin{figure}
\begin{center}
\includegraphics[width=0.5 \textwidth]{figures/reg_computation_graph.png}
\end{center}
\caption{Computation graph for the regularized linear regression example in Section~\ref{sec:computation-graph}. The magenta arrows indicate the case which requires the multivariate chain rule because $\weightUni$ is used to compute both $\intermediate$ and $\regularizer$.}
\label{fig:cg-simple}
\end{figure}

The goal of backprop is to compute the derivatives $\costDeriv{\weightUni}$ and $\costDeriv{\bias}$. We do this by repeatedly applying the Chain Rule (Eqn.~\ref{eqn:chain-rule-rewritten}). Observe that to compute a derivative using Eqn.~\ref{eqn:chain-rule-rewritten}, you first need the derivatives for its \emph{children} in the computation graph. This means we must start from the result of the computation (in this case, $\cost$) and work our way backwards through the graph. It is because we work backward through the graph that backprop and reverse mode autodiff get their names.

Let's start with the formal definition of the algorithm. Let $v_1, \ldots, v_N$ denote all of the nodes in the computation graph, in a topological ordering. (A topological ordering is any ordering where parents come before children.) We wish to compute all of the derivatives $\costDeriv{v_i}$, although we may only be interested in a subset of these values. We first compute all of the values in a {\bf forward pass}, and then compute the derivatives in a {\bf backward pass}. As a special case, $v_N$ denotes the result of the computation (in our running example, $v_N = \cost$), and is the thing we're trying to compute the derivatives of. Therefore, by convention, we set $\costDeriv{v_N} = 1$.\marginnote{$\costDeriv{\cost} = 1$ because increasing the cost by $h$ increases the cost by $h$.} The algorithm is as follows:
\begin{itemize}
\item[] For $i = 1, \ldots, N$
\begin{itemize}
\item[] Compute $v_i$ as a function of ${\rm Pa}(v_i)$
\end{itemize}
\item[] $v_N = 1$
\item[] For $i = N-1, \ldots, 1$
\begin{itemize}
\item[] $\costDeriv{v_i} = \sum_{j \in {\rm Ch}(v_i)} \costDeriv{v_j} \frac{\partial v_j}{\partial v_i}$
\end{itemize}
\end{itemize}
Here ${\rm Pa}(v_i)$ and ${\rm Ch}(v_i)$ denote the parents and children of $v_i$.


This procedure may become clearer when we work through the example in full:
\begin{align*}
\costDeriv{\loss_{\rm reg}} &= 1 \\
\costDeriv{\regularizer} &= \costDeriv{\loss_{\rm reg}}\, \frac{\deriv \loss_{\rm reg}}{\deriv \regularizer} \\
                            &= \costDeriv{\loss_{\rm reg}} \, \weightCost  \\
\costDeriv{\loss} &= \costDeriv{\loss_{\rm reg}}\, \frac{\deriv \loss_{\rm reg}}{\deriv \loss} \\
                            &= \costDeriv{\loss_{\rm reg}} \\
\costDeriv{\prediction} &= \costDeriv{\loss}\, \frac{\deriv \loss}{\deriv \prediction} \\
                            &= \costDeriv{\loss}\, (\prediction - \target) \\
\costDeriv{\intermediate} &= \costDeriv{\prediction} \, \frac{\deriv \prediction}{\deriv \intermediate} \\
                            &= \costDeriv{\prediction} \, \logistic^\prime(\intermediate) \\
\costDeriv{\weightUni} &= \costDeriv{\intermediate} \, \frac{\partial \intermediate}{\partial \weightUni} + \costDeriv{\regularizer} \frac{\deriv \regularizer}{\deriv \weightUni}  \\
                            &= \costDeriv{\intermediate} \, \inputUni + \costDeriv{\regularizer} \, \weightUni  \\
\costDeriv{\bias} &= \costDeriv{\intermediate} \, \frac{\partial \intermediate}{\partial \bias} \\
                            &= \costDeriv{\intermediate}
\end{align*}
Since we've derived a procedure for computing $\costDeriv{\weightUni}$ and $\costDeriv{\bias}$, we're done. Let's write out this procedure without the mess of the derivation, so that we can compare it with the \naive method of Section \ref{sec:naive}:
\begin{align*}
\costDeriv{\loss_{\rm reg}} &= 1 \\
\costDeriv{\regularizer} &= \costDeriv{\loss_{\rm reg}} \, \weightCost  \\
\costDeriv{\loss} &= \costDeriv{\loss_{\rm reg}} \\
\costDeriv{\prediction} &= \costDeriv{\loss}\, (\prediction - \target) \\
\costDeriv{\intermediate} &= \costDeriv{\prediction} \, \logistic^\prime(\intermediate) \\
\costDeriv{\weightUni} &= \costDeriv{\intermediate} \, \inputUni + \costDeriv{\regularizer} \, \weightUni  \\
\costDeriv{\bias} &= \costDeriv{\intermediate}
\end{align*}
The derivation, and the final result, are much cleaner than with the \naive method. There are no redundant computations here.\marginnote{Actually, there's one redundant computation, since $\sigma(\intermediate)$ can be reused when computing $\sigma^\prime(\intermediate)$. But we're not going to focus on this point.} Furthermore, the procedure is \emph{modular}: it is broken down into small chunks that can be reused for other computations. For instance, if we want to change the loss function, we'd only have to modify the formula for $\costDeriv{\prediction}$. With the \naive method, we'd have to start over from scratch.


\section{Backprop on a multilayer net}

Now we come to the prototypical use of backprop: computing the loss derivatives for a multilayer neural net. This introduces no new ideas beyond what we've already discussed, so think of it as simply another example to practice the technique. We'll use a multilayer net like the one from the previous lecture, and squared error loss with multiple output units:
\begin{align*}
\intermediateK{\hidIdx} &= \sum_{\featIdx} \weightLIJ{1}{\hidIdx}{\featIdx} \inputJ{\featIdx} + \biasLI{1}{\hidIdx} \\
        \hiddenI{\hidIdx} &= \logistic(\intermediateK{\hidIdx}) \\
        \predictionK{\outputIdx} &= \sum_{\hidIdx} \weightLIJ{2}{\outputIdx}{\hidIdx} \hiddenI{\hidIdx} + \biasLI{2}{\outputIdx} \\
        \loss &= \frac{1}{2} \sum_\outputIdx (\predictionK{\outputIdx} - \targetK{\outputIdx})^2
\end{align*}
As before, we start by drawing out the computation graph for the network. The case of two input dimensions and two hidden units is shown in Figure \ref{fig:cg-backprop}(a). Because the graph clearly gets pretty cluttered if we include all the units individually, we can instead draw the computation graph for the vectorized form (Figure \ref{fig:cg-backprop}(b)), as long as we can mentally convert it to Figure \ref{fig:cg-backprop}(a) as needed.

\begin{figure}
\begin{center}
(a) \includegraphics[width=0.4 \textwidth]{figures/mlp_graph.png}
(b) \includegraphics[width=0.4 \textwidth]{figures/mlp_graph_vectorized.png}
\end{center}
\caption{{\bf (a)} Full computation graph for the loss computation in a multilayer neural net. {\bf (b)} Vectorized form of the computation graph.}
\label{fig:cg-backprop}
\end{figure}

Based on this computation graph, we can work through the derivations of the backwards pass just as before.\marginnote{One you get used to it, feel free to skip the step where we write down $\lossDeriv{\loss}$.}
\begin{align*}
        \costDeriv{\loss} &= 1 \\
        \costDeriv{\predictionK{\outputIdx}} &= \costDeriv{\loss} \, (\predictionK{\outputIdx} - \targetK{\outputIdx}) \\
        \costDeriv{\weightLIJ{2}{\outputIdx}{\hidIdx}} &= \costDeriv{\predictionK{\outputIdx}} \, \hiddenI{\hidIdx} \\
        \costDeriv{\biasLI{2}{\outputIdx}} &= \costDeriv{\predictionK{\outputIdx}} \\
        \costDeriv{\hiddenI{\hidIdx}} &= \sum_{\outputIdx} \costDeriv{\predictionK{\outputIdx}} \weightLIJ{2}{\outputIdx}{\hidIdx} \\
        \costDeriv{\intermediateK{\hidIdx}} &= \costDeriv{\hiddenI{\hidIdx}} \, \logistic^\prime(\intermediateK{\hidIdx}) \\
        \costDeriv{\weightLIJ{1}{\hidIdx}{\featIdx}} &= \costDeriv{\intermediateK{\hidIdx}} \, \inputJ{\featIdx} \\
        \costDeriv{\biasLI{1}{\hidIdx}} &= \costDeriv{\intermediateK{\hidIdx}} \\
\end{align*}
Focus especially on the derivation of $\lossDeriv{\hiddenI{\hidIdx}}$, since this is the only step which actually uses the multivariable Chain Rule.

Once we've derived the update rules in terms of indices, we can find the vectorized versions the same way we've been doing for all our other calculations. For the forward pass:
      \begin{align*}
        \intermediates &= \weightMatL{1} \inputVec + \biasesL{1} \\
        \hiddens &= \logistic(\intermediates) \\
        \predictions &= \weightMatL{2} \hiddens + \biasesL{2} \\
        \loss &= \frac{1}{2} \| \targets - \predictions \|^2
      \end{align*}
And the backward pass:
      \begin{align*}
        \costDeriv{\loss} &= 1 \\
        \costDeriv{\predictions} &= \costDeriv{\loss} \, (\predictions - \targets) \\
        \costDeriv{\weightMatL{2}} &= \costDeriv{\predictions} \hiddens^\transpose \\
        \costDeriv{\biasesL{2}} &= \costDeriv{\predictions} \\
        \costDeriv{\hiddens} &= \weightMat^{(2) \transpose} \costDeriv{\predictions} \\
        \costDeriv{\intermediates} &= \costDeriv{\hiddens} \circ \logistic^\prime(\intermediates) \\
        \costDeriv{\weightMatL{1}} &= \costDeriv{\intermediates} \inputVec^\transpose \\
        \costDeriv{\biasesL{1}} &= \costDeriv{\intermediates}
      \end{align*}





% \subsection{Vectorization}

% We've just derived a backprop routine which we could implement directly in Python, but doing so would be very inefficient. Just as we did for linear regression and logistic regression, we need to vectorize the computations. This means, first of all, representing the network parameters using matrices and vectors. Let $\ndim$, $\nhid$, and $\nout$ denote the numbers of input dimensions, hidden units, and output units, respectively. There is one first-layer weight for every pair of an input dimension and a hidden unit, so we will represent the first-layer weights with an $\ndim \times \nhid$ matrix $\weightMatL{1}$. Similarly, there is one first-layer bias for every hidden unit, so the first-layer biases will be represented by an $\nhid$-dimensional vector $\biasesL{1}$. The second layer weights and biases are represented analogously.

% We also need to vectorize the inputs and activations. Just as we did for linear regression, we'll store the inputs as an $\ndata \times \ndim$ matrix $\inputMatrix$. Each data point has its own set of hidden activations, so the hidden activations will be represented by an $\ndata \times \nhid$ matrix $\hiddenMatrix$. The other activations are represented analogously.

% The loss is summed over the training examples, so let's write this out explicitly in the formulas: \TODO{should I divide by $\ndata$ here?}
% \begin{align*}
% \lossDeriv{\weightLKI{2}{\outputIdx}{\hidIdx}} &= \sum_{\dataIdx=1}^\ndata \lossDeriv{\outIntermediateK{\outputIdx}^{(\ndata)}} \, \hiddenI{\hidIdx}^{(\ndata)} \\
% \lossDeriv{\biasLK{2}{\outputIdx}} &= \sum_{\dataIdx=1}^\ndata \lossDeriv{\outIntermediateK{\outputIdx}^{(\ndata)}} \\
% \lossDeriv{\weightLIJ{1}{\hidIdx}{\dimIdx}} &= \sum_{\dataIdx=1}^\ndata \lossDeriv{\intermediateK{\hidIdx}^{(\ndata)}} \, \inputJ{\dimIdx}^{(\ndata)} \\
% \lossDeriv{\biasLI{1}{\dimIdx}} &= \sum_{\dataIdx=1}^\ndata \lossDeriv{\intermediateK{\hidIdx}^{(\ndata)}}
% \end{align*}

% Now, we can vectorize all the backprop computations exactly the way we vectorized the gradient computations for linear regression and logistic regression. There's nothing conceptually new here. As before, we will use $\partial \loss / \partial \predictionMatrix$ as shorthand for the matrix of partial derivatives with respect to individual elements of $\predictionMatrix$, and $\circ$ denotes elementwise products.\marginnote{As with our previous vectorizations, you should sanity check the derivation by making sure the dimensions of the matrices match up.}
% \begin{align*}
% \lossDeriv{\loss} &= 1 \\
% \lossDeriv{\predictionMatrix} &= \lossDeriv{\loss} \, \frac{\partial \loss}{\partial \predictionMatrix} \\
% \lossDeriv{\outIntermediateMat} &= \lossDeriv{\predictionMatrix} \circ \activationFunctionTwo^\prime(\outIntermediateMat) \\
% \lossDeriv{\weightMatL{2}} &= \hiddenMatrix^\transpose \lossDeriv{\outIntermediateMat} \\
% \lossDeriv{\biasesL{2}} &= \onesVec^\transpose \lossDeriv{\outIntermediateMat} \\
% \lossDeriv{\hiddenMatrix} &= \lossDeriv{\outIntermediateMat}\,  [\weightMatL{2}]^\transpose \\
% \lossDeriv{\intermediateMatrix} &= \lossDeriv{\hiddenMatrix} \circ \activationFunction^\prime(\intermediateMatrix) \\
% \lossDeriv{\weightMatL{1}} &= \inputMatrix^\transpose \lossDeriv{\intermediateMatrix} \\
% \lossDeriv{\biasesL{1}} &= \onesVec^\transpose \lossDeriv{\intermediateMatrix}.
% \end{align*}
% We can easily translate this into an efficient NumPy implementation of backprop: \TODO{}


\section{Appendix: why the weird notation?}


Recall that the partial derivative $\partial \cost / \partial \weightUni$ means, how much does $\cost$ change when you make an infinitesimal change to $\weightUni$, holding everything else fixed?  But this isn't a well-defined notion, because it depends what we mean by ``holding everything else fixed.''  In particular, Eqn.~\ref{eqn:cost} defines the cost as a function of two arguments; writing this explicitly,
\begin{equation}
\cost(\loss, \weightUni) = \loss + \frac{\weightCost}{2} \weightUni^2. \label{eqn:gotcha}
\end{equation}
Computing the partial derivative of this function with respect to $\weightUni$,
\begin{equation}
\frac{\partial \cost}{\partial \weightUni} = \weightCost \weightUni. \label{eqn:gotcha-deriv}
\end{equation}
But in the previous section, we (correctly) computed
\begin{equation}
\frac{\partial \cost}{\partial \weightUni} = (\sigma(\weightUni \inputUni + \bias) - \target) \sigma^\prime (\weightUni \inputUni + \bias) \inputUni + \weightCost \weightUni. \label{eqn:gotcha-deriv-full}
\end{equation}
What gives?  Why do we get two different answers?

The problem is that mathematically, the notation $\partial \cost / \partial \weightUni$ denotes the partial derivative of a \emph{function} with respect to one of its \emph{arguments}. We make an infinitesimal change to one of the arguments, while holding the rest of the arguments fixed. When we talk about partial derivatives, we need to be careful about what are the arguments to the function. When we compute the derivatives for gradient descent, we treat $\cost$ as a function of the parameters of the model --- in this case, $\weightUni$ and $\bias$. In this context, $\partial \cost / \partial \weightUni$ means, how much does $\cost$ change if we change $\weightUni$ while holding $\bias$ fixed?  By contrast, Eqn.~\ref{eqn:gotcha} treats $\cost$ as a function of $\loss$ and $\weightUni$; in Eqn.~\ref{eqn:gotcha}, we're making a change to the second argument to $\cost$ (which happens to be denoted $\weightUni$), while holding the first argument fixed.

Unfortunately, we need to refer to \emph{both} of these interpretations when describing backprop, and the partial derivative notation just leaves this difference implicit. Doubly unfortunately, our field hasn't consistently adopted any notational conventions which will help us here. There are dozens of explanations of backprop out there, most of which simply ignore this issue, letting the meaning of the partial derivatives be determined from context. This works well for experts, who have enough intuition about the problem to resolve the ambiguities. But for someone just starting out, it might be hard to deduce the meaning from context.

That's why I picked the bar notation. It's the least bad solution I've been able to come up with.

%In the interest of precision and clarity, I'll adopt the following conventions throughout this course. They're the least bad solution I've been able to come up with.
% \begin{itemize}
% \item When we're discussing backprop computations:
% \begin{itemize}
% \item Partial derivative notation will always denote partial derivatives of specific expressions, as in Eqn.~\ref{eqn:gotcha-deriv}.
% \item Derivatives of the loss with respect to other variables will be denoted with a bar. For instance, the LHS of Eqn.~\ref{eqn:gotcha-deriv-full} will be denoted $\costDeriv{\weightUni}$.
% \end{itemize}
% \item Outside the context of backprop, $\cost$ will always be treated as a function of the parameters (weights and biases) of a model. $\partial \cost / \partial \weightUni$ will always have the meaning as in Eqn.~\ref{eqn:gotcha-deriv-full}, i.e.~it will be the same as the value $\costDeriv{\weightUni}$ computed during backprop. There will be no reason to refer to derivatives with respect to intermediate quantities.
% \end{itemize}





\end{document}












