\documentclass[11pt]{article}
%\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
%\geometry{landscape}                % Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{epstopdf}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{hyperref}
\usepackage{color}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

% margin notes
\usepackage{marginnote}
\usepackage[top=2cm, bottom=2cm, outer=7cm, inner=2cm, heightrounded, marginparwidth=5cm, marginparsep=1cm]{geometry}
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
\renewcommand*{\marginfont}{\footnotesize}

\input{../../misc/macros.tex}


\title{Lecture 1, Part 3: Training a Classifier}
\author{Roger Grosse}
\date{}

\begin{document}
\maketitle

\section{Introduction}

Now that we've defined what binary classification is, let's actually train a classifier. We'll approach this problem in much the same way as we did linear regression: define a model and a cost function, and minimize the cost using gradient descent. The one thing that makes the classification case harder is that it's not obvious what loss function to use. We can't just use the classification error itself, because the gradient is zero almost everywhere!  Instead, we'll define a \emph{surrogate loss function}, i.e.~an alternative loss function which is easier to optimize.

\subsection{Learning Goals}
\begin{itemize}
\item Understand why classification error and squared error are problematic cost functions for classification.
\item Know what cross-entropy is and understand why it can be easier to optimize than squared error (assuming a logistic activation function).
\item Be able to derive the gradient descent updates for all of the models and cost functions mentioned in this lecture and to implement the learning algorithms in Python.
\item Know what hinge loss is, and how it relates to cross-entropy loss.
\item Understand how binary logistic regression can be generalized to multiple variables.
\item Know what it means for a function to be convex, how to check convexity visually, and why it's important for optimization. 
\begin{itemize}
  \item Algebraically proving functions to be convex is beyond the scope of this course.
\end{itemize}
\item Know how to check the correctness of gradients using finite differences.
\end{itemize}


\section{Choosing a cost function}

Recall the setup from the previous lecture. We have a set of training examples $\{(\inputI{\dataIdx}, \targetI{\dataIdx})\}_{\dataIdx=1}^\ndata$, where the $\inputI{\dataIdx}$ are vector-valued inputs, and $\targetI{\dataIdx}$ are binary-valued targets. We would like to learn a binary linear classifier, where we compute a linear function of $\inputI{\dataIdx}$ and threshold it at zero:
\begin{equation}
\prediction = \left\{ \begin{array}{ll} 1 & \textrm{if } \weights^\transpose \inputVec > 0 \\ 0 & \textrm{otherwise.} \end{array} \right.
\end{equation}
In the last lecture, our goal was to correctly classify every training example. But this might be impossible if the dataset isn't linearly separable. Even if it's possible to correctly classify every training example, it may be undesirable since then we might just overfit!

How can we define a sensible learning criterion when the dataset isn't linearly separable?  One natural criterion is to minimize the number of misclassified training examples. We can formalize this with the {\bf classification error} loss, or the {\bf 0-1 loss}:
\begin{equation}
\zeroOneLoss(\prediction, \target) = \left\{ \begin{array}{ll} 0 & \textrm{if } \prediction = \target \\ 1 & \textrm{otherwise.} \end{array} \right.
\end{equation}
As always, the cost function is just the loss averaged over the training examples; in this case, that corresponds to the {\bf error rate}, or fraction of misclassified examples. How do we make this small?


\subsection{Attempt 1: 0-1 loss}

As a first attempt, let's try to minimize 0-1 loss directly. In order to compute the gradient descent updates, we need to compute the partial derivatives $\partial \zeroOneLoss / \partial \weightJ{\featIdx}$. Rather than mechanically deriving this derivative, let's think about what it means. It means, how much does $\zeroOneLoss$ change if you make a very small change to $\weightJ{\featIdx}$?  As long as we're not on the classification boundary, making a small enough change to $\weightJ{\featIdx}$ will have \emph{no effect} on $\zeroOneLoss$, because the prediction won't change. This implies that $\partial \zeroOneLoss / \partial \weightJ{\featIdx} = 0$, as long as we're not on the boundary. Gradient descent will go nowhere. (If we are on the boundary, the cost is discontinuous, which certainly isn't any better!) OK, we certainly can't optimize 0-1 loss with gradient descent.\marginnote{Near the end of the course, when we discuss reinforcement learning, we'll see an algorithm which can minimize 0-1 loss directly. It's nowhere near as efficient as gradient descent, though, so we still need the techniques of this lecture!}

%As a first attempt, let's try to minimize 0-1 loss directly. In order to compute the gradient descent updates, we need to compute the partial derivatives $\partial \zeroOneLoss / \partial \weightJ{\featIdx}$. Recall that, in general, a partial derivative is just the single-variable derivative with respect to one variable, with all other variables held fixed. Observe also that we have a composition of functions: $\zeroOneLoss$ is a function of $\prediction$, and $\prediction$ is a function of $\weightJ{\featIdx}$ (treating the other weights as constants).\marginnote{Note the difference in notation here: $\deriv$ is used for single-variable derivatives, while $\partial$ is used for partial derivatives.} Therefore,
%\begin{equation}
%\frac{\partial \zeroOneLoss}{\partial \weightJ{\featIdx}} = \frac{\deriv \zeroOneLoss}{\deriv \prediction} \frac{\partial \prediction}{\partial \weightJ{\featIdx}}
%\end{equation}
%Rather than mechanically computing these derivatives, let's think about what they mean. 
%

\subsection{Attempt 2: linear regression}

Since that didn't work, let's try using something we already know: linear regression. Recall that this assumes a linear model and the squared error loss function:
\begin{align}
\prediction &= \weights^\transpose \inputVec + \bias \\
\squaredErrorLoss(\prediction, \target) &= \frac{1}{2} (\prediction - \target)^2
\end{align}
We've already seen two ways of optimizing this: gradient descent, and a closed-form solution. But does it make sense for classification?  One obvious problem is that the predictions are real-valued rather than binary. But that's OK, since we can just pick some scheme for binarizing them, such as thresholding at $\prediction = 1/2$. When we replace a loss function we trust with another one we trust less but which is easier to optimize, the replacement one is called a {\bf surrogate loss function}.

But there's still a problem. Suppose we have a positive example, i.e.~$\target = 1$. If we predict $\prediction = 1$, we get a cost of 0, whereas if we make the wrong prediction $\prediction = 0$, we get a cost of 1/2; so far, so good. But suppose we're really confident that this is a positive example, and predict $\prediction = 9$. Then we pay a cost of $\frac{1}{2} (9 - 1)^2 = 32$. This is far higher than the cost for $\prediction = 0$, so the learning algorithm will try very hard to prevent this from happening. That's not bad in itself, but it means that something else might need to be sacrificed, if it's impossible to match all of the targets exactly. Perhaps the sacrifice will be that it incorrectly classifies some other training examples.


\subsection{Attempt 3: logistic nonlinearity}

The problem with linear regression is that the predictions were allowed to take arbitrary real values. But it makes no sense to predict anything smaller than 0 or larger than 1.\marginnote{If you predict $\prediction >1$, then regardless of the target, you can decrease the loss by setting $\prediction =1$. Similarly for $\prediction < 0$.} Let's fix this problem by applying a {\bf nonlinearity}, or {\bf activation function}, which squashes the predictions to be between 0 and 1. In particular, we'll use something called the {\bf logistic function}:
\begin{equation}
\sigmoid(\intermediate) = \frac{1}{1 + e^{-\intermediate}}.
\end{equation}
This is a kind of {\bf sigmoidal}, or S-shaped, function:
\begin{center}
\includegraphics[width=0.4 \textwidth]{figures/logistic_function.png}
\end{center}
What's important about this function is that it increases monotonically, with asymptotes at 0 and 1. (Plus, it's smooth, so we can compute derivatives.)\marginnote{Another advantage of the logistic function is that calculations tend to work out very nicely.} 


We refine the model as follows:
\begin{align}
\intermediate &= \weights^\transpose \inputVec + \bias \\
\prediction &= \sigmoid(\intermediate) \\
\squaredErrorLoss(\prediction, \target) &= \frac{1}{2} (\prediction - \target)^2.
\end{align}
Notice that this model solves the problem we observed with linear regression. As the predictions get more and more confident on the correct answer, the loss continues to decrease. 

To derive the gradient descent updates, we'll need the partial derivatives of the cost function. We'll do this by applying the Chain Rule twice: first to compute $\deriv \squaredErrorLoss / \deriv \intermediate$, and then again to compute $\partial \squaredErrorLoss / \partial \weightJ{\featIdx}$. But first, let's note the convenient fact that\marginnote{This is equivalent to the elegant identity $\sigmoid^\prime(\intermediate) = \sigmoid(\intermediate) (1 - \sigmoid(\intermediate))$.}
\begin{align}
\frac{\partial \prediction}{\partial \intermediate} &= \frac{e^{-\intermediate}}{(1 + e^{-\intermediate})^2} \nonumber \\
&= \prediction (1 - \prediction). \label{eqn:sigmoid-derivative}
\end{align}
Now for the Chain Rule:
\begin{align}
\frac{\deriv \squaredErrorLoss}{\deriv \intermediate} &= \frac{\deriv \squaredErrorLoss}{\deriv \prediction} \frac{\deriv \prediction}{\deriv \intermediate} \nonumber \\
&= (\prediction - \target) \prediction (1 - \prediction) \label{eqn:se-derivative-formula} \\
\frac{\partial \squaredErrorLoss}{\partial \weightJ{\featIdx}} &= \frac{\deriv \squaredErrorLoss}{\deriv \intermediate} \frac{\partial \intermediate}{\partial \weightJ{\featIdx}} \nonumber \\
&= \frac{\deriv \squaredErrorLoss}{\deriv \intermediate} \cdot \inputJ{\featIdx}. \label{eqn:se-derivative-formula-weights}
\end{align}
Done!\marginnote{At this point, you should stop and sanity check the equations we just derived, e.g.~checking that they have the sign that they ought to. Get in the habit of doing this.}  Why don't we go one step further and plug Eqn.~\ref{eqn:se-derivative-formula} into Eqn.~\ref{eqn:se-derivative-formula-weights}?  The reason is that our goal isn't to compute a \emph{formula} for $\partial \squaredErrorLoss / \partial \weightJ{\featIdx}$; as computer scientists, our goal is to come up with a \emph{procedure} for computing it. The two formulas above give us a procedure which we can implement directly in Python. One advantage of doing it this way is that we can reuse some of the work we've done in computing the derivative with respect to the bias:\marginnote{Reusing computation of derivatives is one of the main insights behind backpropagation, one of the central algorithms in this course.}
\begin{align}
\frac{\deriv \squaredErrorLoss}{\deriv \bias} &= \frac{\deriv \squaredErrorLoss}{\deriv \intermediate} \frac{\partial \intermediate}{\partial \bias} \nonumber \\
&= \frac{\deriv \squaredErrorLoss}{\deriv \intermediate} \label{eqn:se-derivative-formula-bias}
\end{align}
If we had expanded out the entire formula, it might not be obvious to us that we can reuse computation like this.

\begin{figure}
\begin{center}
\includegraphics[width=0.5 \textwidth]{figures/logistic_least_squares_problem.png}
\end{center}
\caption{Visualization of derivatives of squared error loss with logistic nonlinearity, for a training example with $\target=1$. The derivative $\deriv \cost / \deriv \intermediate$ corresponds to the slope of the tangent line.}
\label{fig:logistic-least-squares-problem}
\end{figure}

So far, so good. But there's one hitch. Let's suppose you classify one of the training examples extremely wrong, e.g.~you confidently predict a negative label with $\intermediate = -5$, which gives $\prediction \approx 0.0067$, for a positive example (i.e.~$\target = 1$). Plugging these values into Eqn~\ref{eqn:se-derivative-formula}, we find that $\partial \squaredErrorLoss / \partial \intermediate \approx -0.0066$. This is a pretty small value, considering how big the mistake was. As shown in Figure~\ref{fig:logistic-least-squares-problem}, the more confident the wrong prediction, the \emph{smaller} the gradient is!  The most badly misclassified examples will have hardly any effect on the training. This doesn't seem very good. We say the learning algorithm does not have a strong {\bf gradient signal} for those training examples.

The problem with squared error loss in the classification setting is that it doesn't distinguish bad predictions from extremely bad predictions.\marginnote{Think about how the argument in this paragraph relates to the one in the previous paragraph.} If $\target = 1$, then a prediction of $\prediction = 0.01$ has roughly the same squared-error loss as a prediction of $\prediction = 0.00001$, even though in some sense the latter is more wrong. This isn't necessarily a problem in terms of the cost function itself: whether $0.00001$ is inherently much worse than $0.01$ depends on the situation. (If all we care about is classification error, they're essentially equivalent.) But \emph{from the perspective of optimization}, the fact that the losses are nearly equivalent is a big problem.\marginnote{Actually, the effect discussed here can also be beneficial, because it makes the algorithm \emph{robust}, in that it can learn to ignore mislabeled examples. Cost functions like this are sometimes used for this reason. However, when you do use it, you should be aware of the optimization difficulties it creates!} If we can increase $\prediction$ from $0.00001$ to $0.0001$, that means we're ``getting warmer,'' but this doesn't show up in the squared-error loss. We'd like a loss function which reflects our intuitive notion of getting warmer.


\subsection{Final touch: cross-entropy loss}
\label{sec:logistic-regression}

The problem with squared-error loss is that it treats $\prediction = 0.01$ and $\prediction = 0.00001$ as nearly equivalent (for a positive example). We'd like a loss function which makes these very different. One such loss function is {\bf cross-entropy (CE)}.\marginnote{You'll sometimes see cross-entropy abbreviated XE.} This is defined as follows:
\begin{equation}
\crossEntropyLoss(\prediction, \target) = \left\{ \begin{array}{ll} -\log \prediction & \textrm{if } \target = 1 \\ -\log 1 - \prediction & \textrm{if } \target = 0 \end{array} \right. \label{eqn:cross-entropy-cases}
\end{equation}
In our earlier example, we see that $\crossEntropyLoss(0.01, 1) = 4.6$, whereas $\crossEntropyLoss(0.00001, 1) = 11.5$, so cross-entropy treats the latter as much worse than the former. 

When we do calculations, it's cumbersome to use the case notation, so we instead rewrite Eqn.~\ref{eqn:cross-entropy-cases} in the following form. You should check that they are equivalent:
\begin{equation}
\crossEntropyLoss(\prediction, \target) = -\target \log \prediction - (1 - \target) \log 1 - \prediction.
\end{equation}

Remember, \marginnote{See if you can derive the equations for the asymptote lines.} the logistic function squashes $y$ to be between 0 and 1, but cross-entropy draws big distinctions between probabilities close to 0 or 1. Interestingly, these effects cancel out: Figure~\ref{fig:logistic-regression-loss} plots the loss as a function of $\intermediate$. You get a sizable gradient signal even when the predictions are very wrong.

\begin{figure}
\begin{center}
\includegraphics[width=0.4 \textwidth]{figures/logistic_cross_entropy.png}
\end{center}
\caption{Plot of cross-entropy loss as a function of the input $\intermediate$ to the logistic activation function.}
\label{fig:logistic-regression-loss}
\end{figure}

When we combine the logistic activation function with cross-entropy loss, you get {\bf logistic regression}:
\begin{equation}
\begin{aligned}
\intermediate &= \weights^\transpose \inputVec + \bias \\
\prediction &= \sigmoid(\intermediate) \\
\crossEntropyLoss &= -\target \log \prediction - (1 - \target) \log 1 - \prediction.
\end{aligned}
\label{eqn:naive-logistic-regression}
\end{equation}

Now let's compute the derivatives. We'll do it two different ways: the mechanical way, and the clever way. Let's do the mechanical way first, as an example of the chain rule for derivatives. Remember, our job here isn't to produce a formula for the derivatives, the way we would in calculus class. Our job is to give a procedure for computing the derivatives which we could translate into NumPy code. The following does that: \marginnote{The second step of this derivation uses Eqn.~\ref{eqn:sigmoid-derivative}.}
\begin{equation}
\begin{aligned}
\frac{\deriv \crossEntropyLoss}{\deriv \prediction} &= -\frac{t}{y} + \frac{1-t}{1-y} \\
\frac{\deriv \crossEntropyLoss}{\deriv \intermediate} &= \frac{\deriv \crossEntropyLoss}{\deriv \prediction} \frac{\deriv \prediction}{\deriv \intermediate} \\
&= \frac{\deriv \crossEntropyLoss}{\deriv \prediction} \cdot y(1-y) \\
\frac{\partial \crossEntropyLoss}{\partial \weightJ{\dimIdx}} &= \frac{\deriv \crossEntropyLoss}{\deriv \intermediate} \frac{\partial \intermediate}{\partial \weightJ{\dimIdx}} \\
&= \frac{\deriv \crossEntropyLoss}{\deriv \intermediate} \cdot \inputJ{\dimIdx}
\end{aligned}
\label{eqn:naive-derivative}
\end{equation}

This can be translated directly into NumPy (exercise: how do you vectorize this?). If we were good little computer scientists, we would stop here. But today we're going to be naughty computer scientists and break the abstraction barrier between the activation function (logistic) and the cost function (cross-entropy).

\subsection{Logistic-cross-entropy function}

There's a big problem with Eqns.~\ref{eqn:naive-logistic-regression} and \ref{eqn:naive-derivative}: what happens if we have a positive example ($\target = 1$), but we confidently classify it as a negative example ($\intermediate \ll 0$, implying $\prediction \approx 0$)?  This is likely to happen at the very beginning of training, so we should be able to handle it. But if $\prediction$ is small enough, it could be smaller than the smallest floating point value, i.e.~{\bf numerically zero}. Then when we compute the cross-entropy, we take the log of 0 and get $-\infty$. Or if this doesn't happen, think about Eqn.~\ref{eqn:naive-derivative}. Since $\prediction$ appears in the denominator, $\deriv \crossEntropyLoss / \deriv \prediction$ will be extremely large in magnitude, which again can cause numerical difficulties. These bugs are very subtle, and can be hard to track down if you don't expect them.

What we do instead is combine the logistic function and cross-entropy loss into a single function, which we term {\bf logistic-cross-entropy}:
\begin{equation}
\logisticCrossEntropyLoss(\intermediate, \target) = \crossEntropyLoss(\logistic(\intermediate), \target) = \target \log (1 + e^{-\intermediate}) + (1-\target) \log (1 + e^{\intermediate})
\end{equation}
This still isn't numerically stable if we implement it directly, since $e^\intermediate$ could blow up. But most scientific computing environments provide a numerically stable {\bf log-sum-exp} routine.\footnote{The log-sum-exp trick is pretty neat. \url{https://hips.seas.harvard.edu/blog/2013/01/09/computing-log-sum-exp/}} In numpy, this is \verb+np.logaddexp+. So the following code would compute the logistic-cross-entropy:
\begin{verbatim}
E = t * np.logaddexp(0, -z) + (1-t) * np.logaddexp(0, z)
\end{verbatim}

Now to compute the loss derivatives:
\begin{equation}
\begin{aligned}
\frac{\deriv \logisticCrossEntropyLoss}{\deriv \intermediate} &= \frac{\deriv}{\deriv \intermediate} \left[ \target \log (1 + e^{-\intermediate}) + (1-\target) \log (1 + e^{\intermediate}) \right] \\
&= -\target \cdot \frac{e^{-\intermediate}}{1+e^{-\intermediate}} + (1-\target) \frac{e^\intermediate}{1 + e^{\intermediate}} \\
&= -\target (1-\prediction) + (1-\target)\prediction \\
&= \prediction - \target
\end{aligned}
\end{equation}
This is like magic!\marginnote{This isn't a coincidence. The reason it happens is beyond the scope of this course, but if you're curious, look up ``generalized linear models.''}  We took a somewhat complicated formula for the logistic activation function, combined it with a somewhat complicated formula for the cross-entropy loss, and wound up with a stunningly simple formula for the loss derivative!  Observe that this is exactly the same formula as for $\deriv \squaredErrorLoss / \deriv \prediction$ in the case of linear regression. And it has the same intuitive interpretation: if $\prediction > \target$, you made too positive a prediction, so you want to shift your prediction in the negative direction. Conversely, if $\prediction < \target$, you want to shift your prediction in the positive direction.


\subsection{Another alternative: hinge loss}

\begin{figure}
\begin{center}
\includegraphics[width=0.7 \textwidth]{figures/loss_comparison_with_hinge.png}
\end{center}
\caption{Comparison of the loss functions considered so far.}
\label{fig:loss-function-comparison}
\end{figure}

Another loss function you might encounter is {\bf hinge loss}. Here, $\prediction$ is a real value, and $\target \in \{-1, 1\}$.
\[ \hingeLoss(\prediction, \target) = \max(0, 1 - \target \prediction) \]
Hinge loss is plotted in Figure \ref{fig:loss-function-comparison} for a positive example. One useful property of hinge loss is that it's an upper bound on 0--1 loss; this is a useful property for a surrogate loss function, since it means that if you make the hinge loss small, you've also made 0--1 loss small. A linear model with hinge loss is known as a {\bf support vector machine (SVM)}:
\begin{align}
\prediction &= \weights^\transpose \inputVec + \bias \\
\hingeLoss &= \max(0, 1 - \target \prediction)
\end{align}
If you take CSC411, you'll learn a lot about SVMs, including their statistical motivation, how to optimize them efficiently and how to make them nonlinear (using something called the ``kernel trick''). But you already know one optimization method: you already know enough to derive the gradient descent updates for an SVM.

Interestingly, even though SVMs came from a different community and had a different sort of motivation from logistic regression, the algorithms behave very similarly in practice. The reason has to do with the loss functions. Figure~\ref{fig:loss-function-comparison} compares hinge loss to cross-entropy loss; even though cross-entropy is smoother, the asymptotic behavior is the same, suggesting the loss functions are basically pretty similar.

All of the loss functions covered so far is shown in Figure~\ref{fig:loss-function-comparison}. Take the time to review them, to understand their strengths and weaknesses.


\section{Multiclass classification}

So far we've talked about binary classification, but most classification problems involve more than two categories. Fortunately, this doesn't require any new ideas: everything pretty much works by analogy with the binary case. The first question is how to represent the targets. We could represent them as integers, but it's more convenient to use a {\bf one-hot vector}, also called a {\bf one-of-K encoding}:
\begin{equation}
\targets = \underbrace{ \left( 0, \ldots, 0, 1, 0, \ldots, 0 \right)}_{\textrm{entry $k$ is 1}}
\end{equation}
Now let's design the whole model by analogy with the binary case. 

First of all, consider the linear part of the model. We have $\numClasses$ outputs and $\nfeat$ inputs. To represent a linear function, we'll need a $\numClasses \times \nfeat$ {\bf weight matrix}, as well as a $\numClasses$-dimensional {\bf bias vector}. We first compute the intermediate quantities as follows:
\begin{equation}
\intermediates = \weightMat \inputVec + \biases.
\end{equation}
This is the general form of a linear function from $\mathbb{R}^\nfeat$ to $\mathbb{R}^\numClasses$.

Next, the activation function. We saw that the logistic function was a good thing to use in the binary case. There's a multivariate generalization of the logistic function called the {\bf softmax function}:\marginnote{Try plugging in $\numClasses=2$ to figure out how the softmax relates to the logistic function.}
\begin{equation}
\predictionK{\classIdx} = \softmax(\intermediateK{1}, \ldots, \intermediateK{\numClasses})_{\classIdx} = \frac{e^{\intermediateK{\classIdx}}}{\sum_{\classIdx^\prime} e^{\intermediateK{\classIdx^\prime}}}
\end{equation}
Importantly, the outputs of the softmax function are nonnegative and sum to 1, so they can be interpreted as a probability distribution over the $\numClasses$ classes (just like the output of the logistic could be interpreted as a probability). The inputs to the softmax are called the {\bf logits}.\marginnote{Think about the logits as the ``log-odds'', because when you exponentiate them you get the odds ratios of the probabilities.} Note that when one of the $\intermediateK{\classIdx}$'s is much larger than the others, the output of the softmax will be approximately the argmax, in the one-hot encoding. Hence, a more accurate name might be ``soft-argmax.''

Finally, the loss function. Cross-entropy can be generalized to the multiple-output case:\marginnote{You'll sometimes see $\sigmoid(\intermediates)$ used to denote the softmax function, by analogy with the logistic. But in this course, it will always denote the logistic function.}
\begin{align*}
\crossEntropyLoss(\predictions, \targets) &= -\sum_{\classIdx=1}^\numClasses \targetK{\classIdx} \log \predictionK{\classIdx} \\
&= -\targets^\transpose (\log \predictions).
\end{align*}
Here, $\log \predictions$ represents the elementwise $\log$. Note that only one of the $\targetK{\classIdx}$'s is 1 and the rest are 0, so the summation has the effect of picking the relevant entry of the vector $\log \predictions$. (See how convenient the one-hot notation is?)\marginnote{Try plugging in $\numClasses=2$ to see how this relates to binary cross-entropy.} Note that this loss function only makes sense for predictions which sum to 1; if you eliminate that constraint, you could trivially minimize the loss by making all the $\predictionK{\classIdx}$'s large.

When we put these things together, we get {\bf multiclass logistic regression}, or {\bf softmax regression}:
\begin{align*}
\intermediates &= \weightMat \inputVec + \biases \\
\predictions &= \softmax(\intermediates) \\
\crossEntropyLoss &= -\targets^\transpose (\log \predictions)
\end{align*}
We won't go through the derivatives in detail, but it basically works out exactly like logistic regression. The softmax and cross-entropy functions interact nicely with each other, so we always combine them into a single {\bf softmax-cross-entropy} function $\softmaxCrossEntropyLoss$ for purposes of numerical stability. The derivatives of $\softmaxCrossEntropyLoss$ have the same elegant formula we've been seeing repeatedly, except this time remember that $\targets$ and $\predictions$ are both vectors:
\begin{equation}
\frac{\partial \softmaxCrossEntropyLoss}{\partial \intermediates} = \predictions - \targets
\end{equation}
Softmax regression is an elegant learning algorithm which can work very well in practice.



%You already know enough to derive the gradient descent update rules. All of the analysis we went through comparing various activations and loss functions for the binary case carries over to multiple outputs. I.e., the softmax activation function and cross-entropy loss interact nicely, giving an elegant learning algorithm which works very well in practice.


\section{Convex Functions}

\begin{figure}
\includegraphics[width=0.48 \textwidth]{figures/convexity.png}
\includegraphics[width=0.48 \textwidth]{figures/convexity_proof.png}
\caption{{\bf Left:} Definition of convexity. {\bf Right:} Proof-by-picture that if the model is linear and $\loss$ is a convex function of $\intermediate = \weights^\transpose \inputVec + \bias$, then it's also convex as a function of $\weights$ and $\bias$.}
\label{fig:convexity}
\end{figure}

An important criterion we often use to compare different loss functions is convexity. Recall that a set $\mathcal{S}$ is convex if the line segment connecting any two points in $\mathcal{S}$ lies entirely within $\mathcal{S}$. Mathematically, this means that for $\inputVec_0, \inputVec_1 \in \mathcal{S}$,
\[ (1 - \lambda) \mathbf{x}_0 + \lambda \mathbf{x}_1 \in \mathcal{S} \quad \textrm{for } 0 \leq \lambda \leq 1. \]
The definition of a {\bf convex function} is closely related. A function $f$ is convex if for any $\mathbf{x}_0, \mathbf{x}_1$ in the domain of $f$,
\begin{equation}
f((1 - \lambda) \mathbf{x}_0 + \lambda \mathbf{x}_1) \leq (1 - \lambda) f(\mathbf{x}_0) + \lambda f(\mathbf{x}_1)
\end{equation}
This is shown graphically in Figure~\ref{fig:convexity}. Another way to phrase this requirement is that the line segment connecting any two points on the graph of $f$ must lie above the graph of $f$. Equivalently, the set of points lying above the graph of $f$ must be a convex set. Intuitively, convex functions are bowl-shaped.

Convexity is a really important property from the standpoint of optimization. There are two main reasons for this:
\begin{enumerate}
\item All critical points are global minima, so if you can set the derivatives to zero, you've solved the problem.
\item Gradient descent will always converge to the global minimum.
\end{enumerate}
We'll talk in more detail in a later lecture about what can go wrong when the cost function is not convex. Look back at our comparison of loss functions in Figure~\ref{fig:loss-function-comparison}. You can see visually that squared error, hinge loss, and the logistic regression objective are all convex; 0--1 loss, and logistic-with-least-squares are not convex. It's not a coincidence that the loss functions we might actually try to optimize are the convex ones. There is an entire field of research on convex optimization, which comes up with better ways to minimize convex functions over convex sets, as well as ways to formulate various kinds of problems in terms of convex optimization.

Note that even though convexity is important, most of the optimization problems we'll consider in this course will be non-convex, because training a deep neural network is a non-convex problem, even when the loss function is convex. Nonetheless, convex loss functions somehow still tend to be advantageous from the standpoint of optimization.

\section{Gradient Checking with Finite Differences}

We've derived a lot of gradients so far. How do we know if they're correct?  Fortunately, there's an easy and effective procedure for testing them. Recall the definition of the partial derivative:
\begin{equation}
\frac{\partial}{\partial x_i} f(x_1, \ldots, x_N) = \lim_{h \to 0} \frac{f(x_1, \ldots, x_i + h, \ldots, x_N) - f(x_1, \ldots, x_i, \ldots, x_N)}{h}
\end{equation}
We can check the derivatives numerically by plugging in a small value of $h$, such as $10^{-10}$. This is known as the method of {\bf finite differences}.\marginnote{You don't want to implement your actual learning algorithm using finite differences, because it's very slow, but it's great for testing.} 

\begin{figure}
\begin{center}
\includegraphics[width=0.5 \textwidth]{figures/two_sided.png}
\end{center}
\caption{Estimating a derivative using one-sided and two-sided finite differences.}
\label{fig:finite-differences}
\end{figure}

It's actually better to use the two-sided definition of the partial derivative than the one-sided one, since it is much more accurate:
\begin{equation}
\frac{\partial}{\partial x_i} f(x_1, \ldots, x_N) = \lim_{h \to 0} \frac{f(x_1, \ldots, x_i + h, \ldots, x_N) - f(x_1, \ldots, x_i - h, \ldots, x_N)}{2h}
\end{equation}
An example is shown in Figure~\ref{fig:finite-differences} of estimating a derivative using the one-sided and two-sided formulas.

Gradient checking is really important!  In machine learning, your algorithm can often seem to learn well even if the gradient calculation is totally wrong. This might lead you to skip the correctness checks. But it might work even better if the derivatives are correct, and this is important when you're trying to squeeze out the last bit of accuracy. Wrong derivatives can also lead you on wild goose chases, as you make changes to your system which appear to help significantly, but actually are only helping because they compensate for errors in the gradient calculations. If you implement derivatives by hand, gradient checking is the single most important thing you need to do to get your algorithm to work well.



\end{document}










