\documentclass[11pt]{article}
%\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
%\geometry{landscape}                % Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{epstopdf}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{hyperref}
\usepackage{color}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

% margin notes
\usepackage{marginnote}
\usepackage[top=2cm, bottom=2cm, outer=7cm, inner=2cm, heightrounded, marginparwidth=5cm, marginparsep=1cm]{geometry}
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
\renewcommand*{\marginfont}{\footnotesize}

\input{../../misc/macros.tex}



% \newcommand{\TODO}[1]{{\color{red} \bf [[TODO: #1]]}}

% \newcommand{\zeroVec}{\mathbf{0}}
% \newcommand{\onesVec}{\mathbf{1}}
% \newcommand{\deriv}{\mathrm{d}}
% \newcommand{\transpose}{\top}

% \newcommand{\dataIdx}{i}
% \newcommand{\featIdx}{j}
% \newcommand{\hidIdx}{i}
% \newcommand{\featIdxTwo}{j^\prime}
% \newcommand{\nfeat}{D}
% \newcommand{\ndata}{N}

% \newcommand{\inputIJ}[2]{x^{(#1)}_{#2}}
% \newcommand{\inputI}[1]{{\bf x}^{(#1)}}
% \newcommand{\inputJ}[1]{x_{#1}}
% \newcommand{\inputVec}{{\bf x}}
% \newcommand{\inputUni}{x}
% \newcommand{\inputMatrix}{\mathbf{X}}
% \newcommand{\targetI}[1]{y^{(#1)}}
% \newcommand{\target}{y}
% \newcommand{\targets}{\mathbf{y}}
% \newcommand{\prediction}{\hat{y}}
% \newcommand{\predictionI}[1]{\hat{y}^{(#1)}}
% \newcommand{\predictions}{\hat{\mathbf{y}}}
% \newcommand{\intermediate}{z}
% \newcommand{\hiddenI}[1]{h_{#1}}
% \newcommand{\hiddens}{\mathbf{h}}
% \newcommand{\weights}{{\bf w}}
% \newcommand{\weightJ}[1]{w_{#1}}
% \newcommand{\weightLIJ}[3]{w^{(#1)}_{#2 #3}}
% \newcommand{\weightLJ}[2]{w^{(#1)}_{#2}}
% \newcommand{\weightUni}{w}
% \newcommand{\bias}{b}
% \newcommand{\biasLI}[2]{\bias^{(#1)}_{#2}}
% \newcommand{\biasL}[1]{\bias^{(#1)}}
% \newcommand{\threshold}{r}
% \newcommand{\featureJ}[1]{\phi_{#1}}
% \newcommand{\featureVec}{{\boldsymbol \phi}}
% \newcommand{\loss}{\mathcal{L}}
% \newcommand{\zeroOneLoss}{\loss_{\rm 0-1}}
% \newcommand{\squaredErrorLoss}{\loss_{\rm SE}}
% \newcommand{\crossEntropyLoss}{\loss_{\rm CE}}
% \newcommand{\cost}{\mathcal{E}}
% \newcommand{\lrate}{\alpha}
% \newcommand{\featureMap}{{\boldsymbol \phi}}
% \newcommand{\sigmoid}{\sigma}
% \newcommand{\activationFunction}{\phi}
% \newcommand{\activationFunctionL}[1]{\activationFunction^{(#1)}}
% \newcommand{\parityFunction}{f_{\rm par}}


\title{Lecture 2, Part 1: Multilayer Perceptrons}
\author{Roger Grosse}
\date{}

\begin{document}
\maketitle

\section{Introduction}

So far, we've only talked about linear models: linear regression and linear binary classifiers. We noted that there are functions that can't be represented by linear models; for instance, linear regression can't represent quadratic functions, and linear classifiers can't represent XOR. We also saw one particular way around this issue: by defining features, or basis functions. E.g., linear regression can represent a cubic polynomial if we use the feature map $\featureMap(\inputUni) = (1, \inputUni, \inputUni^2, \inputUni^3)$. We also observed that this isn't a very satisfying solution, for two reasons:
\begin{enumerate}
\item The features need to be specified in advance, and this can require a lot of engineering work.
\item It might require a very large number of features to represent a certain set of functions; e.g.~the feature representation for cubic polynomials is cubic in the number of input features.
\end{enumerate}

In this lecture, and for the rest of the course, we'll take a different approach.\marginnote{Some people would claim that the methods covered in this course are really ``just'' adaptive basis function representations. I've never found this a very useful way of looking at things.} We'll represent complex nonlinear functions by connecting together lots of simple processing units into a \emph{neural network}, each of which computes a linear function, possibly followed by a nonlinearity. In aggregate, these units can compute some surprisingly complex functions. By historical accident, these networks are called \emph{multilayer perceptrons}.


\subsection{Learning Goals}

\begin{itemize}
\item Know the basic terminology for neural nets
\item Given the weights and biases for a neural net, be able to compute its output from its input
\item Be able to hand-design the weights of a neural net to represent functions like XOR
\item Understand how a hard threshold can be approximated with a soft threshold
\item Understand why shallow neural nets are universal, and why this isn't necessarily very interesting
\end{itemize}


\section{Multilayer Perceptrons}

In the first lecture, we introduced our general neuron-like processing unit:
\[ a = \phi \left( \sum_j w_j x_j + b \right), \]
where the $x_j$ are the inputs to the unit, the $w_j$ are the weights, $b$ is the bias, $\phi$ is the nonlinear activation function, and $a$ is the unit's activation. We've seen a bunch of examples of such units:
\begin{itemize}
\item Linear regression uses a linear model, so $\phi(z) = z$.
\item In binary linear classifiers, $\phi$ is a hard threshold at zero.
\item In logistic regression, $\phi$ is the logistic function $\sigma(z) = 1/(1+e^{-z})$.
\end{itemize}
A {\bf neural network} is just a combination of lots of these units. Each one performs a very simple and stereotyped function, but in aggregate they can do some very useful computations. For now, we'll concern ourselves with {\bf feed-forward neural networks}, where the units are arranged into a graph without any cycles, so that all the computation can be done sequentially. This is in contrast with {\bf recurrent neural networks}, where the graph can have cycles, so the processing can feed into itself. These are much more complicated, and we'll cover them later in the course.

\begin{figure}
\begin{center}
\includegraphics[width=0.5 \textwidth]{figures/mlp.png}
\hspace{0.05 \textwidth}
\includegraphics[width=0.4 \textwidth]{figures/network_as_features.png}
\end{center}
\caption{A multilayer perceptron with two hidden layers. {\bf Left:} with the units written out explicitly. {\bf Right:} representing layers as boxes.}
\label{fig:mlp}
\end{figure}


The simplest kind of feed-forward network is a {\bf multilayer perceptron (MLP)}, as shown in Figure~\ref{fig:mlp}.\marginnote{MLP is an unfortunate name. The \emph{perceptron} was a particular algorithm for binary classification, invented in the 1950s. Most multilayer perceptrons have very little to do with the original perceptron algorithm.} Here, the units are arranged into a set of {\bf layers}, and each layer contains some number of identical units. Every unit in one layer is connected to every unit in the next layer; we say that the network is {\bf fully connected}. The first layer is the {\bf input layer}, and its units take the values of the input features. The last layer is the {\bf output layer}, and it has one unit for each value the network outputs (i.e.~a single unit in the case of regression or binary classifiation, or $K$ units in the case of $K$-class classification). All the layers in between these are known as {\bf hidden layers}, because we don't know ahead of time what these units should compute, and this needs to be discovered during learning. The units in these layers are known as {\bf input units}, {\bf output units}, and {\bf hidden units}, respectively. The number of layers is known as the {\bf depth}, and the number of units in a layer is known as the {\bf width}.\marginnote{Terminology for the depth is very inconsistent. A network with one hidden layer could be called a one-layer, two-layer, or three-layer network, depending if you count the input and output layers.} As you might guess, ``deep learning'' refers to training neural nets with many layers.




%As an example to illustrate the power of MLPs, let's design one that computes the XOR function. Recall that we showed linear classifiers were incapable of representing XOR. We did, however, design linear classifiers to compute the simple functions AND, OR, and NOT. The units are shown in \TODO{Figure} It's pretty easy to build a circuit for XOR out of AND, OR, and NOT gates. We can combine together our simple networks in exactly the same way to compute the XOR function. This is shown in \TODO{Figure}. This network can be simplified slightly to the one shown in \TODO{Figure}.

As an example to illustrate the power of MLPs, let's design one that computes the XOR function. Remember, we showed that linear models cannot do this. We can verbally describe XOR as ``one of the inputs is 1, but not both of them.'' So let's have hidden unit $h_1$ detect if at least one of the inputs is 1, and have $h_2$ detect if they are both 1. We can easily do this if we use a hard threshold activation function. You know how to design such units --- it's an exercise of designing a binary linear classifier. Then the output unit will activate only if $h_1 = 1$ and $h_2 = 0$. A network which does this is shown in Figure~\ref{fig:xor}.

\begin{figure}
\begin{center}
\includegraphics[width=0.35 \textwidth]{figures/xor_completed.png}
\end{center}
\caption{An MLP that computes the XOR function. All activation functions are binary thresholds at 0.}
\label{fig:xor}
\end{figure}


Let's write out the MLP computations mathematically. Conceptually, there's nothing new here; we just have to pick a notation to refer to various parts of the network. As with the linear case, we'll refer to the activations of the input units as $\inputJ{\featIdx}$ and the activation of the output unit as $\prediction$. The units in the $\layerIdx$th hidden layer will be denoted $\hiddenLI{\layerIdx}{\hidIdx}$. Our network is fully connected, so each unit receives connections from all the units in the previous layer. This means each unit has its own bias, and there's a weight for every \emph{pair} of units in two consecutive layers. Therefore, the network's computations can be written out as:
\begin{equation}
\begin{aligned}
\hiddenLI{1}{\hidIdx} &= \activationFunctionL{1} \left( \sum_{\featIdx} \weightLIJ{1}{\hidIdx}{\featIdx} \inputJ{\featIdx} + \biasLI{1}{\hidIdx} \right) \\
\hiddenLI{2}{\hidIdx} &= \activationFunctionL{2} \left( \sum_{\featIdx} \weightLIJ{2}{\hidIdx}{\featIdx} \hiddenLI{1}{\featIdx} + \biasLI{2}{\hidIdx} \right) \\
\predictionK{\hidIdx} &= \activationFunctionL{3} \left( \sum_{\featIdx} \weightLIJ{3}{\hidIdx}{\featIdx} \hiddenLI{2}{\featIdx} + \biasLI{3}{\hidIdx} \right)
\end{aligned}
\end{equation}
Note that we distinguish $\activationFunctionL{1}$ and $\activationFunctionL{2}$ because different layers may have different activation functions. 

Since all these summations and indices can be cumbersome, we usually write the computations in vectorized form. Since each layer contains multiple units, we represent the activations of all its units with an {\bf activation vector} $\hiddensL{\layerIdx}$. Since there is a weight for every pair of units in two consecutive layers, we represent each layer's weights with a {\bf weight matrix} $\weightMatL{\layerIdx}$. Each layer also has a {\bf bias vector} $\biasesL{\layerIdx}$. The above computations are therefore written in vectorized form as:
\begin{equation}
\begin{aligned}
\hiddensL{1} &= \activationFunctionL{1} \left( \weightMatL{1} \inputVec + \biasesL{1} \right) \\
\hiddensL{2} &= \activationFunctionL{2} \left( \weightMatL{2} \hiddensL{1} + \biasesL{2} \right) \\
\predictions &= \activationFunctionL{3} \left( \weightMatL{3} \hiddensL{2} + \biasesL{3} \right) \\
\end{aligned}
\end{equation}
When we write the activation function applied to a vector, this means it's applied independently to all the entries.

Recall how in linear regression, we combined all the training examples into a single matrix $\inputMatrix$, so that we could compute all the predictions using a single matrix multiplication. We can do the same thing here. We can store all of each layer's hidden units for all the training examples as a matrix $\hiddenMatL{\layerIdx}$. Each row contains the hidden units for one example. The computations are written as follows (note the transposes):\marginnote{If it's hard to remember when a matrix or vector is transposed, fear not. You can usually figure it out by making sure the dimensions match up.}
\begin{equation}
\begin{aligned}
\hiddenMatL{1} &= \activationFunctionL{1} \left( \inputMatrix \weightMat^{(1) \transpose} + \onesVec \biases^{(1) \transpose} \right) \\
\hiddenMatL{2} &= \activationFunctionL{2} \left( \hiddenMatL{1} \weightMat^{(2) \transpose} + \onesVec \biases^{(2) \transpose} \right) \\
\predictionMatrix &= \activationFunctionL{3} \left( \hiddenMatL{2} \weightMat^{(3) \transpose} + \onesVec \biases^{(3) \transpose} \right) \\
\end{aligned}
\end{equation}
These equations can be translated directly into NumPy code which efficiently computes the predictions over the whole dataset.


\section{Feature Learning}

We already saw that linear regression could be made more powerful using a feature mapping. For instance, the feature mapping $\featureMap(x) = (1, x, x^2, x^e)$ can represent third-degree polynomials. But static feature mappings were limited because it can be hard to design all the relevant features, and because the mappings might be impractically large. Neural nets can be thought of as a way of learning nonlinear feature mappings. E.g., in Figure \ref{fig:mlp}, the last hidden layer can be thought of as a feature map $\featureMap(\inputVec)$, and the output layer weights can be thought of as a linear model using those features. But the whole thing can be trained end-to-end with backpropagation, which we'll cover in the next lecture. The hope is that we can learn a feature representation where the data become linearly separable:

\begin{center}
\includegraphics[width=0.6 \textwidth]{figures/features_cartoon.png}
\end{center}

\begin{figure}
\begin{center}
\includegraphics[width=0.5 \textwidth]{figures/mnist.png}
\includegraphics[width=0.4 \textwidth]{figures/mnist_filters.png}
\end{center}
\caption{{\bf Left:} Some training examples from the MNIST handwritten digit dataset. Each input is a $28 \times 28$ grayscale image, which we treat as a 784-dimensional vector. {\bf Right:} A subset of the learned first-layer features. Observe that many of them pick up oriented edges.}
\label{fig:mnist}
\end{figure}

Consider training an MLP to recognize handwritten digits. (This will be a running example for much of the course.) The input is a $28 \times 28$ grayscale image, and all the pixels take values between 0 and 1. We'll ignore the spatial structure, and treat each input as a 784-dimensional vector.\marginnote{Later on, we'll talk about convolutional networks, which use the spatial structure of the image.} This is a multiway classification task with 10 categories, one for each digit class. Suppose we train an MLP with two hidden layers. We can try to understand what the first layer of hidden units is computing by visualizing the weights. Each hidden unit receives inputs from each of the pixels, which means the weights feeding into each hidden unit can be represented as a 784-dimensional vector, the same as the input size. In Figure \ref{fig:mnist}, we display these vectors as images.

In this visualization, positive values are lighter, and negative values are darker. Each hidden unit computes the dot product of these vectors with the input image, and then passes the result through the activation function. So if the light regions of the filter overlap the light regions of the image, and the dark regions of the filter overlap the dark region of the image, then the unit will activate. E.g., look at the third filter in the second row. This corresponds to an {\bf oriented edge}: it detects vertical edges in the upper right part of the image. This is a useful sort of feature, since it gives information about the locations and orientation of strokes. Many of the features are similar to this; in fact, oriented edges are a very commonly learned by the first layers of neural nets for visual processing tasks.

It's harder to visualize what the second layer is doing. We'll see some tricks for visualizing this in a few weeks. We'll see that higher layers of a neural net can learn increasingly high-level and complex features.



\section{Expressive Power}

Linear models are fundamentally limited in their expressive power: they can't represent functions like XOR. Are there similar limitations for MLPs?  It depends on the activation function.

\subsection{Linear networks}

Deep linear networks are no more powerful than shallow ones. The reason is simple: if we use the linear activation function $\activationFunction(x) = x$ (and forget the biases for simplicity), the network's function can be expanded out as $\predictions = \weightMatL{\numLayers} \weightMatL{\numLayers-1} \cdots \weightMatL{1} \inputVec$. But this could be viewed as a single linear layer with weights given by $\weightMat = \weightMatL{\numLayers} \weightMatL{\numLayers-1} \cdots \weightMatL{1}$. Therefore, a deep linear network is no more powerful than a single linear layer, i.e.~a linear model.


\subsection{Universality}

As it turns out, nonlinear activation functions give us much more power: under certain technical conditions, even a shallow MLP (i.e.~one with a single hidden layer) can represent arbitrary functions. Therefore, we say it is {\bf universal}. 

Let's demonstrate universality in the case of binary inputs.\marginnote{This argument can easily be made into a rigorous proof, but this course won't be concerned with mathematical rigor.}  We do this using the following game: suppose we're given a function mapping input vectors to outputs; we will need to produce a neural network (i.e.~specify the weights and biases) which matches that function. The function can be given to us as a table which lists the output corresponding to every possible input vector. If there are $\nfeat$ inputs, this table will have $2^\nfeat$ rows. An example is shown in Figure \ref{fig:universal}. For convenience, let's suppose these inputs are $\pm 1$, rather than 0 or 1. All of our hidden units will use a hard threshold at 0 (but we'll see shortly that these can easily be converted to soft thresholds), and the output unit will be linear.

\begin{figure}
\begin{center}
\includegraphics[width=0.7\textwidth]{figures/universal.png}
\end{center}
\caption{Designing a binary threshold network to compute a particular function.}
\label{fig:universal}
\end{figure}

Our strategy will be as follows: we will have $2^\nfeat$ hidden units, each of which recognizes one possible input vector. We can then specify the function by specifying the weights connecting each of these hidden units to the outputs. For instance, suppose we want a hidden unit to recognize the input $(-1, 1, -1)$. This can be done using the weights $(-1, 1, -1)$ and bias $-2.5$, and this unit will be connected to the output unit with weight 1. (Can you come up with the general rule?) Using these weights, any input pattern will produce a set of hidden activations where exactly one of the units is active. The weights connecting inputs to outputs can be set based on the input-output table. Part of the network is shown in Figure \ref{fig:universal}.

Universality is a neat property, but it has a major catch: the network required to represent a given function might have to be extremely large (in particular, exponential). In other words, not all functions can be represented {\bf compactly}. We desire compact representations for two reasons:
\begin{enumerate}
\item We want to be able to compute predictions in a reasonable amount of time.
\item We want to be able to train a network to \emph{generalize} from a limited number of training examples; from this perspective, universality simply implies that a large enough network can memorize the training set, which isn't very interesting.
\end{enumerate}

\subsection{Soft thresholds}

In the previous section, our activation function was a step function, which gives a hard threshold at 0. This was convenient for designing the weights of a network by hand. But recall from last lecture that it's very hard to directly learn a linear classifier with a hard threshold, because the loss derivatives are 0 almost everywhere. The same holds true for multilayer perceptrons. If the activation function for any unit is a hard threshold, we won't be able to learn that unit's weights using gradient descent. The solution is the same as it was in last lecture: we replace the hard threshold with a soft one.

Does this cost us anything in terms of the network's expressive power?  No it doesn't, because we can approximate a hard threshold using a soft threshold. In particular, if we use the logistic nonlinearity, we can approximate a hard threshold by scaling up the weights and biases:
\begin{center}
\includegraphics[width=0.75 \textwidth]{figures/soft_threshold.png}
\end{center}


\subsection{The power of depth}

If shallow networks are universal, why do we need deep ones?  One important reason is that deep nets can represent some functions more \emph{compactly} than shallow ones. For instance, consider the parity function (on binary-valued inputs):
\begin{equation}
\parityFunction(\inputJ{1}, \ldots, \inputJ{\nfeat}) = \left\{ \begin{array}{ll} 1 & \textrm{if $\sum_\featIdx \inputJ{\featIdx}$ is odd} \\ 0 & \textrm{if it is even.} \end{array} \right.
\end{equation}
We won't prove this, but it requires an exponentially large shallow network to represent the parity function. On the other hand, it can be computed by a deep network whose size is \emph{linear} in the number of inputs. Designing such a network is a good exercise.


\end{document}










