\documentclass[11pt]{article}
%\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
%\geometry{landscape}                % Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{epstopdf}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{hyperref}
\usepackage{color}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

\newtheorem{example}{Example}

% margin notes
\usepackage{marginnote}
\usepackage[top=2cm, bottom=2cm, outer=7cm, inner=2cm, heightrounded, marginparwidth=5cm, marginparsep=1cm]{geometry}
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ... 
\renewcommand*{\marginfont}{\footnotesize}

\input{../../misc/macros.tex}


\title{Lecture 1, Part 2: Linear Classification}
\author{Roger Grosse}
\date{}                                           % Activate to display a given date or no date

\begin{document}
\maketitle

\section{Introduction}

Last time, we saw an example of a learning task called regression. There, the goal was to predict a scalar-valued target from a set of features. This time, we'll focus on a slightly different task: {\bf binary classification}, where the goal is to predict a \emph{binary-valued} target. Here are some examples of binary classification problems:
\begin{itemize}
\item You want to train a medical diagnosis system to predict whether a patient has a given disease. You have a training set consisting of a set of patients, a set of features for those individuals (e.g.~presence or absence of various symptoms), and a label saying whether or not the patient had the disease.
\item You are running an e-mail service, and want to determine whether a given e-mail is spam. You have a large collection of e-mails which have been hand-labeled as spam or non-spam.
\item You are running an online payment service, and want to determine whether or not a given transaction is fraudulent. You have a labeled training dataset of fraudulent and non-fraudulent transactions; features might include the type of transaction, the amount of money, or the time of day.
\end{itemize}

Like regression, binary classification is a very restricted kind of task. Most learning problems you'll encounter won't fit nicely into one of these two categories. Our motivation for focusing on binary classification is to introduce several fundamental ideas that we'll use throughout the course. In this lecture, we discuss how to view both data points and linear classifiers as vectors. Next lecture, we discuss the perceptron, a particular classification algorithm, and use it as an example of how to efficiently implement a learning algorithm in Python. Starting next week, we'll look at supervised learning in full generality, and see that regression and binary classification are just special cases of a more general learning framework.

This lecture focuses on the geometry of classification. We'll look in particular at two spaces:
\begin{itemize}
\item The input space, where each data case corresponds to a vector. A classifier corresponds to a decision boundary, or a hyperplane such that the positive examples lie on one side, and negative examples lie on the other side.
\item Weight space, where each set of classification weights corresponds to a vector. Each training case corresponds to a constraint in this space, where some regions of weight space are ``good'' (classify it correctly) and some regions are ``bad'' (classify it incorrectly).
\end{itemize}
The idea of weight space may seem pretty abstract, but it is very important that you become comfortable with it, since it underlies nearly everything we do in the course.

Using our understanding of input space and weight space, the limitations of linear classifiers will become immediately apparent. We'll see some examples of datasets which are not linearly separable (i.e.~no linear classifier can correctly classify all the training cases), but which become linearly separable if we use a basis function representation.


\subsection{Learning goals}

\begin{itemize}
\item Know what is meant by binary linear classification.
\item Understand why an explicit threshold for a classifier is redundant. Understand how we can get rid of the bias term by adding a ``dummy'' feature.
\item Be able to specify weights and biases by hand to represent simple functions (\emph{e.g.}~AND, OR, NOT).
\item Be familiar with input space and weight space.
\begin{itemize}
\item Be able to plot training cases and classification weights in both input space and weight space.
\end{itemize}
\item Be aware of the limitations of linear classifiers.
\begin{itemize}
\item Know what is meant by convexity, and be able to use convexity to show that a given set of training cases is not linearly separable.
\item Understand how we can sometimes still separate the classes using a basis function representation.
\end{itemize}
\end{itemize}



%\subsection{Resources}

%\begin{itemize}
%\item This lecture corresponds roughly to Coursera lectures 2b, 2c, and 2e.
%\item Pretty much every machine learning course or textbook teaches at least one binary linear classifier (e.g.~perceptrons, SVM, logistic regression, linear discriminant analysis). Unfortunately, most of them take for granted most of the material in this lecture. You can find some relevant resources in ``binary linear classifiers,'' or any of the algorithms listed above, on Metacademy.
%\item Suggested background:
%\begin{itemize}
%\item We'll be plotting lots of linear inequalities. If you're not comfortable with this, you should review the lecture sequence on Khan Academy: \url{https://www.khanacademy.org/math/algebra/linear-equations-and-inequalitie/graphing-linear-inequalities/v/graphing-inequalities}
%\end{itemize}
%\end{itemize}


\section{Binary linear classifiers}

We'll be looking at classifiers which are both {\bf binary} (they distinguish between two categories) and {\bf linear} (the classification is done using a linear function of the inputs). As in our discussion of linear regression, we assume each input is given in terms of $\nfeat$ scalar values, called {\bf input dimensions} or {\bf features}, which we think summarize the important information for classification. (Some of the features, e.g.~presence or absence of a symptom, may in fact be binary valued, but we're going to treat these as real-valued anyway.) The $\featIdx$th feature for the $\dataIdx$th training example is denoted $\inputIJ{\dataIdx}{\featIdx}$. All of the features for a given training case are concatenated together to form a vector, which we'll denote $\inputI{\dataIdx}$. (Recall that vectors and matrices are shown in boldface.)

Associated with each data case is a binary-valued {\bf target}, the thing we're trying to predict. By definition, a binary target takes two possible values, which we'll call {\bf classes}, and which are typically referred to as {\bf positive} and {\bf negative}. (E.g., the positive class might be ``has disease'' and the negative class might be ``does not have disease.'') Data cases belonging to these classes are called {\bf positive examples} and {\bf negative examples}, respectively. The {\bf training set} consists of a set of $\ndata$ pairs $(\inputI{\dataIdx}, \targetI{\dataIdx})$, where $\inputI{\dataIdx}$ is the input and $\targetI{\dataIdx}$ is the binary-valued target, or {\bf label}. Since the training cases come with labels, they're referred to as {\bf labeled examples}. Confusingly, even though we talk about positive and negative examples, the $\targetI{\dataIdx}$ typically take values in $\{0, 1\}$, where 0 corresponds to the ``negative'' class. Sorry, you'll just have to live with this terminology.

Our goal is to correctly classify all the training cases (and, hopefully, examples not in the training set). In order to do the classification, we need to specify a {\bf model}, which determines how the predictions are computed from the inputs. As we said before, our model for this week is binary linear classifiers.

The way binary linear classifiers work is simple: they compute a linear function of the inputs, and determine whether or not the value is larger than some {\bf threshold} $\threshold$. Recall from Lecture 2 that a linear function of the input can be written as
\[ \weightJ{1} \inputJ{1} + \cdots + \weightJ{\nfeat} \inputJ{\nfeat} + \bias = \weights^T \inputVec + \bias, \]
 where $\weights$ is a {\bf weight vector} and $\bias$ is a scalar-valued {\bf bias}. Therefore, the prediction $\prediction$ can be computed as follows:
\begin{align*}
\intermediate &= \weights^T \inputVec + \bias \\
\prediction &= \left\{ \begin{array}{ll} 1 & \textrm{ if } \intermediate \geq \threshold \\ 0 & \textrm{ if } \intermediate < \threshold \end{array} \right.
\end{align*}
This is the model we'll use for the rest of the week.

\subsection{Thresholds and biases}
\label{sec:thresholds-and-biases}

Dealing with thresholds is rather inconvenient, but fortunately we can get rid of them entirely. In particular, observe that
\[ \weights^T \inputVec + \bias \geq \threshold \Longleftrightarrow \weights^T \inputVec + \bias - \threshold \geq 0. \]
In other words, we can obtain an equivalent model by replacing the bias with $\bias - \threshold$ and setting $\threshold$ to 0. From now on, we'll assume (without loss of generality) that the threshold is 0. Therefore, we rewrite the model as follows:
\begin{align*}
\intermediate &= \weights^T \inputVec + \bias \\
\prediction &= \left\{ \begin{array}{ll} 1 & \textrm{ if } \intermediate \geq 0 \\ 0 & \textrm{ if } \intermediate < 0 \end{array} \right.
\end{align*}

In fact, it's possible to eliminate the bias as well. We simply add another input dimension $\inputJ{0}$, called a {\bf dummy feature}, which always takes the value 1. Then
\[ \weightJ{0} \inputJ{0} + \weightJ{1} \inputJ{1} + \cdots + \weightJ{\nfeat} \inputJ{\nfeat} = \weightJ{0} + \weightJ{1} \inputJ{1} + \cdots + \weightJ{\nfeat} \inputJ{\nfeat}, \]
so $\weightJ{0}$ effectively plays the role of a bias. We can then simply write
\[ \intermediate = \weights^T \inputVec. \]
Eliminating the bias often simplifies the statements of algorithms, so we'll sometimes use it for notational convenience. However, you should be aware that, when actually implementing the algorithms, the standard practice is to include the bias parameter explicitly.

\subsection{Some examples}

Let's look at some examples of how to represent simple functions using linear classifiers --- specifically, AND, OR, and NOT.

\begin{quote}
\begin{example}
\label{example:not}
Let's start with NOT, since it only involves a single input. Here's a ``training set'' of inputs and targets we're trying to match:
\begin{center}
\begin{tabular}{c|c}
$\inputJ{1}$ & $\target$ \\
\hline
0 & 1 \\
1 & 0
\end{tabular}
\end{center}
Each of the training cases provides a constraint on the weights and biases. Let's start with the first training case. If $\inputJ{1} = 0$, then $\target = 1$, so we need $\intermediate = \weightJ{1} \inputJ{1} + \bias = \bias \geq 0$. Technically we could satisfy this constraint with $\bias = 0$, but it's good practice to avoid solutions where $\intermediate$ lies on the decision boundary. Therefore, let's tentatively set $\bias = 1$.

Now let's consider the second training case. The input is $\inputJ{1} = 1$ and the target is $\target = 0$, so we need $\intermediate = \weightJ{1} \cdot 1 + \bias = \weightJ{1} + 1 < 0$. We can satisfy this inequality with $\weightJ{1} = -2$. This gives us our solution: $\weightJ{1} = -2$, $\bias = 1$.
\end{example}

\begin{example}
\label{example:and}
Now let's consider AND. This is slightly more complicated, since we have 2 inputs and 4 training cases. The training cases are as follows:
\begin{center}
\begin{tabular}{cc|c}
$\inputJ{1}$ & $\inputJ{2}$ & $\target$ \\
\hline
0 & 0 & 0 \\
0 & 1 & 0 \\
1 & 0 & 0 \\
1 & 1 & 1
\end{tabular}
\end{center}
Just like in the previous example, we can start by writing out the inequalities corresponding to each training case. We get:
\begin{align*}
\bias &< 0 \\
\weightJ{2} + \bias &< 0 \\
\weightJ{1} + \bias &< 0 \\
\weightJ{1} + \weightJ{2} + \bias &> 0
\end{align*}
From these inequalities, we immediately see that $\bias < 0$ and $\weightJ{1}, \weightJ{2} > 0$. The simplest way forward at this point is probably trial and error. Since the problem is symmetric with respect to $\weightJ{1}$ and $\weightJ{2}$, we might as well decide that $\weightJ{1} = \weightJ{2}$. So let's try $\bias = -1, \weightJ{1} = \weightJ{2} = 1$ and see if it works. The first and fourth inequalities are clearly satisfied, but the second and third are not, since $\weightJ{1} + \bias = \weightJ{2} + \bias = 0$. So let's try making the bias a bit more negative. When we try $\bias = -1.5, \weightJ{1} = \weightJ{2} = 1$, we see that all four inequalities are satisfied, so we have our solution.
\end{example}
\end{quote}

Following these examples, you should attempt the OR function on your own.



\section{The geometric picture}

Now let's move on to the main concepts of this lecture: data space and weight space. These are the spaces that the inputs and the weight vectors live in, respectively. It's very important to become comfortable thinking about these spaces, since we're going to treat the inputs and weights as vectors for the rest of the term.

In this lecture, we're going to focus on two-dimensional input and weight spaces. But keep in mind that this is a vast oversimplification: in practical settings, these spaces are typically many thousands, or even millions, of dimensions. It's pretty much impossible to visualize spaces this high-dimensional.

\subsection{Data space}

The first space to be familiar with is {\bf data space}, or {\bf input space}. Each point in this space corresponds to a possible input vector. (We're going to abuse mathematical terminology a bit by using ``point'' and ``vector'' interchangeably.) It's customary to represent positive and negative examples with the symbols ``$+$'' and ``$-$'', respectively.

Once we've chosen the weights $\weights$ and bias $\bias$, we can divide the data space into a region where the points are classified as positive (the {\bf positive region}), and a region where the points are classified as negative (the {\bf negative region}). The boundary between these regions, i.e.~the set where $\weights^T \inputVec + \bias = 0$, is called the {\bf decision boundary}. Think back to your linear algebra class, and recall that the set determined by this equation is a {\bf hyperplane}. The set of points on one side of the hyperplane is called a {\bf half-space}. Examples are shown in Figure~\ref{fig:data-space-weight-space}

When we plot examples in two dimensions, the hyperplanes are actually lines. But you shouldn't think of them as lines --- you should think of them as hyperplanes.

If it's possible to choose a linear decision boundary that correctly classifies all of the training cases, the training set is said to be {\bf linearly separable}. As we'll see later, not all training sets are linearly separable.

\begin{figure}
\begin{center}
(a) \includegraphics[width=0.17 \textwidth]{figures/not_boundary.png}
(b) \includegraphics[width=0.17 \textwidth]{figures/not_weight.png}
(c) \includegraphics[width=0.17 \textwidth]{figures/and_boundary.png}
(d) \includegraphics[width=0.17 \textwidth]{figures/and_weight.png}
\end{center}
\caption{{\bf (a)} Training examples and for NOT function, in data space. {\bf (b)} NOT, in weight space. {\bf (c)} Slice of data space for AND function corresponding to $\inputJ{0} = 1$. {\bf (d)} Slice of weight space for AND function corresponding to $\weightJ{0} = -1$.}
\label{fig:data-space-weight-space}
\end{figure}

\subsection{Weight space}

As you'd expect from the name, weight vectors are also vectors, and the space they live in is called {\bf weight space}. In this section, we'll assume there is no bias parameter unless stated otherwise. (See Section \ref{sec:thresholds-and-biases}.) Each point in weight space is a possible weight vector.

Consider a positive training case $(\inputVec, 1)$. The set of weight vectors which correctly classify this training case is given by the linear inequality $\weights^T \inputVec \geq 0$. (In fact, it's exactly the sort of inequality we derived in Examples~\ref{example:not} and \ref{example:and}.) Geometrically, the set of points satisfying this inequality is a half-space. 
%Also, observe that the boundary (which is a hyperplane) passes through the origin, since $\zeroVec^T \inputVec = 0$ (where $\zeroVec$ denotes the all-zeros vector). 
For lack of a better term, we'll refer to the side which satisfies the constraint as the {\bf good region}, and the other side as the {\bf bad region}. Similarly, the set of weight vectors which correctly classify a negative training case $(\inputVec, 0)$ is given by $\weights^T \inputVec < 0$; this is also a half-space.\marginnote{We're going to completely ignore the fact that one of these inequalities is strict and the other is not. The question of what happens on the decision boundaries isn't very interesting.} Examples are shown in Figure~\ref{fig:data-space-weight-space}.

The set of weight vectors which correctly classify \emph{all} of the training examples is the intersection of all the half-spaces corresponding to the individual examples. This set is called the {\bf feasible region}. If the feasible region is nonempty, the problem is said to be {\bf feasible}; otherwise it's said to be {\bf infeasible}.

When we draw the constraints in two dimensions, we typically draw the line corresponding to the boundary of the constraint set, and then indicate the good region with an arrow. As with our data space visualizations, you should think of the boundary as a hyperplane, not as a line.\marginnote{There's one constraint per training example. What happened to the fourth constraint in Figure \ref{fig:data-space-weight-space}(d)?}

We can visualize three-dimensional examples by looking at slices. As shown in Figure~\ref{fig:weight-space-3d}, these slices will resemble our previous visualizations, except that the decision boundaries and constraints need not pass through the origin.

\begin{figure}
\begin{center}
\includegraphics[width=0.5 \textwidth]{figures/q1ws3d.pdf}
\end{center}
\caption{Visualizing a slice of a 3-dimensional weight space.}
\label{fig:weight-space-3d}
\end{figure}


\section{The perceptron learning rule}

The perceptron is a kind of binary linear classifier. Recall from last lecture that this means it makes predictions by computing $\weights^T \inputVec + \bias$ and seeing if the result is positive or negative. Here, $\inputVec$ is the input vector, $\weights$ is the weight vector, and $\bias$ is a scalar-valued bias. Recall as well that we can eliminate the bias by adding a dummy dimension to $\inputVec$. For the perceptron algorithm, it will be convenient to represent the positive and negative classes with 1 and -1, instead of 1 and 0 as we use in the rest of the course. Therefore, the classification model is as follows:
\begin{align}
\intermediate &= \weights^T \inputVec \\
\prediction &= \left\{ \begin{array}{ll} 1 & \textrm{ if } \intermediate \geq 0 \\ -1 & \textrm{ if } \intermediate < 0 \end{array} \right.
\end{align}

Here's a rough sketch of the {\bf perceptron algorithm}. We examine each of the training cases one at a time. For each input $\inputI{\dataIdx}$, we compute the prediction $\predictionI{\dataIdx}$ and see if it matches the target $\targetI{\dataIdx}$. If the prediction is correct, we do nothing. If it is wrong, we adjust the weights in a direction that makes it more correct.

Now for the details. First of all, how do we determine if the prediction is correct?  We could simply check if $\predictionI{\dataIdx} = \targetI{\dataIdx}$, but this has a slight problem: if $\inputI{\dataIdx}$ lies exactly on the classification boundary, it is technically classified as positive according to the above definition. But we don't want our training cases to lie on the decision boundary, since this means the classification may change if the input is perturbed even slightly. We'd like our classifiers to be more robust than this. Instead, we'll use the stricter criterion
\begin{equation}
\intermediateI{\dataIdx} \targetI{\dataIdx} > 0.
\end{equation}
You should now check that this criterion correctly handles the various cases that may occur.

The other question is, how do we adjust the weight vector?  If the training case is positive and we classify it as negative, we'd like to increase the value of $\intermediate$. In other words, we'd like 
\begin{equation}
\intermediate^\prime = \weights^{\prime T} \inputVec > \weights^T \inputVec = \intermediate, \label{perceptron-requirement}
\end{equation} 
where $\weights^\prime$ and $\weights$ are the new and old weight vectors, respectively. The perceptron algorithm achieves this using the update
\begin{equation}
\weights^\prime = \weights + \learningRate \inputVec,
\end{equation}
where $\learningRate > 0$. We now check that (\ref{perceptron-requirement}) is satisfied:
\begin{align}
\weights^{\prime T} \inputVec &= (\weights + \learningRate \inputVec)^T \inputVec \\
&= \weights^T \inputVec + \learningRate \inputVec^T \inputVec \\
&= \weights^T \inputVec + \learningRate \| \inputVec \|^2.
\end{align}
Here, $\| \inputVec \|$ represents the Euclidean norm of $\inputVec$. Since the squared norm is always positive, we have $\intermediate^\prime > \intermediate$.

Conversely, if it's a negative example which we mistakenly classified as positive, we want to decrease $\intermediate$, so we use a negative value of $\learningRate$. Since it's possible to show that the absolute value of $\learningRate$ doesn't matter, we generally use $\learningRate = 1$ for positive cases and $\learningRate = -1$ for negative cases. We can denote this compactly with
\begin{equation}
\weights \gets \weights + \target \inputVec.
\end{equation}
This rule is known as the {\bf perceptron learning rule}.

Now we write out the perceptron algorithm in full:
\begin{itemize}
\item[] For each training case $(\inputI{\dataIdx}, \targetI{\dataIdx})$,
\begin{itemize}
\item[] $\intermediateI{\dataIdx} \gets \weights^T \inputI{\dataIdx}$
\item[] If $\intermediateI{\dataIdx} \targetI{\dataIdx} \leq 0$,
\begin{itemize}
\item[] $\weights \gets \weights + \targetI{\dataIdx} \inputI{\dataIdx}$
\end{itemize}
\end{itemize}
\end{itemize}
In thinking about this algorithm, remember that we're denoting the classes with -1 and 1 (rather than 0 and 1, as we do in the rest of the course).


\section{The limits of linear classifiers}

Linear classifiers can represent a lot of things, but they can't represent everything. The classic example of what they can't represent is the XOR function. It should be pretty obvious from inspection that you can't draw a line separating the two classes. But how do we actually prove this?

\subsection{Convex sets}

An important geometric concept which helps us out here is {\bf convexity}. A set $S$ is convex if the line segment connecting any two points in $S$ must lie within $S$. It's not too hard to show that if $S$ is convex, then any {\bf weighted average} of points in $S$ must also lie within $S$. A weighted average of points $\inputI{1}, \ldots, \inputI{\ndata}$ is a point given by the linear combination
\[ \inputI{avg} = \lambda_1 \inputI{1} + \cdots + \lambda_\ndata \inputI{\ndata}, \]
where $0 \leq \lambda_i \leq 1$ and $\lambda_1 + \cdots + \lambda_\ndata = 1$. You can think of the weighted average as the center of mass, where the mass of each point is given by $\lambda_i$.

In the context of binary classification, there are two important sets that are always convex:
\begin{enumerate}
\item In data space, the positive and negative regions are both convex. Both regions are half-spaces, and it should be visually obvious that half-spaces are convex. This implies that if inputs $\inputI{1}, \ldots, \inputI{\ndata}$ are all in the positive region, then any weighted average must also be in the positive region. Similarly for the negative region.
\item In weight space, the feasible region is convex. The rough mathematical argument is as follows. Each good region (the set of weights which correctly classify one data point) is convex because it's a half-space. The feasible region is the intersection of all the good regions, so it must be convex because the intersection of convex sets is convex.
\end{enumerate}

\subsection{Showing that functions aren't linearly separable}

Now let's see how convexity can be used to show functions aren't linearly separable.

\begin{quote}
\begin{example}
Let's return to the XOR example. Since the positive region is convex, if we draw the line segment connecting the two positive examples $(0, 1)$ and $(1, 0)$, this entire line segment must be classified as positive. Similarly, if we draw the line segment connecting the two negative examples $(0, 0)$ and $(1, 1)$, the entire line segment must be classified as negative. But these two line segments intersect at $(0.5, 0.5)$, which means this point must be classified as both positive and negative, which is impossible. (See Figure \ref{fig:xor-contradiction}.) Therefore, XOR isn't linearly separable. 
\end{example}

\begin{figure}
\begin{center}
\includegraphics[width=0.2 \textwidth]{figures/xor_contradiction.png}
\end{center}
\caption{The XOR function is not linearly separable.}
\label{fig:xor-contradiction}
\end{figure}

\begin{example}
Our last example was somewhat artificial. Let's now turn to a somewhat more troubling, and practically relevant, limitation of linear classifiers. Let's say we want to give a robot a vision system which can recognize objects in the world. Since the robot could be looking any given direction, it needs to be able to recognize objects regardless of their location in its visual field. I.e., it should be able to recognize a pattern in any possible translation.

\begin{figure}
\begin{center}
\includegraphics[width=0.4 \textwidth]{figures/pattern_A.pdf}
\includegraphics[width=0.4 \textwidth]{figures/pattern_B.pdf}
\end{center}
\caption{No linear hypothesis can separate these two patterns in all possible translations (with wrap-around).}
\label{fig:patterns}
\end{figure}

As a simplification of this situation, let's say our inputs are 16-dimensional binary vectors and we want to distinguish two patterns, A, and B (shown in Figure \ref{fig:patterns}), which can be placed in any possible translation, with wrap-around. (I.e., if you shift the pattern right, then whatever falls off the right side reappears on the left.) Thus, there are 16 examples of A and 16 examples of B that our classifier needs to distinguish.

By convexity, if our classifier is to correctly classify all 16 instances of A, then it must also classify the average of all 16 instances as A. Since 4 out of the 16 values are on, the average of all instances is simply the vectors $(0.25, 0.25, \ldots, 0.25)$. Similarly, for it to correctly classify all 16 instances of B, it must also classify their average as B. But the average is also $(0.25, 0.25, \ldots, 0.25)$. Since this vector can't possibly be classified as both A and B, this dataset must not be linearly separable.

More generally, we can't expect any linear classifier to detect a pattern in all possible translations. This is a serious limitation of linear classifiers as a basis for a vision system.
\end{example}
\end{quote}

\subsection{Circumventing this problem by using feature representations}

We just saw a negative result about linear classifiers. Let's end on a more positive note. In Lecture 2, we saw how linear regression could be made more powerful using a basis function, or feature, representation. The same trick applies to classification. Essentially, in place of $\intermediate = \weights^T \inputVec + \bias$, we use $\intermediate = \weights^T \featureVec(\inputVec) + \bias$, where $\featureVec(\inputVec) = (\featureJ{1}(\inputVec), \ldots, \featureJ{\nfeat}(\inputVec))$ is a function mapping input vectors to feature vectors. Let's see how we can represent XOR using carefully selected features. 

\begin{quote}
\begin{example}
Consider the following feature representation for XOR:
\begin{align*}
\featureJ{1}(\inputVec) &= \inputJ{1} \\
\featureJ{2}(\inputVec) &= \inputJ{2} \\
\featureJ{3}(\inputVec) &= \inputJ{1} \inputJ{2}
\end{align*}
In this representation, our training set becomes
\begin{center}
\begin{tabular}{ccc|c}
$\featureJ{1}(\inputVec)$ & $\featureJ{2}(\inputVec)$ & $\featureJ{3}(\inputVec)$ & $\target$ \\
\hline
0 & 0 & 0 & 0 \\
0 & 1 & 0 & 1 \\
1 & 0 & 0 & 1 \\
1 & 1 & 1 & 0
\end{tabular}
\end{center}
Using the same techniques as in Examples~\ref{example:not} and \ref{example:and}, we find that the following set of weights and biases correctly classifies all the training examples:
\[ \bias = -0.5 \qquad \weightJ{1} = 1 \qquad \weightJ{2} = 1 \qquad \weightJ{3} = -2. \]
\end{example}
\end{quote}

The only problem is, where do we get the features from?  In this example, we just pulled them out of a hat. Unfortunately, there's no recipe for coming up with good features, which is part of what makes machine learning hard. But next week, we'll see how we can \emph{learn} a set of features by training a multilayer neural net.



%\section{Summary}

%\begin{itemize}
%\item In binary classification, the goal is to predict a binary-valued target from a set of inputs. Binary linear classifiers compute a linear function of the inputs and determine if it's larger than a threshold.
%\item The threshold can be replaced with a bias parameter or a dummy feature, whichever is most convenient in a given context.
%\item Input space is the space where input vectors live. The axes correspond to input dimensions. Input vectors correspond to points, and linear classifiers correspond to half-spaces.
%\item Weight space is the space where weight vectors live. The axes correspond to entries of the weight vector. Weight vectors correspond to points, and training cases correspond to half-spaces.
%\item The intersection of all the half-spaces corresponding to different training examples is called the feasible region. If the feasible region is non-empty, the problem is said to be feasible (or linearly separable), otherwise it's said to be infeasible.
%\item Some functions, such as XOR, can't be represented by linear classifiers. However, they often can be represented by linear classifiers on top of a feature representation.
%\end{itemize}



\end{document}












