ADVANCED MACHINE                            January, 2015




Day & Time and Location

M/W 1:10pm-2:25pm

Location HAV 309


Professor Tony Jebara


Office Hours

CEPSR 605, Monday 2:30-3:30 or by appointment


Enze Li, el2742(at)columbia(dot)edu

Office Hours: CEPSR 6LE5 Mon 10am-11am and TA Room Wed 10am-11am

Kui Tang, kt2384(at)columbia(dot)edu

Office Hours: CEPSR 6LE5 Tue 10am-11am or by appointment

Sami Mourad, sm3891(at)columbia(dot)edu

Office Hours: CEPSR 6LE5 Thu 10am-11am and Fri 11am-12pm



Prerequisites: COMS W4771 or permission of instructor. Knowledge of linear

algebra and introductory probability or statistics is required.



Advanced topics in machine learning including: Linear Modeling,
Nonlinear Dimension Reduction, Maximum Entropy,
Exponential Family Models, Conditional Random Fields, Graphical
Models, Structured Support Vector Machines, Feature Selection, Kernel
Selection, Meta-Learning, Multi-Task Learning, Semi-Supervised
Learning, Graph-Based Semi-Supervised Learning, Approximate Inference,
Clustering, and Boosting.


Required Texts:


Primarily through handouts and links to various research papers.


Optional Texts:


Tony Jebara, Machine Learning: Discriminative and Generative.

Michael I. Jordan and Christopher M. Bishop, Introduction to Graphical Models.

Still unpublished. Available online (password-protected) on class home page.


R.O. Duda, P.E. Hart and D.G. Stork, Pattern Classification, John Wiley & Sons, 2001.


Trevor Hastie, Robert Tibshirani and Jerome Friedman, The Elements of Statistical

Learning. Springer Series in Statistics, Springer-Verlag New York USA. 2001.



Graded Work: Grades are based on 2 applied homeworks for 45% of the grade

and a large research level project with a final presentation (55%).



Tentative Schedule:


Topics: A tentative wish list, we’ll see what we can go through!

Week 1

Introduction, Review of Basic Concepts, Representation Issues, Vector and Appearance-Based Models, Correlation and Least Squared Error Methods, Bases, Eigenspace Recognition, Principal Components Analysis

Week 2

Nonlinear Dimensionality Reduction, Manifolds, Kernel PCA, Locally Linear Embedding, Maximum Variance Unfolding, Minimum Volume Embedding

Week 3

Maximum Entropy, Exponential Families, Maximum Entropy Discrimination, Large Margin Probability Models

Week 4

Conditional Random Fields and Linear Models, Iterative Scaling and Majorization

Week 5

Graphical Models, Multi-Class Support Vector Machines, Structured Support Vector Machines, Cutting Plane Algorithms

Week 6

Kernels and Probabilistic Kernels

Week 7

Feature Selection and Kernel Selection, Support Vector Machine Extensions

Week 8

Meta-Learning and Multi-Task Support Vector Machines

Week 9

Semi-Supervised Learning and Graph-Based Semi-Supervised Learning

Week 10  

High-Tree Width Graphical Models, Approximate Inference, Graph Structure Learning

Week 11

Clustering, Spectral Clustering, Normalized Cuts.

Week 12

Boosting, Mixtures of Experts, AdaBoost, Online Learning

Week 13  

Project Presentations

Week 14

Project Presentations



Class Attendance: Class participation and interaction is an important aspect of this

course, ideally the course will run as a seminar where material presented in the class

lectures, recitations, and so forth. Some material will diverge from the textbooks

so regular attendance is important.


Late Policy: If you hand in late work without approval of the instructor or TAs,

you may receive zero credit. Homework is due as announced on its web page.

For the project, please submit on time regardless of additional progress.

For the final project, each day of lateness will cost you a minimum of 15%.

We won't give extensions, regardless of how amitious your project is.


Cooperation on Homework: Copying of solutions is forbidden.


Web Page: The class URL is: and

will contain copies of handouts, homework assignments, solutions and other



Computer Accounts: You need a UNI account for

and you will need access to Matlab for homeworks.