COMS 4771

Tentative Schedule

Topic 1
Introduction, Maximum Likelihood Estimation, Classification via Probabilistic Modeling, Bayes Classifier, Naive Bayes, Evaluating Classifiers
[lec 1]
notes/reading: MLE basics | Multivariate Gaussian | Naive Bayes | Optimality of Bayes classifier | CML 1.1-1.2, 1.4, 2.1 | DL Part I
Topic 2
Generative vs. Discriminative classifiers, Nearest Neighbor classifier, Coping with drawbacks of k-NN, Decision Trees, Model Complexity and Overfitting
[lec 2]
notes/reading: NN and k-d trees | decision trees | CML 1.3, 2.2-2.5, 3.1-3.2
Topic 3
Decision boundaries for classification, Linear decision boundaries (Linear classification), The Perceptron algorithm, Coping with non-linear boundaries, Kernel feature transform, Kernel trick
[lec 3]
notes/reading: CML 4.1-4.8, 11.1-11.2
Topic 4
Support Vector Machines, Large margin formulation, Constrained Optimization, Lagrange Duality, Convexity, Duality Theorems, Dual SVMs
[lec 4]
notes/reading: SVM basics | Lagrange Duality | CVX book | CML 7.7, 11.5-6
Topic 5
Regression, Parametric vs. non-parametric regression, Ordinary least squares regression, Logistic regression, Lasso and ridge regression, Optimal regressor, Kernel regression, consistency of kernel regression
[lec 5]
notes/reading: Logistic regression | Kernel regression
Exam #1
Topic 6
(if time permits)
Statistical theory of learning, PAC-learnability, Occam's razor theorem, VC dimension, VC theorem, Concentration of measure
[lec 6]
notes/reading: PAC and VC tutorial | Intro. Learning Theory | CML 12.3, 12.5-6
Topic 7
Unsupervised Learning, Clustering, k-means, Hierarchical clustering, Gaussian mixture modeling, Expectation Maximization Algorithm
[lec 7]
notes/reading: GMMs and EM | CML 15.1
Topic 8
Dimensionality Reduction, Principal Components Analysis (PCA), non-linear dimension reduction (manifold learning)
[lec 8] notes/reading: PCA tutorial | CML 15.2 | Non-linear Dimension Reduction
Topic 9
(if time permits)
Graphical Models, Bayesian Networks, Markov Random Fields, Inference and learning on graphical models, Markov Chains, Hidden Markov Models (HMMs)
[lec 9] notes/reading: Graphical Models tutorial | HMM tutorial
Exam #2

Resources

Books on ML
The Elements of Statistical Learning by Hastie, Tibshirani and Friedman (link)
Pattern Recognition and Machine Learning by Bishop (link)
A Course in Machine Learning by Daume (link)
Deep Learning by Goodfellow, Bengio and Courville (link)
Understanding Machine Learning by Shalev-Shwartz and Ben-David (link)
Software
MATLAB: download info, learning the basics.
Python: download info, tutorial
LaTeX: download info, learning the basics, typesetting common math formulas.
Background
Probability: Events, discrete and continuous random variables, expectations, joint-, conditional- and marginal distributions, independence, concepts of standard deviation, variance, covariance, and correlation. (refresher 1, refresher 2, refresher 3, refresher 4)
Statistics: Bayes Rule, Priors, Posteriors, Maximum Likelihood Principle (MLE), Basic distributions such as Bernoulli, Binomial, Multinomial, Poisson, Gaussian. Multivariate versions of these distributions, especially Multivariate Gaussian Distribution. (refresher, reference sheet)
Linear Algebra: Vector spaces, subspaces, matrix inversion, matrix multiplication, linear independence, rank, determinants, orthonormality, basis, solving systems of linear equations. Eigenvectors/values, Eigen- and Singular Value Decomposition. Identifying and working with popular types of matrices - eg symmetric matrices, positive (semi-) definite matrices, non-singular matrices, unitary matrices, rotation matrices, etc. (refresher 1, refresher 2, refresher 3, refresher 4)

Multivariate Calculus: Take derivatives and integrals of common functions, gradient, Jacobian, Hessian, compute maxima and minima of common functions. Differentiation of vector valued functions. (basic calculus identities, multivariable differentiation, extrema refresher, refresher 1, refresher 2)

Algorithm basics: Ability to write and analyze simple algorithms; basic understanding of time and space complexity. (refresher 1, refresher 2)

Mathematical maturity: Ability to communicate technical ideas clearly. (refresher 1, refresher 2)

Programming: Ability to program in a high-level language, and familiarity with basic algorithm and datastructure design and good coding principles.
Misc. Books
The Matrix Cookbook
Convex Optimization
Mathematics for Machine Learning

Policies

Homework submission & formatting
Homeworks will contain a mix of programming and written assignments.

The written segment of the homeworks must be typesetted as a PDF document, with all mathematical formulas properly formatted. It is strongly recommended that you prepare your write-up as a LaTeX document (you can use the following templates: homework.tex, homework.cls and homework.pdf); see the Resources section for more details on LaTeX. An MS Word writeup is also fine; you must make sure that all math formulas are typesetted properly.

The programming segment can be done in any high-level programming language. You are expected to have good working knowledge of the programming language you choose, and will receive very little 'debugging' help from the staff. You may use supporting math/scientific libraries for auxiliary tasks (such as matrix multiplication, matrix inversion, etc.), however you must write code for the core algorithms yourself.

Please include your name and UNI on the first page of the written assignment and at the top level comment of your programming assignment.

The written segment of the homework (including plots and comparative experimental studies) must be submitted via Gradescope, and (if the homeworks specifies) the a tarball of the programming files should be handed to the TA by the specified due dates. No late homeworks will be accepted.
Collaboration
Discussion of the homework problems is encouraged, but you must write the solutions individually (or as specified in the Homeworks). Detailed discussion of the solution must only be discussed within the group. General discussion on problem clarification and possible approaches can be discussed with others over EdStem. Everyone must write up their own solutions independently.
Late submissions
No late homeworks will be accepted.
Academic rules of conduct
Students are expected to adhere to the Academic Honesty policy of the Computer Science Department, this policy can be found in full here.
Violations
Violation of any portion of these policies will result in a penalty to be assessed at the instructor's discretion. This may include receiving a zero grade for the assignment in question and a failing grade for the whole course, even for the first infraction.

Machine Learning

COMS 4721 / 4771 Spring 2024

Tentative Schedule

Course Info

Resources

Policies