Lectures

Lectures

Rocco's lecture notes will be posted soon after the class. You are highly encouraged to attend all classes.

For your convenience, here is a link to Courseworks from which you can access class sessions. Lecture recordings can be found here.

Warning: the notes below were generated in real time and have not been edited. They may contain typos or other errors. They may also contain aesthetically displeasing color combinations.

Number	Date	Topics	Notes	References
1	Mon January 11	Introduction, basics	PDF
2	Wed January 13	Basics, Online mistake-bound learning, elimination algorithm	PDF	Blum survey sec. 3.0, 3.1
	Mon January 18	No Class: MLK Day
3	Wed January 20	Learning decision lists, Winnow1	PDF	Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7)
4	Mon January 25	Winnow1, Winnow2, Perceptron	PDF	Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7), handout on Perceptron and kernel methods
5	Wed January 27	Perceptron, kernel methods, general bounds on OLMB learning	PDF	handout on Perceptron and kernel methods, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA)
6	Mon February 1	General bounds on OLMB learning: Halving Algorithm, VC Dimension	PDF	Blum survey sec. 2.0, 2.1, 2.2, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA)
7	Wed February 3	General bounds on OLMB learning: (Randomized) Weighted Majority; start PAC learning	PDF	Blum survey sec. 2.0, 2.1, 2.2, Kearns and Vazirani chapter 1.1-1.3
8	Mon February 8	PAC learning: definition, learning intervals, OLMB-to-PAC conversion	PDF	Kearns and Vazirani chapter 1.1-1.3
9	Wed February 10	PAC learning: more definitional subtleties, hypothesis testing / probability basics, learning via consistent hypotheses	PDF	Kearns and Vazirani chapters 1,2, appendix (Chapter 9)
10	Mon February 15	PAC learning: consistent hypothesis finders, Occam's razor, application to learning sparse disjunctions	PDF	Kearns and Vazirani chapters 1,2
11	Wed February 17	PAC sample-efficient learning sparse disjunctions via Occam and greedy set cover, proper versus improper learning	PDF	Kearns and Vazirani chapters 1,2
12	Mon February 22	Proper PAC learning of 3-term DNF is hard	PDF	Kearns and Vazirani chapters 1,2
13	Mon March 8	Lower bound on PAC learning sample complexity based on VC dimension; Sauer-Shelah lemma	PDF	Kearns and Vazirani chapter 3
14	Wed March 10	Sauer-Shelah lemma; PAC learning using CHF with VC dimension controlling sample complexity required	PDF	Kearns and Vazirani chapter 3
15	Mon March 15	PAC learning using CHF with VC dimension controlling sample complexity required; learning LTFs over Euclidean space; confidence boosting; start accuracy boosting	PDF	Kearns and Vazirani chapter 3
16	Wed March 17	Boosting overview; simple 3-stage accuracy improving procedure	PDF	Kearns and Vazirani chapter 4 through 4.3.2, Schapire boosting overview paper
17	Mon March 22	Boosting by filtering, boosting by sampling; AdaBoost	PDF	Schapire boosting overview paper (through 8.3)
18	Wed March 24	Finish AdaBoost proof, start learning with noise	PDF	Schapire boosting overview paper (through 8.3), Kearns and Vazirani chapter 5
19	Mon March 29	Lower bounds and algorithmic strategies for malicious noise; start random classification noise	PDF	Kearns and Vazirani chapter 5 (through Section 5.2)
20	Wed March 31	Random Classification Noise, Statistical Query Learning	PDF	Kearns and Vazirani chapter 5
21	Mon April 5	Statistical Query Learning, PAC learning with RCN, lower bounds on SQ learning	PDF	Kearns and Vazirani chapter 5
22	Wed April 7	Noise-tolerant learning of PAR, noise and proper learning, start representation-independent hardness of learning	PDF	Kearns and Vazirani chapter 6
23	Mon April 12	Computational hardness of learning based on pseudorandomness	PDF	Kearns and Vazirani chapter 6
24	Wed April 14	Computational hardness of learning based on public-key cryptography / one-way permutations	PDF	Kearns and Vazirani chapter 6

Schedule of Topics

Here is an anticipated list of topics. Note that the ordering of some topics may change, and we may spend more or less than one lecture per topic.

• Introduction to machine learning theory. Learning models and learning problems.
• Online mistake bound learning. Algorithms for simple concept classes. Attribute efficient learning with the Winnow algorithm.
• Winnow algorithm continued. Perceptron algorithm. General bounds for online mistake bound learning. The Halving algorithm, the Standard Optimal Algorithm.
• Best Experts and Weighted Majority.
• The Probably Approximately Correct(PAC) Learning model. PAC learning algorithms for simple concept classes.
• More PAC learning algorithms. Conversions from online learning to PAC learning. (KV chapter 1)
• Occam's Razor: learning by finding consistent hypotheses. Applications (KV chapter 2).
• Computational hardness results for finding consistent hypotheses.
• Vapnik-Chervonenkis dimension. Upper and lower bounds on sample complexity (KV chapter 3).
• VC dimension and sample complexity continued.
• VC dimension and sample complexity continued.
• Weak learning, strong learning, and Boosting (KV chapter 4).
• Boosting continued.
• Boosting continued.
• Learning in the presence of noise. Classification noise, malicious noice,
• Statistical Query learning (KV chapter 5).
• Learning with noise continued.
• Learning with noise continued.
• Cryptographic limitations on efficient learning (KV chapter 6).
• Cryptographic limitations on learning continued.
• Cryptographic limitations and reductions in PAC learning (KV chapters 6,7).
• The model of exact learning from membership and equivalence queries. Learning monotone DNF formulas.
• Learning deterministic finite automata.
• Learning deterministic finite automata continued.