Lectures

Lectures

Rocco's lecture notes will be posted soon after the class. You can also find course videos in the "Video Library" section of Courseworks soon after class.

Warning: the notes below were generated in real time and have not been edited. They may contain typos or other errors. They may also contain aesthetically displeasing color combinations.

Number	Date	Topics	Notes	References
1	Wed Sept 6	Introduction, basics	PDF	Blum survey sec. 3.0
2	Mon Sept 11	Online mistake-bound learning, elimination algorithm, decision lists	PDF	Blum survey sec. 3.0, 3.1
3	Wed Sept 13	Learning decision lists, Winnow1	PDF	Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7)
4	Mon Sept 18	Winnow2, Perceptron	PDF	Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7), handout on Perceptron and kernel methods
5	Wed Sept 20	Perceptron, dual Perceptron, kernel methods	PDF	handout on Perceptron and kernel methods
6	Mon Sept 25	General bounds on OLMB learning: Halving Algorithm, Randomized H.A., start VC Dimension	PDF	Blum survey sec. 2.0, 2.1, 2.2, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA)
7	Wed Sept 27	General bounds on OLMB learning: VC dimension, Weighted Majority algorithm	PDF	Blum survey sec. 2.0, 2.1, 2.2, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA)
8	Mon Oct 2	Randomized Weighted Majority algorithm, intro to PAC learning, PAC learning intervals	PDF	Kearns and Vazirani chapter 1.1-1.3
9	Wed Oct 4	finish PAC learning intervals, OLMB to PAC conversion, definitional issues	PDF	Kearns and Vazirani chapter 1.1-1.3
10	Mon Oct 9	Chernoff bounds, learning by finding consistent hypotheses, Occam's Razor	PDF	Kearns and Vazirani chapters 1,2, appendix (Chapter 9), this handout on probability basics, this handout on Chernoff bounds
11	Wed Oct 11	PAC sample-efficient learning sparse disjunctions via Occam and greedy set cover, start proper versus improper learning	PDF	Kearns and Vazirani chapters 1,2
12	Mon Oct 16	Improper PAC learning of 3-term DNF is computationally easy, proper PAC learning of 3-term DNF is computationally hard	PDF	Kearns and Vazirani chapters 1,2
13	Wed Oct 18	Finish hardness of proper PAC learning 3-term DNF; Lower bound on PAC learning sample complexity based on VC dimension; start upper bound	PDF	Kearns and Vazirani chapter 3
14	Mon Oct 23	No lecture (midterm exam)
15	Wed Oct 25	Upper bound on PAC learning sample complexity based on VC dimension: Sauer-Shelah-Perles lemma	PDF	Kearns and Vazirani chapter 3
15	Mon Oct 30	Upper bound on PAC learning sample complexity based on VC dimension: ``double sample'' argument, application to PAC learning LTFs over \R^n	PDF	Kearns and Vazirani chapter 3 (you can peek at Chapter 4.0-4.3.2 as a head start for next time)
16	Wed Nov 1	Confidence boosting; accuracy boosting overview; start simple 3-stage accuracy improving procedure	PDF	Kearns and Vazirani Chapter 4.0-4.3.2
	Mon Nov 6	No lecture (University holiday)
17	Wed Nov 8	Finish simple 3-stage accuracy improving procedure; boosting over a fixed sample; AdaBoost	PDF, annotated AdaBoost algorithm	Kearns and Vazirani Chapter 4.0-4.3.2; clean AdaBoost handout; Schapire boosting overview paper
18	Thurs Nov 10	AdaBoost analysis; start PAC learning with noise	PDF	Schapire boosting overview paper
19	Wed Nov 15	PAC learning with malicious noise; start PAC learning with random classification noise	PDF	Kearns and Vazirani Chapter 5
20	Mon Nov 20	PAC learning with random classification noise; Statistical Query learning	PDF	Kearns and Vazirani Chapter 5
21	Mon Nov 27	Statistical Query learning algorithms yield RCN-tolerant PAC algorithms; start lower bounds on SQ learning	PDF	Kearns and Vazirani Chapter 5
22	Wed Nov 29	Lower bounds on SQ learning; start cryptographic hardness of learning	PDF	Kearns and Vazirani Chapter 5
23	Mon Dec 4	Cryptographic hardness of learning based on pseudorandomness, start crypto hardness based on PKC	PDF	Kearns and Vazirani Chapter 6
24	Wed Dec 6	Cryptographic hardness of learning based on PKC / trapdoor one-way permutations, discrete cube roots	PDF	Kearns and Vazirani Chapter 6
25	Thurs Dec 8	Cryptographic hardness of learning simple circuits based on discrete cube roots, peek at other topics		Kearns and Vazirani Chapter 6

Schedule of Topics

Here is an anticipated list of topics. Note that the ordering of some topics may change, and we may spend more or less than one lecture per topic.

• Introduction to machine learning theory. Learning models and learning problems.
• Online mistake bound learning. Algorithms for simple concept classes. Attribute efficient learning with the Winnow algorithm.
• Winnow algorithm continued. Perceptron algorithm. General bounds for online mistake bound learning. The Halving algorithm, the Standard Optimal Algorithm.
• Best Experts and Weighted Majority.
• The Probably Approximately Correct(PAC) Learning model. PAC learning algorithms for simple concept classes.
• More PAC learning algorithms. Conversions from online learning to PAC learning. (KV chapter 1)
• Occam's Razor: learning by finding consistent hypotheses. Applications (KV chapter 2).
• Computational hardness results for finding consistent hypotheses.
• Vapnik-Chervonenkis dimension. Upper and lower bounds on sample complexity (KV chapter 3).
• VC dimension and sample complexity continued.
• VC dimension and sample complexity continued.
• Weak learning, strong learning, and Boosting (KV chapter 4).
• Boosting continued.
• Boosting continued.
• Learning in the presence of noise. Classification noise, malicious noice,
• Statistical Query learning (KV chapter 5).
• Learning with noise continued.
• Learning with noise continued.
• Cryptographic limitations on efficient learning (KV chapter 6).
• Cryptographic limitations on learning continued.
• Cryptographic limitations and reductions in PAC learning (KV chapters 6,7).
• The model of exact learning from membership and equivalence queries. Learning monotone DNF formulas.
• Learning deterministic finite automata.
• Learning deterministic finite automata continued.