Rocco's lecture notes will be posted soon after the class. You can also find course videos in the "Video Library" section of Courseworks soon after class.

**Warning:** the notes below were generated in real time and have not been edited. They may contain typos or other errors. They may also contain aesthetically displeasing color combinations.

Number | Date | Topics | Notes | References | ||
---|---|---|---|---|---|---|

1 | Wed Sept 6 | Introduction, basics | Blum survey sec. 3.0 | |||

2 | Mon Sept 11 | Online mistake-bound learning, elimination algorithm, decision lists | Blum survey sec. 3.0, 3.1 | |||

3 | Wed Sept 13 | Learning decision lists, Winnow1 | Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7) | |||

4 | Mon Sept 18 | Winnow2, Perceptron | Blum survey sec. 3.2, Littlestone paper sec. 5 (just through Theorem 7), handout on Perceptron and kernel methods | |||

5 | Wed Sept 20 | Perceptron, dual Perceptron, kernel methods | handout on Perceptron and kernel methods | |||

6 | Mon Sept 25 | General bounds on OLMB learning: Halving Algorithm, Randomized H.A., start VC Dimension | Blum survey sec. 2.0, 2.1, 2.2, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA) | |||

7 | Wed Sept 27 | General bounds on OLMB learning: VC dimension, Weighted Majority algorithm | Blum survey sec. 2.0, 2.1, 2.2, Littlestone paper sec. 1-3 (don't worry about the stuff about the SOA) | |||

8 | Mon Oct 2 | Randomized Weighted Majority algorithm, intro to PAC learning, PAC learning intervals | Kearns and Vazirani chapter 1.1-1.3 | |||

9 | Wed Oct 4 | finish PAC learning intervals, OLMB to PAC conversion, definitional issues | Kearns and Vazirani chapter 1.1-1.3 | |||

10 | Mon Oct 9 | Chernoff bounds, learning by finding consistent hypotheses, Occam's Razor | Kearns and Vazirani chapters 1,2, appendix (Chapter 9), this handout on probability basics, this handout on Chernoff bounds | |||

11 | Wed Oct 11 | PAC sample-efficient learning sparse disjunctions via Occam and greedy set cover, start proper versus improper learning | Kearns and Vazirani chapters 1,2 | |||

12 | Mon Oct 16 | Improper PAC learning of 3-term DNF is computationally easy, proper PAC learning of 3-term DNF is computationally hard | Kearns and Vazirani chapters 1,2 | |||

13 | Wed Oct 18 | Finish hardness of proper PAC learning 3-term DNF; Lower bound on PAC learning sample complexity based on VC dimension; start upper bound | Kearns and Vazirani chapter 3 | |||

14 | Mon Oct 23 | No lecture (midterm exam) | ||||

15 | Wed Oct 25 | Upper bound on PAC learning sample complexity based on VC dimension: Sauer-Shelah-Perles lemma | Kearns and Vazirani chapter 3 | |||

15 | Mon Oct 30 | Upper bound on PAC learning sample complexity based on VC dimension: ``double sample'' argument, application to PAC learning LTFs over \R^n | Kearns and Vazirani chapter 3 (you can peek at Chapter 4.0-4.3.2 as a head start for next time) | |||

16 | Wed Nov 1 | Confidence boosting; accuracy boosting overview; start simple 3-stage accuracy improving procedure | Kearns and Vazirani Chapter 4.0-4.3.2 | |||

Mon Nov 6 | No lecture (University holiday) | |||||

17 | Wed Nov 8 | Finish simple 3-stage accuracy improving procedure; boosting over a fixed sample; AdaBoost | PDF, annotated AdaBoost algorithm | Kearns and Vazirani Chapter 4.0-4.3.2; clean AdaBoost handout; Schapire boosting overview paper | ||

18 | Thurs Nov 10 | AdaBoost analysis; start PAC learning with noise | Schapire boosting overview paper | |||

19 | Wed Nov 15 | PAC learning with malicious noise; start PAC learning with random classification noise | Kearns and Vazirani Chapter 5 | |||

20 | Mon Nov 20 | PAC learning with random classification noise; Statistical Query learning | Kearns and Vazirani Chapter 5 | |||

21 | Mon Nov 27 | Statistical Query learning algorithms yield RCN-tolerant PAC algorithms; start lower bounds on SQ learning | Kearns and Vazirani Chapter 5 | |||

22 | Wed Nov 29 | Lower bounds on SQ learning; start cryptographic hardness of learning | Kearns and Vazirani Chapter 5 | |||

23 | Mon Dec 4 | Cryptographic hardness of learning based on pseudorandomness, start crypto hardness based on PKC | Kearns and Vazirani Chapter 6 | |||

24 | Wed Dec 6 | Cryptographic hardness of learning based on PKC / trapdoor one-way permutations, discrete cube roots | Kearns and Vazirani Chapter 6 | |||

25 | Thurs Dec 8 | Cryptographic hardness of learning simple circuits based on discrete cube roots, peek at other topics | Kearns and Vazirani Chapter 6 |

Here is an anticipated list of topics. Note that the ordering of some topics may change, and we may spend more or less than one lecture per topic.

- Introduction to machine learning theory. Learning models and learning problems.
- Online mistake bound learning. Algorithms for simple concept classes. Attribute efficient learning with the Winnow algorithm.
- Winnow algorithm continued. Perceptron algorithm. General bounds for online mistake bound learning. The Halving algorithm, the Standard Optimal Algorithm.
- Best Experts and Weighted Majority.
- The Probably Approximately Correct(PAC) Learning model. PAC learning algorithms for simple concept classes.
- More PAC learning algorithms. Conversions from online learning to PAC learning. (KV chapter 1)
- Occam's Razor: learning by finding consistent hypotheses. Applications (KV chapter 2).
- Computational hardness results for finding consistent hypotheses.
- Vapnik-Chervonenkis dimension. Upper and lower bounds on sample complexity (KV chapter 3).
- VC dimension and sample complexity continued.
- VC dimension and sample complexity continued.
- Weak learning, strong learning, and Boosting (KV chapter 4).
- Boosting continued.
- Boosting continued.
- Learning in the presence of noise. Classification noise, malicious noice,
- Statistical Query learning (KV chapter 5).
- Learning with noise continued.
- Learning with noise continued.
- Cryptographic limitations on efficient learning (KV chapter 6).
- Cryptographic limitations on learning continued.
- Cryptographic limitations and reductions in PAC learning (KV chapters 6,7).
- The model of exact learning from membership and equivalence queries. Learning monotone DNF formulas.
- Learning deterministic finite automata.
- Learning deterministic finite automata continued.