Fall 2019

Columbia University

- Description and Syllabus
- Instructor: David M. Blei (office hours)
- Recitation Instructor: Dhanya Sridhar
- Teaching Assistant: Charles Margossian
- Meeting: Tue/Thu 10:10AM-11:25AM, 310 Fayerweather
- Recitation (optional): Wed 4:00PM-6:00PM, CS 480 (in Mudd)
- Piazza site
- Final project guidelines
- LaTeX template

The course is open to all PhD students at Columbia University.

Please hand in all assignments through Courseworks.

- Homework 0 (due Friday September 6. Submit your solution here. The filename should start with your UNI.)
- Homework 1; datasets
- Homework 2; datasets

Below are the topics of the class and some readings about each. (Some readings are not yet set.)

The readings are at different levels: some are basic and some are advanced. We chose them to provide fundamental and other interesting material about the topics; the lectures will not necessarily cover or follow all of this material.

In addition to these readings, we will also make available Blei (2019), which is a book in preparation. The materials below are covered in this book.

**Introduction**- "Build, compute, critique, repeat: Data analysis with latent variable models" (Blei, 2014)
- Slides

**The ingredients of probabilistic models**- "Model-based machine learning" (Bishop, 2013)
- "Some issues in the foundations of statistics" (Freedman, 1994)
- "The Elements of Statistical Learning" (Chapter 3, Chapter 7) (Hastie et al., 2009)

**Bayesian mixture models and the Gibbs sampler**- "Identifying Bayesian mixture models" (Betancourt, 2018)
- "Probabilistic inference using Markov chain Monte Carlo methods" (Sections 1-4) (Neal, 1993)

**Mixed-membership models, topic models, and variational inference**- "Probabilistic topic models" (Blei, 2012)
- "Applications of topic models" (Boyd-Graber et al., 2017)
- "Variational inference: A review for statisticians" (Blei et al., 2017)

**Matrix factorization and efficient MAP inference**- "Matrix factorization techniques for recommender systems" (Koren et al., 2009)
- "Learning the parts of objects by non-negative matrix factorization" (Lee and Seung, 1999)
- "Scalable Recommendation with Hierarchical Poisson Factorization" (Gopalan et al., 2015)
- "Tensor decompositions and applications" (Kolda and Bader, 2009)

**Deep generative models and black box variational inference**- "Stochastic backpropagation and approximate inference in deep generative models" (Rezende et al., 2014)
- "Autoencoding variational Bayes" (Kingma and Welling, 2013)
- "Deep exponential families" (Ranganath et al., 2015)
- "Black box variational inference" (Ranganath et al., 2014)
- "Mean field theory for sigmoid belief networks" (Saul et al., 1996)
- "Representation learning: A review and new perspectives" (Bengio et al., 2013)

**Exponential families, conjugate priors, and generalized linear models**- "The exponential family" (Bishop, 2006; Section 2.4)
- "An outline of generalized linear models" (McCullagh and Nelder, 1989; Chapter 2)
- "Conjugate priors for exponential families" (Diaconis and Ylvisaker, 1979)
- "Exponential families in theory and practice" (Efron, 2018)

**Hierarchical models, robust models, and empirical Bayes**- "Multi-level structures" (Gelman and Hill, 2007; Chapter 11)
- "Multi-level linear models: The basics" (Gelman and Hill, 2007; Chapter 12)
- "Bayes, oracle Bayes, and empirical Bayes" (Efron, 2017)

**The theory of graphical models**- "Conditional independence and factorization" (Jordan, 2003; Chapter 2)
- "The elimination algorithm" (Jordan, 2003; Chapter 3)
- "Probability propagation and factor graphs" (Jordan, 2003; Chapter 4)

**Advanced topics in variational inference**- "Graphical models, exponential families, and variational inference" (Waingwright and Jordan, 2008)
- "Covariance, robustness, and variational Bayes" (Broderick et al., 2018)

**Model checking and model diagnosis**- "Bayesianly justifiable and relevant frequency calculations for the applied statistician" (Rubin, 1984)
- "Posterior predictive assessment of model fitness via realized discrepancies" (Gelman et al., 1996)
- "Philosophy and the practice of Bayesian statistics" (Gelman and Shalizi, 2013)

**An introduction to causality**- "Statistics and causal inference" (Holland, 1986)
- "Causal inference in statistics: An overview" (Pearl, 2009)