Foundations of Graphical Models
Fall 2019
Columbia University

Course information

• Description and Syllabus
• Instructor: David M. Blei (office hours)
• Recitation Instructor: Dhanya Sridhar
• Teaching Assistant: Charles Margossian
• Meeting: Tue/Thu 10:10AM-11:25AM, 310 Fayerweather
• Recitation (optional): Wed 4:00PM-6:00PM, CS 480 (in Mudd)
• Piazza site
• Final project guidelines
• LaTeX template

Registering

The course is open to all PhD students at Columbia University.

Assignments

Please hand in all assignments through Courseworks.

• Homework 0 (due Friday September 6. Submit your solution here. The filename should start with your UNI.)
• Homework 1; datasets
• Homework 2; datasets

Topics and readings

Below are the topics of the class and some readings about each. (Some readings are not yet set.)

The readings are at different levels: some are basic and some are advanced. We chose them to provide fundamental and other interesting material about the topics; the lectures will not necessarily cover or follow all of this material.

In addition to these readings, we will also make available Blei (2019), which is a book in preparation. The materials below are covered in this book.

1. Introduction
   • "Build, compute, critique, repeat: Data analysis with latent variable models" (Blei, 2014)
   • Slides
2. The ingredients of probabilistic models
   • "Model-based machine learning" (Bishop, 2013)
   • "Some issues in the foundations of statistics" (Freedman, 1994)
   • "The Elements of Statistical Learning" (Chapter 3, Chapter 7) (Hastie et al., 2009)
3. Bayesian mixture models and the Gibbs sampler
   • "Identifying Bayesian mixture models" (Betancourt, 2018)
   • "Probabilistic inference using Markov chain Monte Carlo methods" (Sections 1-4) (Neal, 1993)
4. Mixed-membership models, topic models, and variational inference
   • "Probabilistic topic models" (Blei, 2012)
   • "Applications of topic models" (Boyd-Graber et al., 2017)
   • "Variational inference: A review for statisticians" (Blei et al., 2017)
5. Matrix factorization and efficient MAP inference
   • "Matrix factorization techniques for recommender systems" (Koren et al., 2009)
   • "Learning the parts of objects by non-negative matrix factorization" (Lee and Seung, 1999)
   • "Scalable Recommendation with Hierarchical Poisson Factorization" (Gopalan et al., 2015)
   • "Tensor decompositions and applications" (Kolda and Bader, 2009)
6. Deep generative models and black box variational inference
   • "Stochastic backpropagation and approximate inference in deep generative models" (Rezende et al., 2014)
   • "Autoencoding variational Bayes" (Kingma and Welling, 2013)
   • "Deep exponential families" (Ranganath et al., 2015)
   • "Black box variational inference" (Ranganath et al., 2014)
   • "Mean field theory for sigmoid belief networks" (Saul et al., 1996)
   • "Representation learning: A review and new perspectives" (Bengio et al., 2013)
7. Exponential families, conjugate priors, and generalized linear models
   • "The exponential family" (Bishop, 2006; Section 2.4)
   • "An outline of generalized linear models" (McCullagh and Nelder, 1989; Chapter 2)
   • "Conjugate priors for exponential families" (Diaconis and Ylvisaker, 1979)
   • "Exponential families in theory and practice" (Efron, 2018)
8. Hierarchical models, robust models, and empirical Bayes
   • "Multi-level structures" (Gelman and Hill, 2007; Chapter 11)
   • "Multi-level linear models: The basics" (Gelman and Hill, 2007; Chapter 12)
   • "Bayes, oracle Bayes, and empirical Bayes" (Efron, 2017)
9. The theory of graphical models
   • "Conditional independence and factorization" (Jordan, 2003; Chapter 2)
   • "The elimination algorithm" (Jordan, 2003; Chapter 3)
   • "Probability propagation and factor graphs" (Jordan, 2003; Chapter 4)
10. Advanced topics in variational inference
    • "Graphical models, exponential families, and variational inference" (Waingwright and Jordan, 2008)
    • "Covariance, robustness, and variational Bayes" (Broderick et al., 2018)
11. Model checking and model diagnosis
    • "Bayesianly justifiable and relevant frequency calculations for the applied statistician" (Rubin, 1984)
    • "Posterior predictive assessment of model fitness via realized discrepancies" (Gelman et al., 1996)
    • "Philosophy and the practice of Bayesian statistics" (Gelman and Shalizi, 2013)
12. An introduction to causality
    • "Statistics and causal inference" (Holland, 1986)
    • "Causal inference in statistics: An overview" (Pearl, 2009)