Topics and readings

Below are topics of the class and some readings about each. These topics and readings are subject to change.

The readings are at different levels: some are basic and some are advanced. They provide foundational and other interesting material about the topics. The lectures will not necessarily cover all of this material.

The main text is the forthcoming book "Probabilistic Models and Machine Learning" by David M. Blei. We will provide a draft to enrolled students. These readings supplement the book.

1. The ingredients of probabilistic models
   • "Build, compute, critique, repeat: Data analysis with latent variable models" (Blei, 2014)
   • "Probabilistic machine learning and artificial intelligence" (Ghahramani, 2015)
   • "Model-based machine learning" (Bishop, 2013)
   • "Some issues in the foundations of statistics" (Freedman, 1994)
2. Linear and Logistic Regression
   • "Advanced Data Analysis from an Elementary Point of View" (relevant chapters) (Shalizi, 2022)
   • "Regression and Other Stories" (relevant chapters) (Gelman et al., 2020)
   • "Elements of Statistical Learning" (relevant chapters) (Tibshirani et al., 2009)
3. Stochastic Optimization
   • "Optimization" (Chapter 5 of Recht and Hardt)
   • "Variance-reduced methods for machine learning" (Gower et al., 2020)
   • "Optimization methods for large-scale machine learning" (Bottou et al., 2016)
4. Markov chain Monte Carlo
   • "The No-U-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo" (Hoffman and Gelman, 2014)
   • "Probabilistic inference using Markov chain Monte Carlo methods" (Neal, 1993)
5. Topic models and mixed-membership models
   • "Probabilistic topic models" (Blei, 2012)
   • "Applications of topic models" (Boyd-Graber et al., 2017)
6. Introduction to variational inference
   • "Variational inference: A review for statisticians" (Blei et al., 2017)
   • "An introduction to variational methods for graphical models" (Jordan et al., 1999)
7. Matrix factorization and efficient MAP inference
   • "Matrix factorization techniques for recommender systems" (Koren et al., 2009)
   • "Learning the parts of objects by non-negative matrix factorization" (Lee and Seung, 1999)
   • "Scalable recommendation with hierarchical Poisson factorization" (Gopalan et al., 2015)
   • "Tensor decompositions and applications" (Kolda and Bader, 2009)
8. Exponential families, conjugate priors, and generalized linear models
   • "The exponential family" (Bishop, 2006; Section 2.4)
   • "An outline of generalized linear models" (McCullagh and Nelder, 1989; Chapter 2)
   • "Conjugate priors for exponential families" (Diaconis and Ylvisaker, 1979)
   • "Exponential families in theory and practice" (Efron, 2018)
9. Hierarchical models, robust models, and empirical Bayes
   • "Multi-level structures" (Gelman and Hill, 2007; Chapter 11)
   • "Multi-level linear models: The basics" (Gelman and Hill, 2007; Chapter 12)
   • "Bayes, oracle Bayes, and empirical Bayes" (Efron, 2019)
10. Deep probabilistic models
    • "Stochastic backpropagation and approximate inference in deep generative models" (Rezende et al., 2014)
    • "Autoencoding variational Bayes" (Kingma and Welling, 2013)
    • "Posterior collapse and latent-variable non-identifiability" (Wang et al., 2022)
    • "Deep exponential families" (Ranganath et al., 2015)
    • "Mean field theory for sigmoid belief networks" (Saul et al., 1996)
    • "Representation learning: A review and new perspectives" (Bengio et al., 2013)
11. Generative artificial intelligence
    • "Improving language understanding by generative pre-training" (Radford et al., 2018)
    • "Direct preference optimization: Your language model is secretly a reward model" (Rafailov et al., 2023)
    • "Deep unsupervised learning using nonequilibrium thermodynamics" (Sohl-Dickstein et al., 2015)
    • "Denoising diffusion probabilistic models" (Ho et al., 2020)
12. Advanced topics in variational inference
    • "Black box variational inference" (Ranganath et al., 2014)
    • "Graphical models, exponential families, and variational inference" (Wainwright and Jordan, 2008)
    • "Monte Carlo gradient estimation in machine learning" (Mohamed e al., 2019)
    • "Automatic differentiation variational inference" (Kucukelbir et al., 2017)
    • "Covariance, robustness, and variational Bayes" (Broderick et al., 2018)
    • "ELBO surgery: Yet another way to carve up the variational evidence lower bound" (Hoffman and Johnson, 2016)
    • "An optimization-centric view on Bayes' rule: Reviewing and generalizing variational inference" (Knoblauch et al., 2022)
13. The theory of graphical models
    • "Conditional independence and factorization" (Jordan, 2003; Chapter 2)
    • "The elimination algorithm" (Jordan, 2003; Chapter 3)
    • "Probability propagation and factor graphs" (Jordan, 2003; Chapter 4)
14. Model criticism and model diagnosis
    • "Bayesianly justifiable and relevant frequency calculations for the applied statistician" (Rubin, 1984)
    • "Posterior predictive assessment of model fitness via realized discrepancies" (Gelman et al., 1996)
    • "Philosophy and the practice of Bayesian statistics" (Gelman and Shalizi, 2013)
    • "Population predictive checks" (Moran et al., 2022)
    • "The posterior predictive null" (Moran et al., 2022)
15. Simulation-based inference
    • "The frontier of simulation-based inference" (Cranmer et al., 2020)
    • "Simulation-based inference: A practical guide" (Deistler et al., 2025)
    • "Benchmarking simulation-based inference" (Lueckmann et al., 2021)
16. Bayesian optimization
    • "Bayesian optimization" (Garnett, 2023)
    • "A tutorial on Bayesian optimization" (Frazier, 2018)
17. An introduction to causality
    • "Statistics and causal inference" (Holland, 1986)
    • "Causal inference in statistics: An overview" (Pearl, 2009)