My Photo   Statistical Methods and Computation: SDGB 7844


Description

This main theme of the course will be algebraic connections between modern statistics and computation. Participants will implement numerical algorithms and learn to use software resources in lab sessions. Topics will include solving linear systems and differential equations, LU decomposition, pseudoinverses, and finite difference methods, transformations and distributions, simulating random walks, sampling and markov chains, numerical integration, quadrature, tensors as multilinear functions, tensor decompositions, the power method, Jennrich's algorithm, Prony method, symbolic and automatic differentiation. If time permits we will work through tutorials on advanced topics in computational statistics including markov chain monte carlo, variational inference and deep learning.

Prerequisites

An introductory background in statistics (some exposure to linear algebra, multivariate calculus and probability).
Textbooks

While no prior programming experience is required and we will introduce the fundamentals, one learns best by doing. The syntax and semantics of any programming language will thus not be the emphasis of the course. Watch any videos and attempt tutorials posted before class. Come prepared to work with a laptop to best use the time.

  1. Introduction to Computation and Programming by Guttag (recommended)
  2. Introducing Monte Carlo Methods by Robert, Casella (more focus on statistics)
  3. Introduction to Scientific Programming and Simulation by Jones et al (introductory R textbook)
Additional Resources

Python for Scientific Computing
Anaconda Distribution
Anaconda - Installation and Using Conda
Numerical and Scientific Computing with Python
IPython Notebook
Ten Minutes to Pandas
Theano
Tensorflow
Duke's Computational Statistics Notes
R Programming and R Studio
Grade Distribution

  • Assignments 30%
  • Participation 30%
  • Final Project 40%
Homework

Please use LaTeX to typeset your work. Visit the www.latex-project.org to download and familiarize yourself with this tutorial to get started. You may use the following template (cls, tex, pdf). It is highly recommended that you use Python and the Jupyter Notebook. Start early on all programming assignments as these take time. Standard academic honesty policy applies.
Tentative Course Outline

  • Overview, Defining Functions, Least Squares
  • Gaussian Elimination, Determinants, Pseudoinverses, Numpy
  • Finding Roots, Gradient Descent and Newton's Method
  • Stochastic Optimization, Natural Gradients
  • Euler's Method, Finite Differences
  • Quadrature, Laplace Approximation
  • Monte Carlo, Importance and Rejection Sampling
  • Markov Chains and Metropolis Hastings
  • Tensor Decompositions, Jennrich's Algorithm
  • Hamiltonian Monte Carlo
  • Stochastic Variational Inference
  • Variational Autoencoders
  • Symbolic and Automatic Differentiation


  • Date Topic(s) Reading Notes Assignments Solutions
    9.5 Overview Guttag or Jones 1 Intro to Scientific Computing
    Gaussian Elimination
    Solving Linear Systems
    Warm Up
    9.12 Least Squares
    Gaussian Integrals
    MLE
    Guttag or Jones 2 Normal Equation
    How Gauss Determined Orbit of Ceres
    Homework 0
    Least Squares Tutorial
    Solution
    9.19 LU Decomposition
    Parity and Inversions
    Determinants
    Guttag or Jones 3 Permuations and Determinants
    Landau Notation
    Elimination Tutorial
    9.26 Finding Roots
    Gradient Descent
    Newton's Method
    Guttag or Jones 4 First and Second Order Approximations
    Newton Rhapson
    Unconstrained Minimization
    Optimization Tutorial
    10.3 Maximum Likelihood Estimation
    Logistic Regression
    Numerical Optimization
    Guttag or Jones 5 Maximum Likelihood Estimation
    Logistic Regression
    Homework 1
    data
    Solution
    10.10 Bayesian Inference
    Ridge Regression
    Lagrange Multipliers
    Guttag or Jones 6 MAP Estimation
    Regularized Regression
    Lagrange Multipliers
    10.17 Curve Fitting
    Vandermonte Matrices
    Bias and Variance
    Guttag or Jones 7 Vandermonte Matrices
    Curve Fitting
    Pseudoinverses
    10.24 Tutorial on Assignment 1 Guttag or Jones 8
    10.31 Numerical Integration
    Quadrature
    Newton Coates and Gauss Hermite
    Hermite Polynomials
    Guttag or Jones 9 Monte Carlo
    Quadrature
    Hermite Polynomials
    11.7 Quadrature
    Monte Carlo
    Laplace Approximation
    Guttag or Jones 10 Importance Sampling
    Law of Large Numbers
    Laplace Approx