Events

University Commencement

10:30 AM to 6:30 PM

Morningside Campus

Columbia University’s Commencement ceremony of the 272nd academic year will take place on Wednesday, May 20, 2026, with two celebrations held on the Morningside Campus: one for graduate degree candidates and one for undergraduate degree candidates.

Graduate Ceremony 10:30 a.m. - 12:00 p.m.
Undergraduate Ceremony 5:00 - 6:30 p.m.

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

2:00 PM to 3:00 PM

CSB 480

Phillip Isola, MIT

Abstract:
Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse, task-improving specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples N parameter perturbations at random, selects the top K, and ensembles predictions via majority vote. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.

Bio:
Phillip Isola is an associate professor in EECS at MIT. He studies computer vision, machine learning, robotics, and AI. He completed his Ph.D. in Brain & Cognitive Sciences at MIT, and has since spent time at UC Berkeley, OpenAI, and Google Research. His current research focuses on trying to scientifically understand human-like intelligence.