Carl Vondrick - Columbia University - Computer Vision and Machine Learning

Research

Our group studies computer vision and machine learning. By training machines to observe and interact with their surroundings, we aim to create robust and versatile models for perception. We often investigate visual models that capitalize on large amounts of unlabeled data and transfer across tasks and modalities. Other interests include scene dynamics, sound and language and beyond, interpretable models, and perception for robotics.

Joining Us

We are always looking for people with unusual creativity, strong technical skills, and passion for hard problems.

Postdoc: We have openings for a postdoc in computer vision broadly. To apply, please reach out to me with your CV and research statement.

Prospective PhD students: Please apply here and mention my name on the application. Every year, the lab accepts one or two PhD students. Due to volume, we unfortunately cannot respond to emails about PhD admissions.

Current Columbia undergrad or MS students: Please email me directly with your CV. We generally recruit students who have taken relevant computer vision and machine learning courses.

External Visitors: We do not have openings for external visitors from different institutions at this time. We unfortunately are unable to respond to emails about this.

News

Congratulations to Dr. Chengzhi Mao for successfully defending his PhD!
Congratulations to Arjun, Basile, Chengzhi, Didac, Ishaan, Mia, Purva, Ruoshi and Sachit for their ICCV 2023 papers!
Congratulations to Chengzhi for his ICML 2023 paper!
Congratulations to Scott for winning the NSF graduate research fellowship!
Congratulations to Ruoshi, Purva, Dídac, Basile, Sachit, Chengzhi, and Revant on their CVPR 2023 papers!
Congratulations to Sachit, Chengzhi, and Scott for their ICLR 2023 papers!

PhD Students & Postdoctoral Fellows

Arjun Mani
NSF PhD Fellow

Basile Van Hoorick
PhD Student

Chengzhi Mao
Postdoctoral Fellow

Dídac Surís
Microsoft PhD Fellow

Mia Chiquier
Amazon PhD Fellow

Purva Tendulkar
Presidential PhD Fellow

Ruoshi Liu
PhD Student

Sachit Menon
NSF PhD Fellow
Presidential PhD Fellow

Sruthi Sudhakar
NSF PhD Fellow

Utkarsh Mall
Postdoctoral Fellow

Alumni

Undergraduate/Masters

Ishaan Preetam Chandratreya, B.S. 2023 → PhD at MIT
Revant Teotia, M.S. 2023 → PhD at NYU
Scott Geng, B.S. 2023, NSF Fellowship → PhD at UW
Hui Lu, M.S. + B.S. 2022 → Facebook
Jillian Ross, B.S. 2021 → PhD at MIT
Amogh Gupta, M.S. 2020 → Amazon
Dave Epstein, B.S. 2020, CRA Honorable Mention → PhD at UC Berkeley

All Papers

Representative Papers

2023

ViperGPT: Visual Inference via Python Execution for Reasoning
Dídac Surís*, Sachit Menon*, Carl Vondrick
ICCV 2023 (Oral)
Paper Project Page Code

Zero-1-to-3: Zero-shot One Image to 3D Object
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, Carl Vondrick
ICCV 2023
Paper Project Page Code Demo

Muscles in Action
Mia Chiquier, Carl Vondrick
ICCV 2023
Paper Project Page

SurfsUp: Learning Fluid Simulation for Novel Surfaces
Arjun Mani*, Ishaan Preetam Chandratreya*, Elliot Creager, Carl Vondrick, Richard Zemel
ICCV 2023
Paper Project Page

Landscape Learning for Neural Network Inversion
Ruoshi Liu, Chengzhi Mao, Purva Tendulkar, Hao Wang, Carl Vondrick
ICCV 2023
Paper Blog Post

SHIFT3D: Synthesizing Hard Inputs For Tricking 3D Detectors
Hongge Chen, Zhao Chen, Greg Meyer, Dennis Park, Carl Vondrick, Ashish Shrivastava, and Yuning Chai
ICCV 2023
Paper

Objaverse-XL: A Universe of 10M+ 3D Objects
Matt Deitke, et al.
arXiv 2023
Paper

Robust Perception through Equivariance
Chengzhi Mao, Lingyu Zhang, Abhishek Joshi, Junfeng Yang, Hao Wang, Carl Vondrick
ICML 2023
Paper Project Page

Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection
Ruoshi Liu, Carl Vondrick
CVPR 2023
Paper Project Page

What You Can Reconstruct from a Shadow
Ruoshi Liu, Sachit Menon, Chengzhi Mao, Dennis Park, Simon Stent, Carl Vondrick
CVPR 2023
Paper Blog Post

Tracking through Containers and Occluders in the Wild
Basile Van Hoorick, Pavel Tokmakov, Simon Stent, Jie Li, Carl Vondrick
CVPR 2023
Paper Project Page Datasets Code

FLEX: Full-Body Grasping Without Full-Body Grasps
Purva Tendulkar, Dídac Surís, Carl Vondrick
CVPR 2023
Paper Project Page

Doubly Right Object Recognition: A Why Prompt for Visual Rationales
Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, Carl Vondrick
CVPR 2023
Paper

Affective Faces for Goal-Driven Dyadic Communication
Scott Geng*, Revant Teotia*, Purva Tendulkar, Sachit Menon, Carl Vondrick
arXiv 2023
Paper Project Page

Visual Classification via Description from Large Language Models
Sachit Menon, Carl Vondrick
ICLR 2023 (Oral)
Paper Project Page Code Demo

Understanding Zero-Shot Adversarial Robustness for Large-Scale Models
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, Carl Vondrick
ICLR 2023
Paper

2022

Adversarially Robust Video Perception by Seeing Motion
Lingyu Zhang*, Chengzhi Mao*, Junfeng Yang, Carl Vondrick
arXiv 2022
Paper Project Page

Task Bias in Vision-Language Models
Sachit Menon*, Ishaan Preetam Chandratreya*, Carl Vondrick
arXiv 2022
Paper

Private Multiparty Perception for Navigation
Hui Lu, Mia Chiquier, Carl Vondrick
NeurIPS 2022
Paper Project Page Code

Representing Spatial Trajectories as Distributions
Dídac Surís, Carl Vondrick
NeurIPS 2022
Paper Project Page

Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse
Sachit Menon, David Blei, Carl Vondrick
UAI 2022
Paper

Revealing Occlusions with 4D Neural Fields
Basile Van Hoorick, Purva Tendulkar, Dídac Surís, Dennis Park, Simon Stent, Carl Vondrick
CVPR 2022 (Oral)
Paper Project Page Talk

Globetrotter: Connecting Languages by Connecting Images
Dídac Surís, Dave Epstein, Carl Vondrick
CVPR 2022 (Oral)
Paper Project Page Code

Causal Transportability for Visual Recognition
Chengzhi Mao*, Kevin Xia*, James Wang, Hao Wang, Junfeng Yang, Elias Bareinboim, Carl Vondrick
CVPR 2022
Paper

It's Time for Artistic Correspondence in Music and Video
Dídac Surís, Carl Vondrick, Bryan Russell, Justin Salamon
CVPR 2022
Paper Project Page

UnweaveNet: Unweaving Activity Stories
Will Price, Carl Vondrick, Dima Damen
CVPR 2022
Paper

There is a Time and Place for Reasoning Beyond the Image
Xingyu Fu, Ben Zhou, Ishaan Preetam Chandratreya, Carl Vondrick, Dan Roth
ACL 2022 (Oral)
Paper Code + Data

Real-Time Neural Voice Camouflage
Mia Chiquier, Chengzhi Mao, Carl Vondrick
ICLR 2022 (Oral)
Paper Project Page Science

Discrete Representations Strengthen Vision Transformer Robustness
Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, Irfan Essa
ICLR 2022
Paper

2021

Full-Body Visual Self-Modeling of Robot Morphologies
Boyuan Chen, Robert Kwiatkowski, Carl Vondrick, Hod Lipson
Science Robotics 2022
Paper Project Page Code

The Boombox: Visual Reconstruction from Acoustic Vibrations
Boyuan Chen, Mia Chiquier, Hod Lipson, Carl Vondrick
CoRL 2021
Paper Project Page Video Overview

Adversarial Attacks are Reversible with Natural Supervision
Chengzhi Mao, Mia Chiquier, Hao Wang, Junfeng Yang, Carl Vondrick
ICCV 2021
Paper Code

Dissecting Image Crops
Basile Van Hoorick, Carl Vondrick
ICCV 2021
Paper Code

Learning the Predictability of the Future
Dídac Surís*, Ruoshi Liu*, Carl Vondrick
CVPR 2021
Paper Project Page Code Models Talk

Generative Interventions for Causal Learning
Chengzhi Mao, Amogh Gupta, Augustine Cha, Hao Wang, Junfeng Yang, Carl Vondrick
CVPR 2021
Paper Code

Learning Goals from Failure
Dave Epstein, Carl Vondrick
CVPR 2021
Paper Project Page Data Code Talk

Visual Behavior Modelling for Robotic Theory of Mind
Boyuan Chen, Carl Vondrick, Hod Lipson
Scientific Reports 2021
Paper Project Page

2020

Listening to Sounds of Silence for Speech Denoising
Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, Changxi Zheng
NeurIPS 2020
Paper Project Page

Multitask Learning Strengthens Adversarial Robustness
Chengzhi Mao, Amogh Gupta, Vikram Nitin, Baishakhi Ray, Shuran Song, Junfeng Yang, Carl Vondrick
ECCV 2020 (Oral)
Paper

We Have So Much In Common: Modeling Semantic Relational Set Abstractions in Videos
Alex Andonian, Camilo Fosco, Mathew Monfort, Allen Lee, Carl Vondrick, Rogerio Feris, Aude Oliva
ECCV 2020
Paper Project Page

Learning to Learn Words from Visual Scenes
Dídac Surís*, Dave Epstein*, Heng Ji, Shih-Fu Chang, Carl Vondrick
ECCV 2020
Paper Project Page Code Talk

Oops! Predicting Unintentional Action in Video
Dave Epstein, Boyuan Chen, Carl Vondrick
CVPR 2020
Paper Project Page Data Code Talk

2019

Metric Learning for Adversarial Robustness
Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, Baishakhi Ray
NeurIPS 2019
Paper Code

VideoBERT: A Joint Model for Video and Language Representation Learning
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, Cordelia Schmid
ICCV 2019
Paper Blog

Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, Shih-Fu Chang
CVPR 2019
Paper Code

Relational Action Forecasting
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Rahul Sukthankar, Kevin Murphy, Cordelia Schmid
CVPR 2019 (Oral)
Paper

Moments in Time Dataset: one million videos for event understanding
Mathew Monfort et al
PAMI 2019
Paper Project Page

2018

Tracking Emerges by Colorizing Videos
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, Kevin Murphy
ECCV 2018
Paper Blog

The Sound of Pixels
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, Antonio Torralba
ECCV 2018
Paper Project Page

Actor-centric Relation Network
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, Cordelia Schmid
ECCV 2018
Paper

AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
Chunhui Gu et al
CVPR 2018 (Spotlight)
Paper Project Page

2017

Following Gaze in Video
Adria Recasens, Carl Vondrick, Aditya Khosla, Antonio Torralba
ICCV 2017
Paper

Generating the Future with Adversarial Transformers
Carl Vondrick, Antonio Torralba
CVPR 2017
Paper Project Page

Cross-Modal Scene Networks
Yusuf Aytar*, Lluis Castrejon*, Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
PAMI 2017
Paper Project Page

See, Hear, and Read: Deep Aligned Representations
Yusuf Aytar, Carl Vondrick, Antonio Torralba
arXiv 2017
Paper Project Page

2016

Generating Videos with Scene Dynamics
Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
NeurIPS 2016
Paper Project Page Code NBC Scientific American New Scientist MIT News

SoundNet: Learning Sound Representations from Unlabeled Video
Yusuf Aytar*, Carl Vondrick*, Antonio Torralba
NeurIPS 2016
Paper Project Page Code NPR New Scientist Week Junior MIT News

Anticipating Visual Representations with Unlabeled Video
Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
CVPR 2016 (Spotlight)
Paper Project Page NPR CNN AP Wired Stephen Colbert MIT News

Predicting Motivations of Actions by Leveraging Text
Carl Vondrick, Deniz Oktay, Hamed Pirsiavash, Antonio Torralba
CVPR 2016
Paper Dataset

Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Lluis Castrejon*, Yusuf Aytar*, Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
CVPR 2016
Paper Project Page Demo

Visualizing Object Detection Features
Carl Vondrick, Aditya Khosla, Hamed Pirsiavash, Tomasz Malisiewicz, Antonio Torralba
IJCV 2016
Paper Project Page Slides MIT News

2015

Do We Need More Training Data?
Xiangxin Zhu, Carl Vondrick, Charless C. Fowlkes, Deva Ramanan
IJCV 2015
Paper Dataset

Learning Visual Biases from Human Imagination
Carl Vondrick, Hamed Pirsiavash, Aude Oliva, Antonio Torralba
NeurIPS 2015
Paper Project Page Technology Review

Where are they looking?
Adria Recasens*, Aditya Khosla*, Carl Vondrick, Antonio Torralba
NeurIPS 2015
Paper Project Page Demo

2014

Assessing the Quality of Actions
Hamed Pirsiavash, Carl Vondrick, Antonio Torralba
ECCV 2014
Paper Project Page

2013

HOGgles: Visualizing Object Detection Features
Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, Antonio Torralba
ICCV 2013 (Oral)
Paper Project Page Slides MIT News

2012

Do We Need More Training Data or Better Models for Object Detection?
Xiangxin Zhu, Carl Vondrick, Deva Ramanan, Charless C. Fowlkes
BMVC 2012
Paper Dataset

Efficiently Scaling Up Crowdsourced Video Annotation
Carl Vondrick, Donald Patterson, Deva Ramanan
IJCV 2012
Paper Project Page

2011

Video Annotation and Tracking with Active Learning
Carl Vondrick, Deva Ramanan
NeurIPS 2011
Paper Project Page

A Large-scale Benchmark Dataset for Event Recognition
Sangmin Oh, et al.
CVPR 2011
Paper Project Page

2010

Efficiently Scaling Up Video Annotation with Crowdsourced Marketplaces
Carl Vondrick, Deva Ramanan, Donald Patterson
ECCV 2010
Paper Project Page

Teaching

Computer Vision II, Spring 2023
Representation Learning, Fall 2022
Computer Vision II, Spring 2022
Representation Learning, Fall 2021
Computer Vision II, Summer 2021
Representation Learning, Fall 2020
Computer Vision, Fall 2019
Advanced Computer Vision, Spring 2019
Computer Vision, Fall 2018

Funding

National Science Foundation
Defense Advanced Research Projects Agency
Toyota Research Institute
Amazon Research
Google