Carl Vondrick

Lab Members

Parita Pooj (PhD)
Boyuan Chen (PhD, with Hod Lipson)
Yiliang Shi (PhD, with Eugene Wu)
Dídac Surís (PhD, in Fall 2019)
Dave Epstein (Undergrad)
Suman Mulumudi (Undergrad)
James Shin (Undergrad)

Teaching

Talks

Funding

NSF CRII, 2019-2021
Amazon Gift, 2018-2019

Service

Area Chair, CVPR 2018-2020
Area Chair, NeurIPS 2019
Area Chair, ICML 2019
Organizer, Workshop on Multimodal Video Analysis, ICCV 2019
Organizer, Workshop on Self-supervised Learning, ICML 2019
Organizer, Workshop on Learning from Unlabeled Video, CVPR 2019
Organizer, Tutorial on Unsupervised Visual Learning, CVPR 2018

Code and Data

Press

Research

Our group focuses on computer vision and machine learning. By training machines to observe and interact with their surroundings, we believe we can create robust and versatile models for perception. We often develop visual models that capitalize on large amounts of unlabeled data and transfer across tasks and modalities. Other interests include sound and language, interpretable models, high-level reasoning, and perception for robotics.

News

Postdoc opening: The computer vision group is looking for a postdoctoral fellow. More info here.
We are organizing some upcoming workshops on video and self-supervised learning:
- Multimodal Video Analysis at ICCV 2019.
- Self-supervised Learning at ICML 2019.
- Learning from Unlabeled Videos at CVPR 2019.
Blog post: by learning to colorize videos, visual tracking emerges.
Checkout our new audio-visual source separation demo.

All Papers

Representative Papers

2019

VideoBERT: A Joint Model for Video and Language Representation Learning New!
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, Cordelia Schmid
arXiv 2019
Paper

Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding New!
Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, Shih-Fu Chang
CVPR 2019
Paper

Relational Action Forecasting New!
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Rahul Sukthankar, Kevin Murphy, Cordelia Schmid
CVPR 2019
Coming Soon

Moments in Time Dataset: one million videos for event understanding New!
Mathew Monfort et al
PAMI 2019
Paper Project Page

2018

Tracking Emerges by Colorizing Videos New!
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, Kevin Murphy
ECCV 2018
Paper Blog

The Sound of Pixels New!
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, Antonio Torralba
ECCV 2018
Paper Project Page

Actor-centric Relation Network New!
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, Cordelia Schmid
ECCV 2018
Paper

AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
Chunhui Gu et al
CVPR 2018
Paper Project Page

2017

Following Gaze in Video
Adria Recasens, Carl Vondrick, Aditya Khosla, Antonio Torralba
ICCV 2017
Paper

Generating the Future with Adversarial Transformers
Carl Vondrick, Antonio Torralba
CVPR 2017
Paper Project Page

Cross-Modal Scene Networks
Yusuf Aytar*, Lluis Castrejon*, Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
PAMI 2017
Paper Project Page

See, Hear, and Read: Deep Aligned Representations
Yusuf Aytar, Carl Vondrick, Antonio Torralba
arXiv 2017
Paper Project Page

2016

Generating Videos with Scene Dynamics
Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
NeurIPS 2016
Paper Project Page Code NBC Scientific American New Scientist MIT News

SoundNet: Learning Sound Representations from Unlabeled Video
Yusuf Aytar*, Carl Vondrick*, Antonio Torralba
NeurIPS 2016
Paper Project Page Code NPR New Scientist Week Junior MIT News

Anticipating Visual Representations with Unlabeled Video
Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
CVPR 2016
Paper Project Page NPR CNN AP Wired Stephen Colbert MIT News

Predicting Motivations of Actions by Leveraging Text
Carl Vondrick, Deniz Oktay, Hamed Pirsiavash, Antonio Torralba
CVPR 2016
Paper Dataset

Who is Mistaken?
Benjamin Eysenbach, Carl Vondrick, Antonio Torralba
arXiv 2016
Paper Project Page

Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Lluis Castrejon*, Yusuf Aytar*, Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
CVPR 2016
Paper Project Page Demo

Visualizing Object Detection Features
Carl Vondrick, Aditya Khosla, Hamed Pirsiavash, Tomasz Malisiewicz, Antonio Torralba
IJCV 2016
Paper Project Page Slides MIT News

2015

Do We Need More Training Data?
Xiangxin Zhu, Carl Vondrick, Charless C. Fowlkes, Deva Ramanan
IJCV 2015
Paper Dataset

Learning Visual Biases from Human Imagination
Carl Vondrick, Hamed Pirsiavash, Aude Oliva, Antonio Torralba
NeurIPS 2015
Paper Project Page Technology Review

Where are they looking?
Adria Recasens*, Aditya Khosla*, Carl Vondrick, Antonio Torralba
NeurIPS 2015
Paper Project Page Demo

2014

Assessing the Quality of Actions
Hamed Pirsiavash, Carl Vondrick, Antonio Torralba
ECCV 2014
Paper Project Page

2013

HOGgles: Visualizing Object Detection Features
Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, Antonio Torralba
ICCV 2013
Paper Project Page Slides MIT News

2012

Do We Need More Training Data or Better Models for Object Detection?
Xiangxin Zhu, Carl Vondrick, Deva Ramanan, Charless C. Fowlkes
BMVC 2012
Paper Dataset

Efficiently Scaling Up Crowdsourced Video Annotation
Carl Vondrick, Donald Patterson, Deva Ramanan
IJCV 2012
Paper Project Page

2011

Video Annotation and Tracking with Active Learning
Carl Vondrick, Deva Ramanan
NeurIPS 2011
Paper Project Page

A Large-scale Benchmark Dataset for Event Recognition
Sangmin Oh, et al.
CVPR 2011
Paper Project Page

2010

Efficiently Scaling Up Video Annotation with Crowdsourced Marketplaces
Carl Vondrick, Deva Ramanan, Donald Patterson
ECCV 2010
Paper Project Page