Add to GMail
Foundation Models for Robotic Manipulation: Opportunities and Challenges
11:40 AM to 12:55 PM
CSB 451 CS Auditorium
Yunzhu Li, University of Illinois Urbana-Champaign
Abstract:
Foundation models, such as GPT-4 Vision, have marked significant achievements in the fields of natural language and vision, demonstrating exceptional abilities to adapt to new tasks and scenarios. However, physical interaction—such as cooking, cleaning, or caregiving—remains a frontier where foundation models and robotic systems have yet to achieve the desired level of adaptability and generalization. In this talk, I will discuss the opportunities for incorporating foundation models into classic robotic pipelines to endow robots with capabilities beyond those achievable with traditional robotic tools. The talk will focus on three key improvements in (1) task specification, (2) low-level, and (3) high-level scene modeling. The core idea behind this series of research is to introduce novel representations and integrate structural priors into robot learning systems, incorporating the commonsense knowledge learned from foundation models to achieve the best of both worlds. I will demonstrate how such integration allows robots to interpret instructions given in free-form natural language and enables category-level generalization for free. I will also show how the foundation models can be augmented with additional memory mechanisms, like an action-conditioned scene graph, for a wide range of real-world manipulation tasks involving rigid, articulated, and nested objects (e.g., Matryoshka dolls), and deformable objects. Towards the end of the talk, I will discuss the limitations of the current foundation models, challenges that still lie ahead, and potential avenues to address these challenges.
Class Day Graduate Ceremony
3:00 PM to 5:00 PM
South Lawn, Morningside Campus
Countdown to Graduate Class Day
A celebration of the accomplishments of all graduate degree candidates of the Fu Foundation School of Engineering and Applied Sciences.
Class Day Undergraduate Ceremony
11:45 AM to 1:30 PM
South Lawn, Morningside Campus
Countdown to Undergraduate Class Day
A celebration of the accomplishments of bachelor's degree candidates of the Fu Foundation School of Engineering and Applied Sciences.
CS Awards Ceremony and Celebration
1:00 PM to 4:00 PM
CSB 451 CS Auditorium
This event celebrates all our graduating Undergraduates, Masters, and Doctoral students with the CS award ceremony in CSB 451 (Lecture Hall). We will have food and refreshments at the CS lounge, Traub Conference Room, and courtyard.
Add to GMail