“Feature Fields for Robotics: Language-Grounded Perception and Mapping at Multiple Scales”

ABSTRACT
What kind of representation do robots need in order to be as generally capable as humans in handling unseen scenarios? Recent work in vision and vision-language foundation models has become quite good at telling what is in a scene, but they do not capture the geometry needed for handling physical contact. State-of-the-art methods in inverse graphics capture detailed 3D geometry, but they are missing the semantics. In this talk, I will present a way to combine accurate 3D geometry with rich semantics into a single representation format called distilled feature fields and ways to use this representation for perception during few-shot manipulation with a robotic arm. Using features sourced from the vision-language model, CLIP, our method allows the user to designate novel objects for manipulation via free-text natural language, and can generalize to unseen expressions and novel categories of objects. I will also present ways to scale feature fields up for building maps and the dual purpose of building realistic physics simulators for reinforcement learning. Finally, I will present our recent effort in building a unified representation for semantics, geometry, and physics called Feature Splatting.

PRESENTER
Ge wants to make dramatic improvements to intelligent robots by rethinking ways to integrate perception with control and finding better and more general ways to represent motor skills and the surrounding physical environment. Ge’s past work includes fundamental contributions to deep Q-learning, self-supervised representation learning for decision-making systems, and agile locomotion on legged robots. Ge’s recent work on open-ended generalization in robotic manipulation won The Best Paper Award at CoRL 2023.
Ge obtained his Ph.D. in condensed matter physics from the University of Chicago. He was a visiting student with Pieter Abbeel at Berkeley and a research scientist intern at Google DeepMind and Facebook AI Research. He is currently a postdoc fellow with the NSF Institute of AI and Fundamental Interactions (IAIFI) and spends most of his time at MIT CSAIL.