DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics

Overview

DiffVL combines vision and language with differentiable physics to tackle soft-body manipulation. The framework connects language-conditioned goals with visual observations, helping specify what a robot should do with deformable objects.

Making tasks easier to specify

The work also explores how a user interface and large language models can help non-experts set up tasks and datasets, reducing the manual effort involved in defining manipulation problems.

My contribution

I developed the vision-language-driven differentiable physics framework for soft-body manipulation during my research at MIT.

The work was accepted to NeurIPS 2023.