On the Evaluation of Generative Robotic Simulations

Overview

EvalTask introduces a framework for evaluating generative robotic simulations. Rather than considering generated tasks in isolation, it examines their quality, diversity, and ability to generalize.

What we evaluate

• Task realism: whether generated tasks form meaningful robotic scenarios.
• Trajectory completeness: whether the generated trajectories capture the task adequately.
• Generalization: where current generative simulation pipelines encounter limitations.

My contribution

I designed the evaluation framework and developed automatic LLM/VLM-based metrics. I validated the scores against human judgments and identified generalization bottlenecks in current pipelines.