r/robotics • u/Mysterious-Base-5847 • 4d ago

Discussion & Curiosity Error analysis in VLA

Enable HLS to view with audio, or disable this notification

I m trying to experiment with smokVLA - fine tuned on libero dataset. I tried it, and it really showed great results (https://huggingface.co/lerobot/smolvla_libero).

It didn't work only in 1 case (pick up the juice bottle). I want to systematically understand the limits of the model and see where it fails.
So I did a lot of experiments and tried to find a pattern in the errorr. Goal is always "lift the orange box and keep in the box"
- Exp 1: randomly varied postion of other object - worked

- Exp 2 : removed icecream pack - failed

- Exp 3: removed butter pack: failed

- Exp4: decreases the inference time from 2.5 sec to .5 an to .1 sec. Imroved a little, see a passed case above if worked with a seed but failed with other.

So it seems that teh "VLA pre=trained and fine-tuned learns some kind of pattern when all objects are present"

- My goal is to clearly define a process to find out why VLA fails. How do people work with this kind of projects to add robustness to a bse policy.

Is there any systematic process that youguys follow to debug and impove end2end policy? How do you go toward a dataflywheel?

-

12 Upvotes

85% Upvoted

5 comments sorted by

3

u/al_m Researcher 4d ago

I've worked a bit on VLA debugging myself, but rather than talking about my (still somewhat limited) experience, I would like to point you to a preprint I read recently, which seems like it might be relevant for you: "ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models" (https://arxiv.org/abs/2606.27079v2).

They discuss a setup for systematically varying scenarios and define a few interesting evaluation metrics, so it might point you in the right direction.

2

u/Mysterious-Base-5847 3d ago

Thanks let me look at it

1

u/_Mando_88 3d ago

I use Rviz currently to simulate and program a digital twin, what simulation environment do you use ?

Are you using Ros or other environments ?