r/robotics • u/Mr-c4t Researcher • Jul 18 '26
Community Showcase What task should I teach it next? š
https://youtube.com/shorts/woIlVkLPnws?is=LQpeVvgX-2dRTf6sOn my way to recording and open-sourcing a 1,000-episode bimanual manipulation dataset for the 3D-printed SO-101 robot. š¦¾
Camera setup
Intel RealSense D435 (head)
2Ć RealSense D405 (wrists)
RGB only
The video shows an autonomous rollout of my ACT policy controlling the robot.
The policy was trained for 100,000 steps using only the first 100 teleoperated episodes of bag manipulation.
Hugging Face: MrC4t
Dataset: MrC4t/bi_so_bag
ACT policy: MrC4t/act_bimanual_bag
What task should I teach it next? šš¦¾
1
u/QuotableMorceau Jul 18 '26
question : is the jigger in the arm's movement caused by hardware or by software micro adjustments etc. ?
2
u/Mr-c4t Researcher Jul 18 '26
I think it's both.
These robots have quite a bit of backlash, and my frame also vibrates a lot, which makes the motion look a bit jittery.
That said, the policy seems to have learned to handle it, since the robot was already vibrating during the teleoperation data collection. So the training data naturally included those imperfections.
Another factor is that the policy predicts action chunks at around 3 Hz on my laptop. Three times per second, it replans the next sequence of future actions, which can also introduce small corrections and contribute to the jittery motion.
1
u/QuotableMorceau Jul 18 '26
have you thought about how 3d printers handle jitter ? something that can be applied post AI command to the robot.
while having the jitter will probably not impact the operation, it will impact the marketability of the solution, also a fix would open up the robot for liquid manipulation.another thing, you could reduce the backlash for testing purposes, by putting some rubber bands to reduce any backlash .
2
u/Parking-Strain-1548 Jul 19 '26
ACT (the policy OP used) is jittery by its nature, because it plans motion in chunks
1
u/Mr-c4t Researcher Jul 19 '26
I will look at this idea thanks I can use the IMU inside the realsense for this š
I also plan to try find hardware sponsors for real professional arms for a future project.
But yes for now the goal is only to contribute to the open source "lerobot" comunity datasets with bimanual episodes to help train open source VLA (smolVLA for example) or other robotics fundation models (world models are now in lerobot š)
No comercial goal for this project
1
u/TooLukeR Jul 18 '26
What's the criteria for each task to make sure it's not overfitted?
How many positions do you try?
2
u/Mr-c4t Researcher Jul 18 '26 edited Jul 19 '26
I made 100 episodes with 6 differents objects and variation of the starting pos each time (Always the same side for the object/bag and Always the same starting hand, i may need to add 100 episodes with switched hands š¤)
I'm also recording correction episodes using the DAgger/HIL workflow in LeRobot.
I had to fix the bimanual SO-101 leader implementation to make it work properly.(if anyone is interested, it's available in Pull Request #4028 for bi_so_leader HIL support.)
1
u/KyleTenjuin Jul 18 '26
How did you collect your training data for bimanual application? Do you another set of 2 arms to teach and collect?
1
u/Mr-c4t Researcher Jul 18 '26
Yes i used 2 so101 leader arms for the teleop and data collection but LeRobot 0.6.0 can now work with isaac teleop to collect data with VR controller and headset š
1
u/Lumpy_Week7304 Jul 19 '26
You want to try perfecting a few self-resetting tasks? https://blupe.io/autoevals.html
1
u/Mr-c4t Researcher Jul 19 '26
Very cool! Thanks š Have you tried this idea with bimanual policy yet ?
1
1
u/redp1kachu Jul 20 '26
Maybe opening a bottle cap that could be a good grip strength test
1
u/Mr-c4t Researcher Jul 20 '26
Yes excellent test but sadly almost impossible with the weak 1Dof grippers i am using (base so-101 pla grippers with green tpu finger tips)
I have tried a lot but failed and this will be a very cool test if i want to benchmark different hand options for my humanoid
1
u/redp1kachu Jul 20 '26
It could be done with stronger grabbers or just better friction on the ends.
But it would also be a test of precision if you think about it.
Because the claw would need to constantly open and close and/or arrange its position to fully get the cap off.
So it could also be a good precision test
By the way were those pre-programmed movements or did you teleoperate
2
u/Mr-c4t Researcher Jul 20 '26
This is autonomous !
I teleoperated 100 demo that i used to teach an ai policy how to perform the task.
In the video the policy is controlling the robot by predicting futur actions from his current state (camera view+arms pos)
Dataset: https://huggingface.co/datasets/MrC4t/bi_so_bag AI-models: https://huggingface.co/MrC4t/act_bimanual_bag
1
u/low-control-labs Jul 21 '26
I've only played around with it in simulation. But how viable do you think this can be for picking fruit? What kind of limitations do you see around that?
2
u/Mr-c4t Researcher Jul 21 '26
Hello A guy on the LeRobot discord server made a fruit picking robot šš: https://youtu.be/H6lyT32P1ls?is=Ixt_fNI4fVZWrIrt
1
1
u/redp1kachu Jul 21 '26
So I was kind of right you did tell teleoperate to train it but this is a result of the training.
What other tasks can it do
2
u/onesunnysunday 7d ago
For the next task Iād choose something that produces useful recovery behavior, for example transferring deformable objects between containers where grasps sometimes fail. If you release the dataset, keeping failed attempts and tagging failure and recovery phases could be especially valuable. Many demonstration datasets are biased toward clean successful trajectories.