The first time we let the arm run on its own, it worked. It reached down, grabbed the sock, lifted it, and dropped it into the cardboard box, with no joystick, no me, just a model I'd trained overnight on my laptop. My wife and I genuinely cheered.
Then we ran it again.
This post is the whole story: the part I told everyone, and the part that took me two training runs to understand. Real commands, real numbers, and the most useful mistake I've made in robotics so far.
Learning to drive an arm
Before a robot can learn from you, you have to be worth learning from. In imitation learning you teach by demonstration, and demonstrating means teleoperating: you move the leader arm by hand, the follower mirrors you, and the system records everything.
Nobody tells you that teleoperation is a skill. The first sessions were genuinely clumsy: the movements didn't feel natural, and completing the task at all was hard. I failed plenty: the sock fell outside the box; the sock bounced off the rim and landed outside; and more than once the drop hit the box's edge and flipped the whole container 90 degrees, because a cardboard box weighs nothing. (File that detail away; it comes back later.)
Then, slowly, the arm became something like part of my body. Precision arrived, the failures got rare, and I could do the task smoothly. That's the version of me the robot got to learn from.
Recording 54 episodes
A demonstration in robotics is called an episode: one complete run of the task, captured as data: both camera streams (the gripper's view and an external view) plus the positions of all six motors, frame by frame. I recorded 54 of them in one sitting of about two hours, with tiny breaks to rest, deliberately moving the sock and the box around between episodes (sock left and box right, then reversed, then everything in the middle) so the model wouldn't just memorize one layout.
Recording runs through LeRobot's CLI. Simplified to the parts that matter (the ports and camera indexes are from my setup; yours will differ):
lerobot-record \
--robot.type=so101_follower \
--robot.port=/dev/tty.usbmodem<follower> \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 1920, height: 1080, fps: 30}}" \
--teleop.type=so101_leader \
--teleop.port=/dev/tty.usbmodem<leader> \
--dataset.repo_id=${HF_USER}/so101-sock-in-box \
--dataset.num_episodes=54 \
--dataset.single_task="Grab the sock and put it in the box"Episodes record at 30 fps by default. The dataset uploads to the Hugging Face Hub as you go, which turns out to be the quiet superpower of this ecosystem: your demonstrations become a portable dataset anyone's training script can consume.
Training on a laptop
I trained an ACT model: Action Chunking Transformer, a policy well suited to robot control because it predicts short chunks of movement at a time instead of twitching one step at a time.
Some vocabulary, briefly, because it all appears in the command below. A step is one round of the model studying a batch of slices from your episodes, guessing the next movements, checking how wrong it was, and adjusting. A batch is how many of those slices it studies per step: 8 here, LeRobot's default, inherited from the original ACT paper. A checkpoint is a saved snapshot of the model mid-training; I saved one every 5,000 steps, so a crash at hour 20 wouldn't mean starting over, and so I could later test whether the final model was actually the best one.
lerobot-train \
--dataset.repo_id=${HF_USER}/so101-sock-in-box \
--policy.type=act \
--policy.device=mps \
--steps=100000 \
--batch_size=8 \
--save_freq=5000 \
--wandb.enable=true \
--output_dir=outputs/train/act_sock_in_box(--policy.device=mps is the Apple-Silicon flag; 100,000 steps and batch 8
are LeRobot's defaults, spelled out here so you know what ran.)
On my M3 MacBook Pro, 100,000 steps took around 23–24 hours. I let it run overnight and through the next day, watching from my phone via Weights & Biases, a free service that charts your training as it runs. That's worth setting up on day one: it shows you the loss (how wrong the model currently is) trending down, and when you train more than once, it lets you compare runs against each other instead of against your memory.
The first run, and the second
You know how the first run went; it's the top of this post. One clean grab, one clean drop, two adults cheering at a robot arm in their living room.
Reality checked in on the runs after that. The arm struggled: missed grasps, half-grabs where the sock slipped away, approaches that stalled. The first success turned out to be closer to a good roll of the dice than a solved task.
We started listing suspects, and I want to be honest about their status: these were hypotheses, and most of them still are.
- Not enough data. 54 episodes is a starter set by any standard.
- Position coverage. We suspected the failures clustered where the sock or box sat in spots our episodes had barely covered.
- Lighting drift. We recorded in the living room across different times of day, no lighting gear; the model may have been fighting shifting shadows.
- Frame rate. Episodes were recorded at 30 fps; whether my MacBook could actually sustain that rate at inference time is a genuinely good question I hadn't measured.
Untangling suspects like these is exactly why I later built RoboLab; at the time, I was eyeballing runs and trusting my memory, which is how you fool yourself.
25 more episodes
We tested the one hypothesis we could act on cheaply: coverage. I recorded about 25 more episodes, aimed specifically at the under-represented sock and box positions, and retrained from scratch on the full 79-episode dataset: same recipe, 100,000 steps, another day of laptop fans.
It was better. Honestly better: call it roughly 3 successes in 10 attempts on the runs we did. And still: it struggled. More data had helped, but it clearly wasn't the whole story, and that's when the actual lesson finally landed.
The real lesson: I chose a bad first task
It's not that ACT sucked. It's that I did a bad job defining a good first task for it.
Look at what I actually asked for. The sock is malleable, squishy, and featherweight: it deforms in the gripper, slides through half-closed jaws, and moves if the gripper so much as brushes it in passing. The cardboard box weighs nothing and shifts or flips when a drop lands on its edge. Remember the container doing a 90-degree flip back in my teleop practice? The physical properties of my objects were quietly part of the machine learning problem the whole time, and I'd picked two of the least cooperative objects in the apartment.
A rigid object (a wooden dice, say) and a heavier, solid container would have made a world of difference: consistent grasps, a target that stays where you put it, failures that mean something. (You may notice the arm on this site's homepage practices with a wooden dice and a rigid tote. Lesson learned.)
What I learned
- Task definition beats model choice. Before asking "which policy?", ask "is this task well-formed?" Rigid objects, stable targets, repeatable starts.
- The first success is not a solved task. One clean run tells you the pipeline works, not that the policy does. Run it twenty times before you celebrate, or at least before you post the video.
- A good session is not a metric. I wasn't logging trials properly, so every conclusion was vibes. Log attempts and outcomes from day one.
- Your objects are part of your ML problem. Squishiness, weight, and friction are features the model has to cope with. Choose them like you'd choose hyperparameters.
- Checkpoints and run tracking are cheap insurance. A 24-hour run on a laptop will test your faith; snapshots every 5k steps and a live loss curve keep it rational.
What's next
The 79 episodes aren't done working. I've already fine-tuned π0.5, a newer, much bigger open policy, on that same dataset, using a rented cloud GPU because my laptop wasn't enough for that one. Whether a smarter model can rescue my badly-designed task with the same data is exactly the kind of question the next post gets to answer. And in parallel: a second, deliberately simple task (rigid object, stable target) to give ACT the fair test I didn't give it the first time.
If you had this arm on your desk, what would you teach it first? The comments are open, and if you've made the same bad-first-task mistake, I'd genuinely love to hear it.