The AI agent was able to configure the hand, use it to see and gently grasp things, and was even able to train another AI model to pick up and place specific objects. And they say AGI is still a few years away! (I’m kidding, maybe it is).
The results convince me that we may be on the verge of a robotics breakthrough. Robots require considerable skill to train and control them. Today’s AI models can make this almost easier.
“AI-powered coding is extremely exciting because it has the potential to bridge the gap between traditional engineering methods, which are reliable but do not generalize, and contemporary vision-language-action models, which generalize but are not yet reliable,” says Ken Goldberg, a roboticist at UC Berkeley who is exploring this approach.
I bought a prebuilt arm called Lerobot 101. It’s part of an open-source project from HuggingFace that makes it relatively inexpensive to start building and experimenting with robotics.
LeRobot comes with two arms: a controller arm that a person operates using a handle and trigger, and a follower arm with a camera that replicates those movements. You can train an AI model by teleoperating the controller arm and teaching the model how to move the follower in response to what it sees on the camera.
building with openclaw
Before using OpenClaw, I spent many hours connecting and calibrating the robot, at one point almost breaking the motors by applying the wrong settings, causing them to overheat.
Then, with the help of OpenClaw and Codex, I was able to vibe code a simple program that would close the claw gripper when a red ball appeared. In the terminal, Codex went through the difficult task of configuring the connection to the robot. Then, with my help, it calibrated the position of its joints. It also wrote a Python script that used several libraries to detect and catch the ball. Vibe-coding is certainly not perfect, and hallucinations can introduce bugs especially when working with different hardware, but the results were impressive.
<a href