I was a little disappointed with China’s World Humanoid Robot Games.1 as fun as real life Rock ‘Em Sock ‘Em Robots yes, what people In fact care about robots doing their workThat’s why robot laundry folding videos are so popular: Until a few years ago we didn’t know how to do it, And this is definitely something that people want! but as this article Very well expressed, basic laundry folding is in a good position given the technologies we have now. It might feel like if our AI technologies can fold laundry, maybe they can do anything – but that’s not true, and we’ll actually have to invent new technologies to be general purpose and useful.
With this in mind I am issuing a challenge to roboticists: Here are My Humanoid Olympic event. Each event will require us to advance cutting-edge technology and unlock new capabilities in robotic manipulation. As people achieve these milestones, I will update my Substack and send real-life medals to the winners.
current state of the art
To talk about why each of these challenges advances the state-of-the-art in robotic manipulation, let’s first talk about what’s working now. What I see working is learning from performance. Generally, people are using the Puppet interface. The most common seem to be two copies of the robot so that a human can hold and move one of them while the other follows, or a virtual reality headset with controllers for hand tracking. They then record 10-30 seconds of activity hundreds of times. From that data, a neural network is trained to mimic those examples. This technology has unlocked tasks that have steps that are somewhat disorganized (such as pulling a corner of a towel and laying it flat) or have high status positions (such as how a towel can be assembled in a myriad of different ways).
When thinking about this method of training robots to do things, it should be clear what some of its limitations are. There are exceptions to each of these, but together they form a general trend.
- No force feedback on wrists.2 The robot can only perform as well as the human teleoperating it can, but we do not yet have good standardized ways of getting high resolution force information to the human teleoperator.
- Limited finger controls.3 It is harder for the teleoperator (and for the foundation model) to see and control the robot much more efficiently than opening and closing all of its fingers.
- No sense of touch.4 Human hands are absolutely packed with sensors. Getting the kind of sensation from robotic hands that can be used by a human teleoperator is not currently possible.
- Medium precision.5 Based on the videos I’ve seen, I think we’ve got about 1-3cm accuracy for the tasks.
Now, onto the events!
Event 1: Doors
Event 2: Laundry
Event 3: Equipment
Event 4: Fingertip Manipulation
Event 5: Wet Manipulation
Event 1: Doors
Things like doors are tricky because of asymmetric forces: you have to hold and turn the handle or knob quite hard, but if you pull hard outside the arch of the door you lose your grip. Also, going through a door requires full body manipulation, which is more than anything I’ve ever seen.
Bronze Medal: Entering a round-knob push door
benji holson
I think it’s pretty close to the state of the art (or maybe it is and I haven’t seen it). I hope to get this medal by December.
Silver Medal: Lever-Handle Self-Closing Push Door Entry
benji holson
Adding self-closing makes this significantly more challenging due to the force involved, although a lever handle is arguably easier (I don’t see many round-knob self-closing doors).6
Gold Medal: Entering Lever-Handle Self-Closing Pull Doors
benji holson
Door boss fight.7 You either have to use another limb to stop the door from closing again, or use mobility to get through the door faster.
Event 2: Laundry
We are just starting to do laundry.
Bronze Medal: Turn a T-shirt inside-out
benji holson
This is probably possible using the techniques we have, but it is a long-term task and may require some tricky two-handed actions to pull the shirt right side out.8
Silver Medal: Turn a sock inside-out
benji holson
I think both the hand insertion and the pinching inside the sock are interesting new challenges.
Gold Medal: Hang Men’s Dress-Shirt
benji holson
Medium sized shirt with one sleeve opening from the inside out. It should have the sleeves fixed and end correctly on the hanger with at least one button. I’d guess it’s 3-10 years old, because the buttons are really hard and it would be hard to get a strong, dexterous hand small enough to fit in the sleeve.
Event 3: Equipment
Humans are creatures of technology and, as useful as our hands are, we mostly use them to hold and manipulate tools. This challenge is about building strength and dexterity in using basic tools.
Bronze Medal: Window Cleaner and Paper Towel
benji holson
The window cleaning fluid bottle is very forgiving in terms of how you hold it, but you need to freely articulate a finger, and the finger needs to be strong enough to spray the fluid.9
Silver Medal: Peanut Butter Sandwich
benji holson
The challenge here is to pick up a knife and then adjust the grip so that it is strong and stable enough to scoop out and spread the peanut butter. Humans use ‘strong tool grip’ for all kinds of activities, but this is very challenging for a robot gripper.10
Gold Medal: Use a Key
benji holson
A keyring containing at least 2 keys and a key chain are inserted into the robot’s waiting palm/gripperWithout putting the keys down, Align and insert the correct key and turn it into the lock. This requires high precision forceful interactions as well as very challenging hand manipulation.
Event 4: Fingertip Manipulation
We humans perform all kinds of manual manipulations by using the structure of our hands to manipulate the things we are holding.
Bronze Medal: Roll Matching Socks
benji holson
Dexterity and precision are required, but not too much force.
Silver Medal: Use Dog Poop Bags
benji holson
When I use a dog-bag I have to perform a sliding action between the fingers to detach the opening of the bag, which is a difficult forceful interaction as well as a motion that I’m not sure most robot hands are capable of doing. If you choose to use one, it’s also harder to tear a bag than having to pull a big long spool out of the holder.11
Gold Medal: Peel an Orange
benji holson
Done without external equipment. It is extremely intricate: high force yet high precision finger movements.
Event 5: Wet Manipulation
If you sit down and write down what you want the robot to do for you, a lot of work gets wasted. Robots generally don’t like to be wet, but if we want to keep them clean for us we have to change that. And wet items can be difficult to hold and use.
Bronze Medal: Wipe the counter-top with a sponge
benji holson
Lightly wet, but there is an exciting risk of getting your entire hand under water if you are not careful. Probably requires at least a pair of splatter resistant hands (or a whole bunch of spares).
Silver Medal: Clear peanut butter from your manipulator
benji holson
It naturally comes after the sandwich one. Water everywhere. It seems that collecting training data on the dog-poop task after a few hours is an important skill.
Gold Medal: Use a sponge to remove grease from pan in the sink
benji holson
Water, soap, grease and an unpleasant task no one wants to do.
terms and conditions
To be eligible to win, a general purpose manipulator robot running autonomously must complete a successful task in real-time video without any cuts. You are given a maximum of 10 times the time it took me to perform each task (a 4 second task could take your robot up to 40 seconds). I reserve the right to be arbitrary in making decisions if things are not going in line with the spirit of the challenge. The first robot to achieve this wins the prize!
To claim your medal please email me at bmholson+olympics@gmail.com with an address to send to. I would be tickled pink if you would give me a picture of your robot wearing a medal. I will also accept future challengers who are at least 25% faster than the current winner. Some medals have already been claimed; You can watch the winning video here. May good luck and circumstances always be in your favor.
Thanks to Jeff Bingham for the advice, fact checking, and great robot video. And thanks to my patient wife for spending an hour filming me doing silly things in a silly outfit.
notes
1 As far as I can tell, Kickboxing was just unitary mini-humanoid robots, and everyone had the same code running, so… I guess it won?
2 TRI has some pretty cool stuff with force control using a large training rig.
3 Tesla’s Optimus has 22 degrees of freedom using the cable drive (because you can’t fit those motors in one hand). I worked on this in 2008 this robot which also had 22 degrees of freedom and was extremely difficult to control (as were all the cables that had to be tensioned correctly). The other hand held a large two-finger gripper that I used for most teleport tasks.
4 Meta is working with some in-finger vision systems which sounds cool.
5 This is probably more of a model limitation than a teleoperation precision limit. Here’s a video of Generalist Robotics doing sub-semi precision work. I love that the hockey stick has become the traditional “messing with robots” tool for ridiculous things like this.
6 Yes, I wore it to work to get this video. you are welcome.
7 I have programmed (not trained) a general purpose mobile manipulator to pass through a self-closing pull door, but it took over 4 minutes (too time consuming to be inefficient) and required a special doorstop. Also the video is not public (also ineligible). Moreover it is really difficult to join a competition and get yourself a gold award even before it starts.
8 The t-shirt starts completely inside-out in a wedge. The finish is tolerably bent, right side out.
9 You should spray 3 good sprays on the window and wipe them with a paper towel so that no ugly streaks remain. Paper towels start on paper-towel rolls, not pre-torn and pre-wrapped.
10 Peanut butter jar starts and stops. The sandwich should be cut in half. (Triangle or rectangular cuts are both acceptable, although your three-year-old may disagree).
11 Fake poo is allowed. The bag starts on a roll but can be in a standard dog-bag holder held by the robot.
This post originally appeared on Substack from Benji Holson, General Robots, about building a general purpose robot company.
From articles on your site
Related articles on the web