The Story Behind the Robotic Hand That’s Ace at Where’s Waldo

You may have recently seen a viral video showcasing a robotic arm (complete with comically tiny synthetic hand) that can correctly identify and point out Waldo in just a few seconds. Knoxville-based agency redpepper are the cheeky people behind the project.

So, is this AI powered robot a fun use of new technology, or the worst thing to happen to Waldo since someone circled him in all the copies at the school library? The internet has been unable to come to a consensus, with some celebrating the fun use of tech, some worrying about its implications for surveillance, and some asking not Where’s Waldo but Why’s Waldo? What was the point for all this? 

There’s Waldo creator Matt Reed tells us: “I’ve always enjoyed the Waldo books and now that I have kids of my own I’m seeing how timeless of a challenge it is. I think that combined with this clip from Short Circuit, combined with all the clever things people are doing with computer vision lately all just snapped together about two weeks ago. I immediately ordered the biggest Waldo book from Amazon and about a week later, There’s Waldo.”

The robot is powered by a Raspberry Pi computer, a webcam that scans the entire page and Google’s cloud based AutoML Vision service. Once the camera takes the image and finds each face, the Raspberry Pi transfers them to AutoML. Here, the neural network that has been trained with other pictures of Waldo scraped from Google Image search seeks to recognise one of the faces as Waldo, and when it finds a match, extends the robot arm to the relevant coordinates. 

Services like AutoML are making AI based tech less of a sci-fi fever dream and more of a daily reality. On his experience, Matt says: “I was surprised at how easy creating visual model was. Just drag and drop a bunch of pics of Waldo and click a button to start training the AI, anybody could do it. Up until AutoML this has been relegated to academics with powerful computer resources and was tedious to hand code… knowing how powerful deep neural networks have become this was a perfect way for me to learn more about how to make a computer identify anything you want it to in images or video.” 

There’s Waldo is the latest in a series of fun technological novelties out of redpepper, following their VR game Moonpie VR and Slack-connected Roomba turned office assistant Slackbot Bot. “We do sprint-based innovation work and are constantly exploring new technologies so we can leverage them throughout our creative process,” Matt said. “To get a solid understanding of a technology's opportunities, I've found the best way is to make something with it. We like to try and put a clever or relatable spin on whatever it is we build so we can make more friends on the internet.”

The attention There’s Waldo has garnered over recent days hasn’t gone unnoticed by Matt, either. The YouTube video has over 200,000 views, while a tweet sharing the clip has around 4.1 million. “The reactions are polarised but the negative ones are generally pretty hilarious,” he said, citing a few of his favourite comments: "Greatest invention ever”, "Do you know how many five-year-olds are going to be out of a job because of you?!?!??!?!”. “Somebody tweeted me, ‘Now have it find Carmen Sandiego.’Challenge Accepted?”

But whatever your thoughts are on artificial intelligence, projects like There’s Waldo are a much needed dose of absurdity. How could we fear this little guy, with its clumsy open hand and love of such a childhood classic?   

But, just to be safe, if there is a robot uprising, make sure you’re not wearing glasses and a red and white striped top.