Microsoft’s newest bot, CaptionBot, aims to amend Tay’s failure

15 Apr 2016

Following the disastrous attempts to learn from millennials using the Twitter chatbot Tay, Microsoft has released another AI program, called CaptionBot, which automatically creates captions for images.

Microsoft’s latest AI, CaptionBot, is following in the footsteps of Facebook, which has begun looking into ways AI algorithms can better identify images, both to help the blind see images on their News Feed, and to help advertisers make more money.

The program, which has just been released for public testing, is a follow on from some of the other image-reading software the tech giant has released over the last year to much publicity, such as last May’s ‘How Old Do I Look?’ which would predict your age by examining a photo of your face.

According to Microsoft, CaptionBot works by using APIs based on computer vision and natural language, and an ability to detect emotion, to describe content of images.

Given that it is a new program based on machine learning, which picks up on things over time, the current image identifier is, it’s safe to say, not very accurate at all bar a few easy examples.

Trying to forget the Tay episode

In its own testing, Microsoft shows how CaptionBot is able to identify someone on a skateboard, but appears to be struggling to make sense of people’s faces, with the bot declaring that nearly everyone is making an uninterested face.

Regardless, it’s safe to say that Microsoft has gotten a little bit more control of how this testing will go, compared with the two disastrous attempts to launch Tay. Its millennial chatbot on Twitter was forced to be taken down thanks to the efforts of the internet to make it say horrible, horrible things.

To help CaptionBot learn, we added a few images we had lying around to see if it could indeed make sense of what it was seeing.

The good and the bad

Giving it a bizarre first test of Robbie Williams pretending to be Superman, it seems CaptionBot couldn’t be wider of the mark.

Robbie CaptionBot

The same goes for legendary astronaut Buzz Aldrin during the Apollo 11 mission, who definitely was not swinging a baseball bat on the moon.

Apollo 11

But we begin to see some improvement with some slightly easier guesses. In this case, it just gets confused between four-legged animals.

Deer field

Finally, it seems to be able to spot yours truly with relative ease.

Gorey smile

All in all, a decent showing from CaptionBot, which is now available for anyone to upload images to and help Microsoft get its AI fine-tuned.

Photo collage image via Shutterstock

Colm Gorey was a senior journalist with Silicon Republic

editorial@siliconrepublic.com