The Independent’s journalism is supported by our readers. When you purchase through links on our site, we may earn commission.

This amazing 'neural network' program describes what it sees on a trip through Amsterdam

The program can identify what it's looking at, live, through an ordinary laptop webcam

Wednesday 25 November 2015 20:31
The program managed to correctly say that it was looking at an image of a man eating a hot dog
The program managed to correctly say that it was looking at an image of a man eating a hot dog

A digital 'neural network' has been put through its paces on the streets of Amsterdam - with a video showing how the program can identify people, vehicles and even different types of food by analysing live webcam images.

The NeuralTalk code was originally designed as a way to automatically caption images - but after some tweaking by American coder and artist Kyle McDonald, it was made to caption live video instead.

NeuralTalk and Walk from Kyle McDonald on Vimeo.

The software isn't 100 per cent accurate, but it's pretty impressive - seeing a man walking with a bag, the software produces the caption: 'Man walking down the street with a suitcase'.

It also manages to correctly identify boats, cars and bicycles (useful for when you're using it in Amsterdam).

The software really comes into its own when McDonald approaches a group of guys at a hot dog stand.

Looking at the laptop's screen, they're surprised to see the program identifying them correctly - as 'a man holding a hot dog in a bun'.

Pointing the webcam at the hot dog stand's counter, the program correctly says it is a 'refrigerator filled with lots of food and drinks'.

It's a cool thing in itself, but the technology could have important applications - such as helping the blind by automatically describing images and videos. But for now, the hot dog stand is a major achievement.

Register for free to continue reading

Registration is a free and easy way to support our truly independent journalism

By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists

Already have an account? sign in

By clicking ‘Register’ you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice.

This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply.

Join our new commenting forum

Join thought-provoking conversations, follow other Independent readers and see their replies


Thank you for registering

Please refresh the page or navigate to another page on the site to be automatically logged inPlease refresh your browser to be logged in