A robot learned to lip sync after watching hours of YouTube videos

The robot learned the ability to use its 26 facial motors by practicing to imitate human lip motions in front of the mirror

Engineers at Columbia University announced on Wednesday that they have managed to invent a robot that is able to learn and recreate facial lip motions for speech and singing for the first time (Jane Nisselson/ Columbia Engineering)

Robots have now mastered one of the most complex of human mannerisms - lip motions, in a technological breakthrough.

With almost half of our attention during face to face interactions focusing on lip gestures, it is no wonder we are so attuned to spot even the slightest change in facial expression.

Up until now, robots have struggled to mimic the way humans move their lips, often appearing to be slightly off-kilter. This can create an “uncanny valley” effect - an eerie feeling of encountering something that looks almost human but doesn't sit right

But that could be about to change as a future of robots indistinguishable from humans edges closer.

Engineers at Columbia University announced on Wednesday that they have managed to invent a robot that is able to learn and recreate facial lip motions for speech and singing for the first time.

The robot learned the ability to use its 26 facial motors by watching hours of YouTube videos and practicing to imitate human lip motions by observing its own reflection in the mirror.

The robot learned the ability to use its 26 facial motors by watching hours of YouTube videos and practicing to imitate human lip motions by observing its own reflection in the mirror (Jane Nisselson/ Columbia Engineering)

Researchers showcased how their robot is able to articulate words in numerous languages and even sing a song out of its AI-generated debut album “hello world_, in the Science Robotics journal.

“The more it interacts with humans, the better it will get,” promised the engineers Hod Lipson, James and Sally Scapa at Columbia’s Creative Machines Lab, where the work was done.

The researchers acknowledge that the lip motion is far from perfect. “We had particular difficulties with hard sounds like ‘B’ and with sounds involving lip puckering, such as ‘W’. But these abilities will likely improve with time and practice,” Mr Lipson said.

“Much of humanoid robotics today is focused on leg and hand motion, for activities like walking and grasping,” said Lipson. “But facial affection is equally important for any robotic application involving human interaction.”

“When the lip sync ability is combined with conversational AI such as ChatGPT or Gemini, the effect adds a whole new depth to the connection the robot forms with the human,” explained Yuhang Hu, who led the study for his PhD.

Researchers showcased how their robot is able to articulate words in numerous languages and even sing a song out of its AI-generated debut album “ hello world_ , in the Science Robotics journal (Hello World)

Lipson and Hu predict warm, lifelike faces on robots to find applications in entertainment, education, medicine, and even elder care. Some economists predict that over a billion humanoids will be manufactured in the next decade.

“There is no future where all these humanoid robots don’t have a face. And when they finally have a face, they will need to move their eyes and lips properly, or they will forever remain uncanny,” Lipson estimates.

This work is part of Lipson’s decade-long quest to make robots connect more effectively with humans. He insists that these abilities must be acquired by learning, rather than being programmed using stiff rules.

“Something magical happens when a robot learns to smile or speak just by watching and listening to humans,” he said. “I’m a jaded roboticist, but I can’t help but smile back at a robot that spontaneously smiles at me.”