Robots Trained by Video: A Leap Toward Autonomous Surgery

Summary: For the first time, a robot has been trained to perform surgical procedures by watching videos of expert surgeons, marking a leap forward in robotic surgery. This breakthrough in “imitation learning” means that robots can learn complex tasks without needing to be programmed for every individual movement.

By training on surgical footage, the robot replicated procedures with skill comparable to human surgeons, demonstrating its ability to adapt and even correct its actions autonomously. Researchers believe this approach could enable faster and more accurate surgical training for robots, opening doors to fully autonomous surgeries in the future.

The technology uses the same foundational AI principles as language models but adapts them to control robotic motion. The study could transform the field of surgery, reducing medical errors and enhancing precision.

Key Facts:

Robots trained via video imitation performed surgeries as effectively as human surgeons.
This AI method accelerates training for robotic surgeries, enabling quicker, more adaptable learning.
The model uses machine learning similar to ChatGPT, but in the “language” of robotic kinematics.

Source: Johns Hopkins Medicine

A robot, trained for the first time by watching videos of seasoned surgeons, executed the same surgical procedures as skillfully as the human doctors.

The successful use of imitation learning to train surgical robots eliminates the need to program robots with each individual move required during a medical procedure and brings the field of robotic surgery closer to true autonomy, where robots could perform complex surgeries without human help.

This shows a robotic arm. — The researchers fed their model hundreds of videos recorded from wrist cameras placed on the arms of da Vinci robots during surgical procedures. Credit: Neuroscience News

“It’s really magical to have this model and all we do is feed it camera input and it can predict the robotic movements needed for surgery,” said senior author Axel Krieger. “We believe this marks a significant step forward toward a new frontier in medical robotics.”

The findings led by Johns Hopkins University researchers are being spotlighted this week at the Conference on Robot Learning in Munich, a top event for robotics and machine learning.

The team, which included Stanford University researchers, used imitation learning to train the da Vinci Surgical System robot to perform fundamental surgical procedures: manipulating a needle; lifting body tissue, and suturing.

The model combined imitation learning with the same machine learning architecture that underpins ChatGPT. However, where ChatGPT works with words and text, this model speaks “robot” with kinematics, a language that breaks down the angles of robotic motion into math.

The researchers fed their model hundreds of videos recorded from wrist cameras placed on the arms of da Vinci robots during surgical procedures. These videos, recorded by surgeons all over the world, are used for post-operative analysis and then archived.

Nearly 7,000 da Vinci robots are used worldwide, and more than 50,000 surgeons are trained on the system, creating a large archive of data for robots to “imitate.”

While the da Vinci system is widely used, researchers say it’s notoriously imprecise. But the team found a way to make the flawed input work. The key was training the model to perform relative movements rather than absolute actions, which are inaccurate.

“All we need is image input and then this AI system finds the right action,” said lead author Ji Woong “Brian” Kim.

“We find that even with a few hundred demos the model is able to learn the procedure and generalize new environments it hasn’t encountered.”

The team trained the robot to perform three tasks: manipulate a needle, lift body tissue, and suture. In each case, the robot trained on the team’s model performed the same surgical procedures as skillfully as human doctors.

“Here the model is so good learning things we haven’t taught it,” Krieger said. “Like if it drops the needle, it will automatically pick it up and continue. This isn’t something I taught it do.”

The model could be used to quickly train a robot to perform any type of surgical procedure, the researchers said. The team is now using imitation learning to train a robot to perform not just small surgical tasks but a full surgery.

Before this advancement, programming a robot to perform even a simple aspect of a surgery required hand-coding every step. Someone might spend a decade trying to model suturing, Krieger said. And that’s suturing for just one type of surgery.

“It’s very limiting,” Krieger said. “What is new here is we only have to collect imitation learning of different procedures, and we can train a robot to learn it in a couple days. It allows us to accelerate to the goal of autonomy while reducing medical errors and achieving more accurate surgery.”

Authors from Johns Hopkins include PhD student Samuel Schmidgall; Associate Research Engineer Anton Deguet; and Associate Professor of Mechanical Engineering Marin Kobilarov. Stanford University authors are PhD student Tony Z. Zhao.