The Last Word: Toyota Research Institute’s Gill Pratt on Using Generative AI to Teach Robots ‘New, Dexterous Skills’

The advancement, leveraging what the institute calls "Diffusion Policy," is said to improve robot utility and be a step toward building "Large Behavior Models" for robots, similar to Large Language Models like ChatGPT which have revolutionized conversational AI.

“Our research in robotics is aimed at amplifying people rather than replacing them.”

Gill Pratt
CEO and Chief Scientist
Toyota Research Institute
.…on Toyota Research Institute’s “breakthrough” generative AI approach to teaching robots new, dexterous skills.

The Toyota Research Institute—a forward-looking arm of Plano-based Toyota Motor North America—has announced a “breakthrough” generative AI approach that can “quickly and confidently teach robots new, dexterous skills.”

The advancement, leveraging what the institute calls “Diffusion Policy,” is said to improve robot utility and be a step toward building “Large Behavior Models” for robots, similar to Large Language Models like ChatGPT which have revolutionized conversational AI.

“This new teaching technique is both very efficient and produces very high-performing behaviors, enabling robots to much more effectively amplify people in many ways,” Pratt said in a statement.

The institute, which has offices in Los Altos, California, and Cambridge, Massachusetts, said that techniques to teach robots new behaviors have until now been “slow, inconsistent, inefficient, and often limited to narrowly defined tasks performed in highly constrained environments.” To program robot behaviors, many hours of sophisticated code-writing was required, along with numerous trial-and-error cycles.

TRI’s new breakthrough could speed and improve that process in multiple ways.

Robots taught 60 ‘difficult’ skills without a single line of new code

TRI says it has already taught robots “more than 60 difficult, dexterous skills using the new approach, including pouring liquids, using tools, and manipulating deformable objects”—all without writing a single line of new code.

The only change was supplying the robot with new data, TRI said. To build on that success, the institute has set a target of teaching “hundreds of new skills” by the end of the 2023 and 1,000 by the end of 2024.

Going beyond ‘pick-and-place’ skills

The newly trained robots aren’t limited to simple “‘pick and place” skills, like moving an object to a different location. TRI’s robots “can now interact with the world in varied and rich ways—which will one day allow robots to support people in everyday situations and unpredictable, ever-changing environments,” the institute said.

Russ Tedrake, VP of robotics research at TRI, says the tasks the newly trained robots perform “are simply amazing. Even one year ago, I would not have predicted that we were close to this level of diverse dexterity.”

Tedrake wears multiple hats: He’s also the Toyota Professor of Electrical Engineering and Computer Science, Aeronautics and Astronautics, and Mechanical Engineering at MIT.

“What’s so exciting about this new approach is the rate and reliability with which we can add new skills,” Tedrake said in a statement. “Because these skills work directly from camera images and tactile sensing, using only learned representations, they are able to perform well even on tasks that involve deformable objects, cloth, and liquids—all of which have traditionally been extremely difficult for robots.”

Using ‘haptic demos’ to train the robots

So how does the institute train these robots? TRI says its robot behavior model “learns from haptic demonstrations from a teacher, combined with a language description of the goal. It then uses an AI-based Diffusion Policy to learn the demonstrated skill. This process allows a new behavior to be deployed autonomously from dozens of demonstrations. Not only does this approach produce consistent, repeatable, and performant results, but it does so with tremendous speed.”

The training utilizes Diffusion Policy, which was developed by TRI and its collaborators in Professor Shuran Song’s group at Columbia University. The new, generative-AI approach to behavior learning. is said to enable “easy and rapid behavior teaching” based on demonstration.

The training also leverages a model-based design for robotics called Drake that provides TRI with “a cutting-edge toolbox and simulation platform,” offering a “high degree of realism” that allows roboticists to develop in both simulation and in reality at “a dramatically increased scale and velocity.” TRI’s internal robot stack is built using Drake’s optimization and systems frameworks, the institute said—with Drake being made open source to catalyze work across the entire robotics community.

So how does Toyota’s North American headquarters in Plano get involved in all this? The HQ acts as a conduit for bringing TRI’s innovations to market with its direct connections to Toyota’s manufacturing and sales operations, the institute noted.

You can see the robot training in action on this TRI YouTube video.

If you want to learn more and hear directly from the TRI robotics research team, the institute is hosting a LinkedIn Live Q&A session on October 4th from Noon to 12:30 pm Central Time.  You can sign up for the event on TRI’s LinkedIn page.

For more of who said what about all things North Texas, check out Every Last Word.

Get on the list.
Dallas Innovates, every day.

Sign up to keep your eye on what’s new and next in Dallas-Fort Worth, every day.

One quick signup, and you’re done.

R E A D   N E X T

  • The 29 new grants approved by CPRIT's Oversight Committee include the first CPRIT Scholar award to Southern Methodist University in Dallas and state funding for Dallas' Baylor Research Institute to join a major National Cancer Institute study.

  • AI Master's Degree UNT

    Innovators across North Texas and entrepreneurs all over the U.S. are racing to launch (and patent) the latest breakthrough AI. That can lead to IP policy issues as emerging tech hits the market. On February 8, the U.S. Patent and Trademark Office will a public meeting to discuss how to promote innovation in AI and Machine Learning tech, to be held virtually and in person at the Arts District Mansion in Dallas. "The takeaways will shape future work on AI and ET policy," says USPTO Director Kathi Vidal.

  • Over 150 scientists across dozens of departments will be part of the elite National Institutes of Health-funded, university-wide interdisciplinary research center. Dallas' UT Southwestern Medical Center is the only institution in Texas to be selected for the NIH initiative. UTSW wants to translate scientific discoveries into new therapeutic strategies for the prevention and treatment of obesity, which it describes as a chronic disease affecting more than 40% of the U.S. population, with medical costs nearing $175 billion.

  • Plano-based Toyota North America is expanding its partnership with Google Cloud to enhance AI-based speech services in Toyota and Lexus vehicles. Audio control is now just a voice command away, processed by in-vehicle voice AI. And a future Speech On-Device offering could offer a range of natural speech functions without even using the cloud—eliminating "dead zones" and tunnel dropouts.

  • Driving Positive Change: How Toyota's Manufacturing Culture Goes Beyond Cars

    In celebration of its 30th anniversary, Plano-based Toyota Production System Support Center is joining with its partners to mark the not-for-profit corporation improving the lives of small and mid-sized manufacturers, food banks, healthcare providers and government agencies. Toyota Production System Support Center (TSSC) has shared its know-how with more than 500 partners, using teachings based on the Toyota Production System (TPS), an organizational manufacturing culture and mindset that engages people to continuously make improvements and create solutions. "Sharing ideas to find better ways of doing things demonstrates that small improvements can make a big difference while enriching peoples' lives and…