“Our research in robotics is aimed at amplifying people rather than replacing them.”

Gill Pratt
CEO and Chief Scientist
Toyota Research Institute
.…on Toyota Research Institute’s “breakthrough” generative AI approach to teaching robots new, dexterous skills.

The Toyota Research Institute—a forward-looking arm of Plano-based Toyota Motor North America—has announced a “breakthrough” generative AI approach that can “quickly and confidently teach robots new, dexterous skills.”

The advancement, leveraging what the institute calls “Diffusion Policy,” is said to improve robot utility and be a step toward building “Large Behavior Models” for robots, similar to Large Language Models like ChatGPT which have revolutionized conversational AI.

“This new teaching technique is both very efficient and produces very high-performing behaviors, enabling robots to much more effectively amplify people in many ways,” Pratt said in a statement.

The institute, which has offices in Los Altos, California, and Cambridge, Massachusetts, said that techniques to teach robots new behaviors have until now been “slow, inconsistent, inefficient, and often limited to narrowly defined tasks performed in highly constrained environments.” To program robot behaviors, many hours of sophisticated code-writing was required, along with numerous trial-and-error cycles.

TRI’s new breakthrough could speed and improve that process in multiple ways.

Robots taught 60 ‘difficult’ skills without a single line of new code

TRI says it has already taught robots “more than 60 difficult, dexterous skills using the new approach, including pouring liquids, using tools, and manipulating deformable objects”—all without writing a single line of new code.

The only change was supplying the robot with new data, TRI said. To build on that success, the institute has set a target of teaching “hundreds of new skills” by the end of the 2023 and 1,000 by the end of 2024.

Going beyond ‘pick-and-place’ skills

The newly trained robots aren’t limited to simple “‘pick and place” skills, like moving an object to a different location. TRI’s robots “can now interact with the world in varied and rich ways—which will one day allow robots to support people in everyday situations and unpredictable, ever-changing environments,” the institute said.

Russ Tedrake, VP of robotics research at TRI, says the tasks the newly trained robots perform “are simply amazing. Even one year ago, I would not have predicted that we were close to this level of diverse dexterity.”

Tedrake wears multiple hats: He’s also the Toyota Professor of Electrical Engineering and Computer Science, Aeronautics and Astronautics, and Mechanical Engineering at MIT.

“What’s so exciting about this new approach is the rate and reliability with which we can add new skills,” Tedrake said in a statement. “Because these skills work directly from camera images and tactile sensing, using only learned representations, they are able to perform well even on tasks that involve deformable objects, cloth, and liquids—all of which have traditionally been extremely difficult for robots.”

Using ‘haptic demos’ to train the robots

So how does the institute train these robots? TRI says its robot behavior model “learns from haptic demonstrations from a teacher, combined with a language description of the goal. It then uses an AI-based Diffusion Policy to learn the demonstrated skill. This process allows a new behavior to be deployed autonomously from dozens of demonstrations. Not only does this approach produce consistent, repeatable, and performant results, but it does so with tremendous speed.”

The training utilizes Diffusion Policy, which was developed by TRI and its collaborators in Professor Shuran Song’s group at Columbia University. The new, generative-AI approach to behavior learning. is said to enable “easy and rapid behavior teaching” based on demonstration.

The training also leverages a model-based design for robotics called Drake that provides TRI with “a cutting-edge toolbox and simulation platform,” offering a “high degree of realism” that allows roboticists to develop in both simulation and in reality at “a dramatically increased scale and velocity.” TRI’s internal robot stack is built using Drake’s optimization and systems frameworks, the institute said—with Drake being made open source to catalyze work across the entire robotics community.

So how does Toyota’s North American headquarters in Plano get involved in all this? The HQ acts as a conduit for bringing TRI’s innovations to market with its direct connections to Toyota’s manufacturing and sales operations, the institute noted.

You can see the robot training in action on this TRI YouTube video.

If you want to learn more and hear directly from the TRI robotics research team, the institute is hosting a LinkedIn Live Q&A session on October 4th from Noon to 12:30 pm Central Time.  You can sign up for the event on TRI’s LinkedIn page.

For more of who said what about all things North Texas, check out Every Last Word.

Get on the list.
Dallas Innovates, every day.

Sign up to keep your eye on what’s new and next in Dallas-Fort Worth, every day.