In a groundbreaking move, Google’s DeepMind Robotics researchers have unveiled innovative methods for training robots that could reshape the landscape of generative AI and robotics in 2024. Traditionally, robots have been confined to repetitive tasks, limiting their adaptability to changes or errors. The team’s newly introduced AutoRT is a game-changer, harnessing large foundational models for better situational awareness and the ability to manage a fleet of robots working collaboratively.
AutoRT utilizes a Visual Language Model (VLM) for enhanced environmental understanding and employs a Large Language Model (LLM) to suggest tasks, reducing the necessity for hard-coded skills. Tested rigorously over the past seven months, AutoRT demonstrated its prowess by orchestrating up to 20 robots simultaneously, handling a total of 52 devices and collecting an impressive dataset of 77,000 trials, including more than 6,000 tasks.
In addition to AutoRT, the team introduced RT-Trajectory, a revolutionary approach leveraging video input for robotic learning. Unlike conventional methods using YouTube videos, RT-Trajectory overlays a two-dimensional sketch of the robot’s arm over the video, significantly enhancing the learning process. This method, showcasing double the success rate of previous training at 63%, not only advances the efficiency of robot-control policies but also taps into previously underutilized rich robotic-motion information present in datasets.
The significance of these advancements lies in the potential to move beyond the limitations of single-purpose robots, opening doors to more adaptive and intelligent agents. These developments mark a crucial step towards robots comprehending natural language commands, reducing the need for rigid programming and unlocking knowledge from existing datasets. As we step into 2024, the intersection of generative AI and robotics is set to redefine the capabilities of non-human workers, paving the way for a new era of digital employees.
Key Highlights:
- AutoRT Revolutionizes Robotic Understanding: Google’s DeepMind Robotics researchers introduce AutoRT, a groundbreaking system that harnesses large foundational models, including Visual Language Models (VLM) and Large Language Models (LLM), to enhance robotic situational awareness. This innovation enables robots to manage complex tasks collaboratively, marking a departure from the traditional single-purpose robotic approach.
- Adaptive Robot Orchestration: AutoRT has been rigorously tested over seven months, demonstrating its ability to orchestrate up to 20 robots simultaneously, handling 52 different devices. The system collected a vast dataset of 77,000 trials, including over 6,000 tasks, showcasing its adaptability and efficiency in varied scenarios.
- RT-Trajectory Enhances Robotic Learning: DeepMind introduces RT-Trajectory, leveraging video input for robotic learning. This method, with a 63% success rate compared to 29% in previous training, overlays a two-dimensional sketch of the robot’s arm on video, offering a novel approach to training and tapping into underutilized rich robotic-motion information in datasets.
- Unlocking Natural Language Understanding: The combination of VLM and LLM in AutoRT signifies a move towards robots understanding more natural language commands, reducing the reliance on hard-coded skills. These advancements mark a pivotal step in developing intelligent agents capable of comprehending and executing tasks based on human-like communication.
- 2024: A Year of AI and Robotics Fusion: As we enter 2024, the intersection of generative AI and robotics is poised to redefine the capabilities of non-human workers. AutoRT and RT-Trajectory represent a significant leap towards creating digital employees that can adapt, learn, and perform tasks efficiently in diverse and dynamic environments.
Reference:
