The revolution in artificial intelligence is stepping out of the digital realm and into the physical world, thanks to groundbreaking work by researchers at the QUT Centre for Robotics. Their latest innovation, “SayPlan,” provides robots with the intelligence of Large Language Models (LLMs) such as ChatGPT, enabling them to understand and interact with the physical world in unprecedented ways. While LLMs have demonstrated remarkable virtual AI assistant capabilities from drafting emails, scheduling meetings, to creating PowerPoint presentations—they have remained confined to the digital sphere, unable to physically engage with the environment around them.
SayPlan is set to change that. At the heart of SayPlan’s breakthrough is its use of 3D Scene Graphs, which provide robots with an understanding of the spatial and semantic layout of their surroundings. This provides LLMs with the necessary grounding required for them to know not just “what” is in the environment, but “where” things are located, enabling the generation of actionable task plans that a robot can physically execute. Whether it’s clearing a dining table or delivering fresh linen, SayPlan equips robots with the ability to perform tasks with a level of common-sense understanding and generality previously unattainable — an embodied AI assistant.
Dr. Krishan Rana who led this research believes the scalability of this framework has the potential to bring the long-awaited service robots to not only our homes but additionally larger public spaces including hospitals and hotels, underscoring its potential to revolutionize the service industry.
For more details, check out the project page with videos of the system in action.
Join us for a PhD!
Check the available PhD topics that build on this work. If you want to pursue a PhD in robotic learning, you can get in contact with Prof Niko Suenderhauf and apply for a fully funded PhD scholarship.