Robot Learning for Everyday Tasks with Large Language Models, Imitation Learning and NeRFs

How can robots learn everyday tasks, such as preparing dinner, tidying up an apartment, or assisting humans in their everyday domestic chores? 

I have multiple PhD projects available in the area of robotic learning, combining modern techniques such as imitation learning, LLMs, Vision-Language Models, and reinforcement learning with classical control and navigation, but also with a fresh take on representing the environment with NeRFs or Gaussian Splatting.

These PhD projects build on our previous work such as SayPlan (oral presentation at CoRL 2023) or ParticleNeRF (best paper honourable mention at WACV 2024).

Research Questions / Topics

Research questions for the available topics include:

  • How can robots learn tasks by observing humans, for example from a single demonstration or from the youtube videos? How can the kinematic chain of a human be mapped to the kinematics of a robot in the process?
  • How can Large Language Models or Vision Language Models be used effectively to guide robot navigation and interaction with the world?
  • What is the best way for robots to learn a library of skills that generalise and transfer to unseen objects?
  • How can LLMs be used to compose such base skills into more and more complex tasks?
  • How can an environment be represented to ease robot learning? Are modern mapping techniques based on NeRFs or Gaussian Splatting helpful?
  • Can ad-hoc simulations of the environment be used as thought-experiments to guide interaction and manipulation of complex scenes, e.g. in clutter or manipulating mechanisms?
  • How can robots improvise when they don’t succeed at first? How can they recover from failures and mistakes, and learn from it to do better next time?

To answer these questions, you will work at the forefront of robotic learning and AI and become an expert in areas such as LLMs, VLMs,  imitation learning, reinforcement learning, NeRFs, Gaussian Splatting, and physics simulations.

You would work with our amazing fleet of robots—both stationary and mobile—and our instrumented experimentation setup, which lets us train and evaluate robots for various chores. You’ll be part of an amazing team of researchers, postdocs and other PhD students at the Centre for Robotics.

Interested in a PhD or MPhil?

Get in touch with Prof Niko Suenderhauf to apply for a PhD or MPhil position and a scholarship.

In your email, you should include the following:

  • your CV
  • academic transcripts of your recent degree(s)
  • a one-page description of your research interests, describing which topic you are interested in, and why
  • a short summary about a previous research experience you are most proud of

Please check your eligibility before applying.

Please understand that due to the high volume of PhD applications I receive, I might not be able to respond to your email. This is especially the case if you did not attach the documents listed above, or if your research interests are not aligned with mine. I will be in touch if I think you would be a good fit to our lab.

 


Chief Investigators