Automatic Text Illustration in Augmented Reality

Aisle of a supermarket with sign hanging on the roof that shows emojis added by an app where text indicating what categories of items are available in the aisle

The purpose of this study, which is part of the ARC project “A Pictorial Communication Framework for Inclusion“, is to explore the design of an application that uses pictures to offer people who cannot read a visual interpretation of what is written in an everyday environment. For example, menus at a cafĂ© or direction signs in a shopping centre could be replaced on the screen of a mobile device by emojis.

Through this study, we also seek to understand which pictures should be offered as replacement for text, for example, emojis, photographs, or a combination. You can see a video of this in the AR Cafe app here: DEMO VIDEO AR CAFE

Finally, we will be observing how the pictures offered on the screen may become a tool for expressing choices, or starting a conversation, for people with intellectual disability who prefer to use images to communicate.


Laurianne Sitbon

William Ma (VRES, Bachelor of IT)

Liam Percy (Honours Engineering Student, Graduated in 2021)

Tia Chapple (Honours Engineering Student, Graduated in 2021)