Weakly Supervised Segmentation of Underwater Imagery

Segmentation of Underwater Imagery

Project dates: 2020 - Ongoing

Introduction

This project explores innovative deep learning strategies for analysing underwater imagery, with significant implications for ecological monitoring. The project is a collaboration between Scarlett Raine and several researchers at the QUT Centre for Robotics and with CSIRO’s Data61, and has been running since 2020.

The project reduces label dependency for training image segmentation models for underwater environments. Underwater surveys provide long-term data for informing management strategies, monitoring and identifying changes in coral reef health, and estimating blue carbon stocks. Recent advances in broad-scale survey methods, such as using semi- and fully-autonomous underwater vehicles, have increased the speed and extent of marine surveys; however, these techniques generate large quantities of imagery which must be analysed by domain experts. This project addresses the need for accurate, automated processing of underwater imagery by reducing the reliance on fully supervised approaches and designing novel weakly supervised approaches which can be trained from fewer labels.

This project is comprised of a sequence of publications which cover four weak annotation styles that significantly reduce the label effort by domain experts: patch-based labels, image-level labels, sparse random point labels, and human-in-the-loop point labels. The project establishes new benchmarks for both coral and seagrass segmentation tasks and makes significant contributions towards reducing the supervision required for automated processing of underwater coral reef and seagrass imagery. The project also creates exciting research opportunities in adaptive navigation for surveys, and ecologist-robot collaboration. To this end, an algorithm for field deployment of coral larvae devices on degraded areas of the reef will be developed as part of this project, enabling large-scale coral re-seeding.

Motivation

Marine surveys are critical for scientists to monitor long term changes in the health of underwater ecosystems and to enable evidence-based decision-making on reef management strategies. Traditionally, surveys are performed manually by divers, snorkellers or using helicopters. However, the range and accuracy of manual surveys is limited and species identification requires visual observation by marine ecologists. Therefore, broad-scale survey methods are becoming increasingly prevalent and utilise remote sensing, drones, towed underwater vehicles, and autonomous underwater vehicles.  The significant increase in the amount of data generated by these methods requires automated methods, such as machine learning and deep learning, to process the information and provide analysis to ecologists in a timely and accurate manner.

Rangerbot
The Rangerbot developed by Matt Dunbabin, collecting images of the Great Barrier Reef (photo credit: Serena Mou).

Semantic segmentation is a computer vision task in which the class of every pixel in a query image is predicted. In recent years, it has become common to train deep learning models to perform semantic segmentation.  These models are often trained using full supervision, i.e. on pairs of images and dense ground truth masks, where the mask is comprised of a class label for every corresponding pixel in the paired image. These dense masks are typically created by human experts who manually label each pixel.  However, underwater imagery has visually unique characteristics which complicate the process of labelling images.  The image complexity and level of domain expertise required to accurately determine the species of each pixel makes it impossible to use crowd-sourcing annotation services such as Amazon Mechanical Turk, and prohibitively costly, time-consuming and difficult for domain experts to provide this level of annotation.

Aims

This project therefore proposes approaches which enable automated processing of underwater imagery without requiring collection and pixel-wise annotation of large quantities of images.  This involves designing deep learning models that require fewer labels, and therefore a weaker supervisory signal, during training.  This project proposes four novel deep learning approaches for weakly supervised segmentation of seagrass meadow and coral reef underwater imagery, spanning four different annotation styles: patch labels, image labels, sparse random point labels, and human-in-the-loop point labels.

This project also considers the development of an algorithm for location guidance of coral larvae devices to suitable substrate types, and enables compliance monitoring through detection and tracking of coral devices post-deployment.  Coral devices have been equipped with temperature-resilient coral larvae, and can improve coral coverage when deployed to degraded areas of the reef.  It is critical that coral devices are deployed to suitable substrate types.  Accurately tracking the deployment of the devices and recording the locations of the devices enables long-term monitoring of intervention outcomes.

Publications

Scarlett Raine presented at the "Research in Focus" event in 2021.
Scarlett Raine presented at the “Research in Focus” event in 2021.

Funding / Grants

  • QUT Centre for Robotics & CSIRO Data 61 (2020)

Chief Investigators

Other Team Members