Segmentation of Underwater Imagery
Project dates: 2020 - Ongoing
Introduction
This project explores innovative deep learning strategies for analysing underwater imagery, with significant implications for ecological monitoring. The project is a collaboration between Scarlett Raine and several researchers at the QUT Centre for Robotics and with CSIRO’s Data61, and has been running since 2020.
The project reduces label dependency for training image segmentation models for underwater environments. Underwater surveys provide long-term data for informing management strategies, monitoring and identifying changes in coral reef health, and estimating blue carbon stocks. Recent advances in broad-scale survey methods, such as using semi- and fully-autonomous underwater vehicles, have increased the speed and extent of marine surveys; however, these techniques generate large quantities of imagery which must be analysed by domain experts. This project addresses the need for accurate, automated processing of underwater imagery by reducing the reliance on fully supervised approaches and designing novel weakly supervised approaches which can be trained from fewer labels.
This project is comprised of a sequence of publications which cover four weak annotation styles that significantly reduce the label effort by domain experts: patch-based labels, image-level labels, sparse random point labels, and human-in-the-loop point labels. The project establishes new benchmarks for both coral and seagrass segmentation tasks and makes significant contributions towards reducing the supervision required for automated processing of underwater coral reef and seagrass imagery. The project also creates exciting research opportunities in adaptive navigation for surveys, and ecologist-robot collaboration. To this end, an algorithm for field deployment of coral larvae devices on degraded areas of the reef will be developed as part of this project, enabling large-scale coral re-seeding.
Motivation
Marine surveys are critical for scientists to monitor long term changes in the health of underwater ecosystems and to enable evidence-based decision-making on reef management strategies. Traditionally, surveys are performed manually by divers, snorkellers or using helicopters. However, the range and accuracy of manual surveys is limited and species identification requires visual observation by marine ecologists. Therefore, broad-scale survey methods are becoming increasingly prevalent and utilise remote sensing, drones, towed underwater vehicles, and autonomous underwater vehicles. The significant increase in the amount of data generated by these methods requires automated methods, such as machine learning and deep learning, to process the information and provide analysis to ecologists in a timely and accurate manner.
Semantic segmentation is a computer vision task in which the class of every pixel in a query image is predicted. In recent years, it has become common to train deep learning models to perform semantic segmentation. These models are often trained using full supervision, i.e. on pairs of images and dense ground truth masks, where the mask is comprised of a class label for every corresponding pixel in the paired image. These dense masks are typically created by human experts who manually label each pixel. However, underwater imagery has visually unique characteristics which complicate the process of labelling images. The image complexity and level of domain expertise required to accurately determine the species of each pixel makes it impossible to use crowd-sourcing annotation services such as Amazon Mechanical Turk, and prohibitively costly, time-consuming and difficult for domain experts to provide this level of annotation.
Aims
This project therefore proposes approaches which enable automated processing of underwater imagery without requiring collection and pixel-wise annotation of large quantities of images. This involves designing deep learning models that require fewer labels, and therefore a weaker supervisory signal, during training. This project proposes four novel deep learning approaches for weakly supervised segmentation of seagrass meadow and coral reef underwater imagery, spanning four different annotation styles: patch labels, image labels, sparse random point labels, and human-in-the-loop point labels.
This project also considers the development of an algorithm for location guidance of coral larvae devices to suitable substrate types, and enables compliance monitoring through detection and tracking of coral devices post-deployment. Coral devices have been equipped with temperature-resilient coral larvae, and can improve coral coverage when deployed to degraded areas of the reef. It is critical that coral devices are deployed to suitable substrate types. Accurately tracking the deployment of the devices and recording the locations of the devices enables long-term monitoring of intervention outcomes.
Publications
- Raine, S., Marchant, R., Kusy, B., Maire, F., Suenderhauf, N. & Fischer, T. (2024). Human-in-the-Loop Segmentation of Multi-species Coral Imagery. Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition Workshops.
- Raine, S., Marchant, R., Kusy, B., Maire, F. & Fischer, T. (2024). Image Labels Are All You Need for Coarse Seagrass Segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.
- Raine, S., Marchant, R., Kusy, B., Maire, F. & Fischer, T. (2022). Point Label Aware Superpixels for Multi-species Segmentation of Underwater Imagery. IEEE Robotics and Automation Letters, 7(3), 8291-8298.
- Raine, S., Marchant, R., Maire, F., Suenderhauf, N. & Kusy, B. (2021). Towards Dynamic Adaptation of Marine Surveys: Leveraging Fine-grained Segmentation from Sparse Labels. Proceedings of the IEEE International Conference on Robotics and Automation Workshops.
- Raine, S., Marchant, R., Moghadam, P., Maire, F., Kettle, B. & Kusy, B. (2020). Multi-species Seagrass Detection and Classification from Underwater Images. Proceedings of the Digital Image Computing: Techniques and Applications (DICTA).
Funding / Grants
- QUT Centre for Robotics & CSIRO Data 61 (2020)