Ethomics is a field of study concerned with the applicationof computer vision techniques to characterise animal behaviour. The Ethscope platform, allows for high throughput ethomic studies with Drosophila Melanogaster (the common fruit fly), an important model organism for behavioural studies. The Gilestro lab has previously developed an offline multiple object tracker to simultaneously track flies in Ethoscope video footage. This tracker is based on the Viola-Jones object detection framework, a fast yet inaccurate method of detection that often underpredicts the number of flies in each frame of video.
This project develops two new tracking pipelines that iterate upon the original. The first marginally improves the detection accuracy of the Viola-Jones framework by extending the dataset used to train it. This is then combined with Simple Online Realtime Tracking (SORT) – a robust online tracking method with a low computational overhead that can be deployed on an Ethoscope to track fruitflies in real time.
The second pipeline replaces the Viola-Jones method with FasterR-CNN, a state-of-the-art Convolutional Neural Network based object detection system that greatly improves the performance of the tracker. This is again combined with the SORT tracking system to create a formidable offline tracking pipeline.
In addition, a third pipeline was developed using another contemporary Convolutional Neural Network object detection system (YOLOv4) in conjunction with DeepSORT. DeepSORT is an extension of SORT that can incorporate a measure of visual similarity in it's assignment of detections to individual tracks. However, the tracking system performed poorly implying fruit flies contain too much visual redundance to deliniate individuals.
Aims: The Ethoscope platform is a modular, Raspberry Pi based system designed for high throughput ethomic studies with Drosophila Melanogaster. The Gilestro Lab have developed an offline multiple object tracker based around the Viola-Jones object detection framework – This project develops several pipelines to improve the accuracy of this model. Each pipeline consists of an object detection method (returning coordinates of detected Drosophila in each frame) and an object tracking method (assigning each detection to individual tracks).
Methods: The first pipeline enriched the Viola-Jones training dataset to improve the accuracy of the classifier and implemented SORT (Simple Online Realtime Tracking) to allow for real-time tracking. The second pipeline replaced the Viola-Jones tracking method for a state-of-the-art Regional Convolutional Network, Faster R-CNN, in an effort to increase the accuracy of detection and therefore tracking. The third pipeline utilised another contemporary Convolutional Neural Network based object detection system (YOLOv4) in combination with DeepSORT – an extension of SORT that incorporates a measure of visual similarity when assigning detections to tracks.
Results: The enriched Viola-Jones + SORT pipeline achieved a minor improvement in accuracy with a high detection framerate allowing it to be deployed on the ethoscope platform for real-time tracking. The Faster R-CNN+ SORT pipeline achieved object detection an order of magnitude more accurate than the Viola-Jones method creating a formidable offline tracking system. The tracking obtained from the YOLOv4/DeepSORT pipeline performed worse than all other pipelines indicating Drosophila contain too much visual redundance to deliniate individuals.
Statement of Problem, Research aims and Information on the various computer vision techniques employed in this study
Learn More
