manvr3d: A Platform for Human-in-the-loop Cell Tracking in Virtual Reality

Keywords

Systems Biology, Virtual Reality, Microscopy, Cell Tracking, Volume Rendering, Eye Tracking

Abstract

We propose manvr3d, a VR platform for immersive, AI-assisted human-in-the-loop cell tracking. Life scientists reconstruct the developmental history of organisms at the cellular level by analyzing 3D time-lapse microscopy images acquired at high spatio-temporal resolution. However, reconstruction of cell trajectories and lineage trees is a highly time consuming and error prone task. Common tools are often limited to 2D image display, which greatly limits spatial understanding and navigation. Deep Learning-based algorithms accelerate this process, yet depend heavily on manually-annotated, high-quality ground truth data and curation. In this work, we bridge the gap between Deep Learning-based cell tracking software and 3D/VR visualization to create a hybrid AI-human-in-the-loop cell tracking system. We lift the incremental annotation, training and proofreading loop of the deep learning model into the third dimension and apply natural user interfaces like hand gestures and eye tracking to accelerate the cell tracking workflow for life scientists. We present here the technical architecture of our platform and first analysis of performance. Our code is released open source.