Unsupervised optical flow with globally optimized cost volume

Abstract

A wide variety of applications rely on computer vision to recover properties of the 3D world from 2D input images. Optical flow is a fundamental task for applications that require motion estimation. Large displacements and repetitive patterns in the images make optical flow estimation a challenging problem. Deep learning has pushed the state-of-the-art on various computer vision tasks, including optical flow estimation. To our knowledge, the supervised learning models have achieved the best performance in optical flow estimation. Supervised learning requires large datasets with ground truth for training, but real image datasets for optical flow are scarce and hard to generate, consequently models are trained with synthetic images instead. Datasets with real images for unsupervised learning of optical flow estimation are abundant and easy to obtain, as ground truth is not required. Images of the same domain can be used for inference and training of the optical flow estimation model with unsupervised learning. For these reasons, unsupervised learning of optical flow estimation is an interesting research field. In this study, we investigate two potential enhancements for the UFlow model, an unsupervised learning optical flow estimation model, with the aim of achieving improved accuracy in scenarios involving large displacements (40 pixels or more). Our focus centers on refining the performance of the model by addressing the cost volume layer. Our contributions are: 1. The implementation of global cost volume computation for UFlow and its evaluation in the two higher levels of the feature pyramid. 2. The integration of globally optimized cost volume (GOCor) with UFlow and its evaluation. 3. The evaluation of the combined effects of global cost volume and GOCor. We discovered that the usage of global cost volume and GOCor increases significantly the computational requirements to train and infer the optical flow, namely the GPU memory required. The results obtained indicate that the concurrent use of GOCor and global cost volume does not yield gains in the optical flow estimation for large displacements, but the sole use of GOCor does.