Understanding the geometry of an existing scene and being able to use this knowledge to produce (and refine) data, is an important task in any research field, particularly in the medical domain where the study and understanding of 3D structures of interest play a crucial role in abnormality detection.
For this task, the state-of-the-art methodology is based on neural radiance fields, or in short NeRF models, which are rapidly adopted by the computer vision community and aim to:
-
To produce a three-dimensional representation of a scene, an object or a structure of interest based on a reduced set of images taken from different vantage points;
-
Understand the geometric properties of a scene, providing depth, normal field and occlusion estimation;
-
Generate "new" images of an object or structure of interest from novel views, thereby contributing to the task of data augmentation.
Applying such techniques in the medical field can be beneficial for producing further insights into the structure of a scene. Moreover, being able to generate new data contributes to the further applicability of data-driven machine learning and deep learning techniques in fields where data scarcity is observed[1].
Among other applications related to the medical domain, NeRF based methods have been proposed for the 3D reconstruction of the human body in different poses as can be seen in the figure below. These methods can be employed for building a 3D avatar of the patient, which is one of the tasks performed in the context of iToBoS
Source: Gao, X., Yang, J., Kim, J., Peng, S., Liu, Z., & Tong, X. (2022). Mps-nerf: Generalizable 3d human rendering from multiview images. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[1] Gao, X., Yang, J., Kim, J., Peng, S., Liu, Z., & Tong, X. (2022)