ArXiv Preprint
Neural Radiance Fields (NeRF) is a powerful novel technology for the
reconstruction of 3D scenes from a set of images captured by static cameras.
Renders of these reconstructions could play a role in virtual presence in the
operating room (OR), e.g. for training purposes. In contrast to existing
systems for virtual presence, NeRF can provide real instead of simulated
surgeries. This work shows how NeRF can be used for view synthesis in the OR. A
depth-supervised NeRF (DS-NeRF) is trained with three or five synchronised
cameras that capture the surgical field in knee replacement surgery videos from
the 4D-OR dataset. The algorithm is trained and evaluated for images in five
distinct phases before and during the surgery. With qualitative analysis, we
inspect views synthesised by a virtual camera that moves in 180 degrees around
the surgical field. Additionally, we quantitatively inspect view synthesis from
an unseen camera position in terms of PSNR, SSIM and LPIPS for the colour
channels and in terms of MAE and error percentage for the estimated depth.
DS-NeRF generates geometrically consistent views, also from interpolated camera
positions. Views are generated from an unseen camera pose with an average PSNR
of 17.8 and a depth estimation error of 2.10%. However, due to artefacts and
missing of fine details, the synthesised views do not look photo-realistic. Our
results show the potential of NeRF for view synthesis in the OR. Recent
developments, such as NeRF for video synthesis and training speedups, require
further exploration to reveal its full potential.
Beerend G. A. Gerats, Jelmer M. Wolterink, Ivo A. M. J. Broeders
2022-11-22
