iSDF: Real-Time Neural Signed Distance Fields
for Robot Perception

Joseph Ortiz1,2 Alexander Clegg2 Jing Dong3 Edgar Sucar1 David Novotny2 Michael Zollhoefer3 Mustafa Mukadam2
1Imperial College London   2Meta AI   3Reality Labs Research

RSS 2022

ReplicaCAD sequence


ScanNet sequence


Abstract

We present iSDF, a continual learning system for real-time signed distance field (SDF) reconstruction. Given a stream of posed depth images from a moving camera, it trains a randomly initialised neural network to map input 3D coordinate to approximate signed distance. The model is self-supervised by minimising a loss that bounds the predicted signed distance using the distance to the closest sampled point in a batch of query points that are actively sampled. In contrast to prior work based on voxel grids, our neural method is able to provide adaptive levels of detail with plausible filling in of partially observed regions and denoising of observations, all while having a more compact representation. In evaluations against alternative methods on real and synthetic datasets of indoor environments, we find that iSDF produces more accurate reconstructions, and better approximations of collision costs and gradients useful for downstream planners in domains from navigation to manipulation.



Video

Method

iSDF is a system for real-time SDF reconstruction that takes as input a stream posed depth images captured by a moving camera and, during online operation, learns a function that approximates the true signed distance field of the environment. The signed distance function is modelled by a multilayer perceptron (MLP), that maps a 3D coordinate to the signed distance value at that point. The model is initialised with random weights and is optimised in real-time with respect to incoming measurements.

More Results

Here we show results comparing iSDF to prior work Voxblox and KinectFusion+. KinectFusion+ is our own baseline implementation that combines KinectFusion with GPU based wavefront propagation.

We evaluate on sequences generated from the ReplicaCAD dataset and ScanNet sequences.

SDF Slices

Here we show slices at constant height of the reconstructed SDF at the end of the sequence. iSDF is the only method that can make predictions outside of the visible region (greyed for other methods) while Voxblox also have significant gaps (shown by the red boxes).



Zero level sets

Surface meshes are produced by querying the predicted SDF on a uniform grid and then running marching cubes to find the zero crossing level set. iSDF is the only method to produces a complete and denoised mesh.



Positive characteristics of iSDF

Left: Although the camera is mounted at a fixed height, iSDF can reconstruct a plausible SDF at floor level which is only partially observed.

Middle: The scene is viewed from the green arrow. Despite only observing the nearside of the beanbag, iSDF can sensibly complete the backside of the object. This is in constrast to prior work which is not predictive.

Right: The camera approaches close to a salt shaker as if it intends to manipulate it. iSDF is the only method that can operate at multiple levels of detail and reconstruct both at room scale and at the scale of small objects.



Quantitative Results

iSDF reconstructs the most accurate signed distance fields, acheiving an SDF error less than 6cm for all sequences. iSDF also gives better approximations of collision costs and gradients demonstrating its utility for downstream planners in both navigation and manipulation.

BibTeX


      @article{Ortiz:etal:iSDF2022,
        title={iSDF: Real-Time Neural Signed Distance Fields for Robot Perception},
        author={Ortiz, Joseph and Clegg, Alexander and Dong, Jing and Sucar, Edgar and Novotny, David and Zollhoefer, Michael and Mukadam, Mustafa},
        journal={arXiv preprint arXiv:2204.02296},
        year={2022}
      }
    

Contact

If you have any questions, please feel free to reach out to Joe Ortiz.