ReplicaCAD sequence
ScanNet sequence
Abstract
We present iSDF, a continual learning system for real-time signed distance field (SDF) reconstruction. Given a stream of posed depth images from a moving camera, it trains a randomly initialised neural network to map input 3D coordinate to approximate signed distance. The model is self-supervised by minimising a loss that bounds the predicted signed distance using the distance to the closest sampled point in a batch of query points that are actively sampled. In contrast to prior work based on voxel grids, our neural method is able to provide adaptive levels of detail with plausible filling in of partially observed regions and denoising of observations, all while having a more compact representation. In evaluations against alternative methods on real and synthetic datasets of indoor environments, we find that iSDF produces more accurate reconstructions, and better approximations of collision costs and gradients useful for downstream planners in domains from navigation to manipulation.
Video
Method
iSDF is a system for real-time SDF reconstruction that takes as input a stream posed depth images captured by a moving camera and, during online operation, learns a function that approximates the true signed distance field of the environment. The signed distance function is modelled by a multilayer perceptron (MLP), that maps a 3D coordinate to the signed distance value at that point. The model is initialised with random weights and is optimised in real-time with respect to incoming measurements.
More Results
Here we show results comparing iSDF to prior work Voxblox and KinectFusion+. KinectFusion+ is our own baseline implementation that combines KinectFusion with GPU based wavefront propagation.
We evaluate on sequences generated from the ReplicaCAD dataset and ScanNet sequences.
SDF Slices
Here we show slices at constant height of the reconstructed SDF at the end of the sequence. iSDF is the only method that can make predictions outside of the visible region (greyed for other methods) while Voxblox also have significant gaps (shown by the red boxes).
Zero level sets
Surface meshes are produced by querying the predicted SDF on a uniform grid and then running marching cubes to find the zero crossing level set. iSDF is the only method to produces a complete and denoised mesh.
Positive characteristics of iSDF
Left: Although the camera is mounted at a fixed height, iSDF can reconstruct a plausible SDF at floor level which is only partially observed.
Middle: The scene is viewed from the green arrow. Despite only observing the nearside of the beanbag, iSDF can sensibly complete the backside of the object. This is in constrast to prior work which is not predictive.
Right: The camera approaches close to a salt shaker as if it intends to manipulate it. iSDF is the only method that can operate at multiple levels of detail and reconstruct both at room scale and at the scale of small objects.
Quantitative Results
iSDF reconstructs the most accurate signed distance fields, acheiving an SDF error less than 6cm for all sequences. iSDF also gives better approximations of collision costs and gradients demonstrating its utility for downstream planners in both navigation and manipulation.
