Acoustic Non-Line-of-Sight Imaging | CVPR 2019

David B. Lindell, Gordon Wetzstein, Vladlen Koltun

We introduce a novel approach to seeing around corners using acoustic echoes. A system of speakers emits sound waves which scatter from a wall to a hidden object and back. Microphones capture the timing of the returning echoes, and we use reconstruction algorithms inspired by synthetic aperture radar and seismic imaging to recover the geometry of the hidden object.

Our method can reconstruct hidden objects using inexpensive, off-the-shelf hardware at longer distances with lower exposure times compared to specialized, state-of-the-art optical systems.

ABSTRACT

Non-line-of-sight (NLOS) imaging enables unprecedented capabilities in a wide range of applications, including robotic and machine vision, remote sensing, autonomous vehicle navigation, and medical imaging. Recent approaches to solving this challenging problem employ optical time-of-flight imaging systems with highly sensitive time-resolved photodetectors and ultra-fast pulsed lasers. However, despite recent successes in NLOS imaging using these systems, widespread implementation and adoption of the technology remains a challenge because of the requirement for specialized, expensive hardware. We introduce acoustic NLOS imaging, which is orders of magnitude less expensive than most optical systems and captures hidden 3D geometry at longer ranges with shorter acquisition times compared to state-of-the-art optical methods. Inspired by hardware setups used in radar and algorithmic approaches to model and invert wave-based image formation models developed in the seismic imaging community, we demonstrate a new approach to seeing around corners.

Hardware Prototype

Photograph of the prototype system. The prototype comprises a linear array of 16 speakers and microphones mounted vertically on a 1 m translation stage. Power amplifiers and a set of audio interfaces drive the speakers and record from the microphones.

Visualizations

Visualization of a 2D acoustic NLOS scanning system.

A similar problem arises in seismology, where shockwaves are used to probe and reconstruct underground surfaces.

Press

Recovering hidden Shape from Sound

Modulated sound waves are emitted from a speaker, travel around the corner to a hidden object, and are then recorded by a microphone as they reflect back. The processed measurements (bottom left) contain peaks indicating the path lengths of sound which travels directly from the speaker to the microphone (A, peak is clipped), to the wall and back (B), and also to the hidden object and back (C). Such measurements are captured for a range of speaker and microphone positions to reconstruct the 3D geometry of the hidden object (bottom right).

Acoustic Transmit Signal

Due to the mirror-like scattering of the wall at acoustic wavelengths, the measurements appear to be captured from a mirrored volume located behind the wall, as if the wall were transparent. The transmit signal is a linear ramp in frequency over time. For a single reflector, the return signal is a delayed version of the transmit signal (top right). The receive and transmit signals are mixed together
and Fourier transformed, producing a sharp peak at a frequency proportional to the distance of the reflector (bottom right).

FILES

technical paper (link)
technical paper supplement (link)
code (link)

CITATION

David B. Lindell, Gordon Wetzstein, and Vladlen Koltun. 2019. Acoustic Non-Line-of-Sight Imaging. In Proc. CVPR.

BibTeX

@inproceedings{Lindell:2019:Acoustic,
author = {David B. Lindell and Gordon Wetzstein and Vladlen Koltun},
title = {{Acoustic non-line-of-sight imaging}},
journal = {Proc. CVPR},
year = {2019}
}

Results

Reconstructed letter “H” captured from around the corner. Reconstructions are shown using a ‘confocal’ subset of closely spaced speakers and microphones, the ‘nonconfocal’ set of all speaker and microphone pairs, and with additional priors on the reconstructed volume.

Reconstruction of multiple acoustic corner reflectors from around the corner.

Comparison to an optical NLOS system. Optical reconstruction of the more distant letter fails due to limited signal. The acoustic system recovers both letters at increased distance from the wall and with much lower acquisition time.

Related Projects

You may also be interested in related projects, where we have developed non-line-of-sight imaging systems:

Metzler et al. 2021. Keyhole Imaging. IEEE Trans. Computational Imaging (link)
Lindell et al. 2020. Confocal Diffuse Tomography. Nature Communications (link)
Young et al. 2020. Non-line-of-sight Surface Reconstruction using the Directional Light-cone Transform. CVPR (link)
Lindell et al. 2019. Wave-based Non-line-of-sight Imaging using Fast f-k Migration. ACM SIGGRAPH (link)
Heide et al. 2019. Non-line-of-sight Imaging with Partial Occluders and Surface Normals. ACM Transactions on Graphics (presented at SIGGRAPH) (link)
Lindell et al. 2019. Acoustic Non-line-of-sight Imaging. CVPR (link)
O’Toole et al. 2018. Confocal Non-line-of-sight Imaging based on the Light-cone Transform. Nature (link)

and direct line-of-sight or transient imaging systems:

Bergman et al. 2020. Deep Adaptive LiDAR: End-to-end Optimization of Sampling and Depth Completion at Low Sampling Rates. ICCP (link)
Nishimura et al. 2020. 3D Imaging with an RGB camera and a single SPAD. ECCV (link)
Heide et al. 2019. Sub-picosecond photon-efficient 3D imaging using single-photon sensors. Scientific Reports (link)
Lindell et al. 2018. Single-Photon 3D Imaging with Deep Sensor Fusions. ACM SIGGRAPH (link)
O’Toole et al. 2017. Reconstructing Transient Images from Single-Photon Sensors. CVPR (link)

Acknowledgements

This project was supported by a Stanford Graduate Fellowship, an NSF CAREER Award (IIS 1553333), a Terman Faculty Fellowship, a Sloan Fellowship, by the KAUST Office of Sponsored Research through the Visual Computing Center CCF grant, the Center for Automotive Research at Stanford (CARS), the DARPA REVEAL program, and a PECASE from the ARO.