Hybrid optical-electronic convolutional neural networks for image classification | Scientific Reports 2018

Julie Chang, Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, Gordon Wetzstein

Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification.

ABSTRACT

Convolutional neural networks (CNNs) excel in a wide variety of computer vision applications, but their high performance also comes at a high computational cost. Despite efforts to increase efficiency both algorithmically and with specialized hardware, it remains difficult to deploy CNNs in embedded systems due to tight power budgets. Here we explore a complementary strategy that incorporates a layer of optical computing prior to electronic computing, improving performance on image classification tasks while adding minimal electronic computational cost or processing time. We propose a design for an optical convolutional layer based on an optimized diffractive optical element and test our design in two simulations: a learned optical correlator and an optoelectronic two-layer CNN. We demonstrate in simulation and with an optical prototype that the classification accuracies of our optical systems rival those of the analogous electronic implementations, while providing substantial savings on computational cost.

CITATION

J. Chang, V. Sitzmann, X. Dun, W. Heidrich, G. Wetzstein. “Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification”, Scientific Reports, 2018.

BibTeX

@article{chang:2018:HybridCNN,
author = {Julie Chang and Vincent Sitzmann and Xiong Dun and Wolfgang Heidrich and Gordon Wetzstein},
title = {Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification},
journal = {Scientific Reports},
year = {2018},
}

 

Acknowledgements

The authors would like to thank Evan Peng for recommending and coordinating fabrication options for the phase mask. The authors also recognize Kevin Ting for rapid prototyping of the rotation mount and Matthew O’Toole, Donald Dansereau, and Joseph Goodman for valuable discussions. This project was generously supported a Sloan Research Fellowship and an NSF CAREER award (IIS 1553333).

 

Related Projects

You may also be interested in related projects, where we apply the idea of Deep Optics, i.e. end-to-end optimization of optics and image processing, to other applications, like monocular depth estimation, 3D object detection, image classification, extended depth-of-field imaging, superresolution imaging, or optical computing.

  • Chang and Wetzstein. 2019. Deep Optics for Monocular Depth Estimation and 3D Object Detection. arXiv (link)
  • Sitzmann et al. 2018. End-to-end Optimization of Optics and Imaging Processing for Achromatic Extended Depth-of-field and Super-resolution Imaging. ACM SIGGRAPH 2018 (link)

ADDITIONAL MATERIAL

 

diffractive optical element

This “photonic AI chip” (i.e. a diffractive optical element or phase mask) is integrated into a conventional lens and lets the optics compute the first layer of a CNN at the speed of light.

Close-up of optimized optical element illuminated with an LED from behind showing the macroscale textures and patterns of the “photonic AI chip”.

Close-up of optical element placed in front of printed text showing how an image gets processed by the “photonic AI chip”.

Optical diffraction pattern (right) formed by directing a single laser beam through the optimized optical element (middle).

Optical convolutional layer design. a) Diagram of a 4 f system that implements optical convolutional (opt-conv) layers by placing a phase mask in the Fourier plane. b) The standard components of a digital convolutional layer, including an input image, a stack of convolutional kernels, and a corresponding output volume. c) The equivalent components in an opt-conv layer, where the kernels and outputs are tiled in a 2D array instead of stacked in the depth dimension.

 

Example images of hybrid optical-electronic camera. An input image (left) is optically processed by our diffractive optical element to create multiple copies of the image on the sensor, each filtered with a different, optimized convolution kernels (center). The electronic portion of the neural network then uses the captured image to classify the object.

 

Learned optical correlator. a) Schematic of an optical correlator, where the conv block consists of the 4 f system shown in Fig. 1. b) Characteristic optimized kernels of a multichannel unconstrained digital convolutional layer, a multichannel nonnegative digital convolutional layer, a single channel opt-conv layer with tiled kernels, and the PSF produced by phase mask optimization with the previous optimized tiled kernels as the target.

 

Hybrid optoelectronic CNN. a) Schematic of a model with a single opt-conv layer, after which the sensor image is processed and fed into subsequent digital CNN layers. b) The optimized phase mask template and microscope images of the fabricated phase mask, at different zoom levels. c) Comparison of simulated and captured versions of the PSF produced by the phase mask, a sample input image, the respective sensor image, and pseudonegative sub-images after subtraction of corresponding positive (top rows) and negative (bottom rows) sub-images.

Related Projects

You may also be interested in related projects, where we apply the idea of Deep Optics, i.e. end-to-end optimization of optics and image processing, to other applications, like image classification, extended depth-of-field imaging, superresolution imaging, or optical computing.

  • Wetzstein et al. 2020. AI with Optics & Photonics. Nature (review paper, link)
  • Martel et al. 2020. Neural Sensors. ICCP & TPAMI 2020 (link)
  • Dun et al. 2020. Learned Diffractive Achromat. Optica 2020 (link)
  • Metzler et al. 2020. Deep Optics for HDR Imaging. CVPR 2020 (link)
  • Chang et al. 2019. Deep Optics for Depth Estimation and Object Detection. ICCV 2019 (link)
  • Peng et al. 2019. Large Field-of-view Imaging with Learned DOEs. SIGGRAPH Asia 2019 (link)
  • Chang et al. 2018. Hybrid Optical-Electronic Convolutional Neural Networks with Optimized Diffractive Optics for Image Classification. Scientific Reports (link)
  • Sitzmann et al. 2018. End-to-end Optimization of Optics and Imaging Processing for Achromatic Extended Depth-of-field and Super-resolution Imaging. ACM SIGGRAPH 2018 (link)