ISSCC 2021 / SESSION 34 / EMERGING IMAGING SOLUTIONS / 34.
34.2 A 21pJ/frame/pixel Imager and 34pJ/frame/pixel Image
Processor for a Low-Vision Augmented-Reality Smart Contact
Lens
Rituraj Singh, Stevo Bailey, Phillip Chang, Ashkan Olyaei, Mohammad Hekmat,
Renaldi Winoto
Mojo Vision, Saratoga, CA
2021 IEEE International Solid- State Circuits Conference (ISSCC) | 978-1-7281-9549-0/20/$31.00 ©2021 IEEE | DOI: 10.1109/ISSCC42613.2021.9365771
Over 253 million people worldwide suffer from permanent visual impairments (e.g.,
glaucoma, diabetic retinopathy, retinitis pigmentosa and macular degeneration) that
cannot be rectified using refractive correction or surgery. Such patients rely on canes,
monoculars, or service dogs for mobility, which while effective, do not provide rich
information and are not hands-free. Recently, low-vision augmented-reality (AR)
headsets have been developed to provide hands-free, real-time, enhanced-image
overlays that highlight objects and obstacles. However, social stigma and form factor
impede their adoption [1]. We introduce a complete, enhanced-image overlay solution
for a socially transparent AR contact lens for low-vision patients that provides a rich set
of controllable image overlays such as contrast enhancement, edge detection, and zoom.
The lens consists of three main components: an outward-looking imager, an image
processor, and a microLED display [2]. The lens captures the scene from the wearer’s
central vision, performs a user-specified image processing task (e.g., edge detection),
and sends the result to the display, which, along with its miniature optics, projects
images directly onto the retina, highlighting outlines of real-world objects.
Figure 34.2.1 shows the lens architecture and its principle of operation. To produce AR
overlays, the eye’s visual axis, the imager, and the display axis are kept near-coaxial.
Putting components inside the contact lens poses three main challenges: limited volume,
scarce power, and need for low latency. Fitting ICs inside the lens and minimizing their
light occlusion in the central vision restricts their die area to roughly 1mm2. The total
lens power budget is also limited, hence low-power design is required. Finally, low
latency is needed to provide a lag-free experience, thereby necessitating integration of
the image processor on the lens. Unlike AR headsets, a contact lens display and imager
can take advantage of being eye-mounted, naturally following the wearer’s gaze point
and significantly reducing the required resolution and power. A 256×256-pixel array
along with the appropriate image enhancements is suitable to help low-vision patients
with visual acuity from 20/75 to 20/400 Snellen.
Conventional image sensors employ column-parallel single-slope ADCs. A column-
parallel structure requires a significant layout area that increases the overall die size.
Moreover, exponentially faster clock generation and distribution is required to achieve
the conversion resolution, which renders them power-inefficient. In this work, we utilize
a passive array of column-sampling capacitors, a single programmable-gain amplifier
(PGA) and a single asynchronous SAR ADC to save power and area. Figure 34.2.2 shows
the image sensor signal chain and its timing diagram. The pixel array is read out, one
row at a time, by sampling a pixel output onto its respective column-parallel capacitor
CIN. The capacitors sequentially transfer their stored charge onto the PGA for
amplification. The PGA output is then quantized by an asynchronous SAR ADC. Utilizing
passive capacitors enables negligible power consumption in the column circuitry while
ensuring fast settling time. Additionally, the asynchronous ADC obviates the need for
an oversampled clock and its associated routing, further saving power. A digital block
generates the required timing signals and provides features such as a choice of
progressive or subsampled read-out mode and power management of analog blocks.
A 2T5 shared-pixel readout architecture is employed. Saving power by lowering pixel
supply voltage leads to SNR degradation. To avoid SNR degradation, pixel current source
IB is duty cycled. The differential column capacitor circuit serves five functions: (1)
storing pixel reset and signal voltages, (2) performing correlated double sampling, (3)
converting pixel output from single-ended to differential, (4) level shifting from the pixel
(3.3V) to analog (1V) voltage domain, and (5) minimizing switch leakage errors. Bottom-
plate sampling is employed to avoid charge-injection errors. After sampling, ΦH is kept
enabled and column decoder switches are sequentially activated to transfer the stored
column charge onto the PGA. A difference charge of -COSVREF is simultaneously injected
to fully utilize the differential input range of the SAR ADC. The PGA’s gain can be adjusted
by varying the feedback capacitance, CFB. Inadvertent charge loss during column address
(COLaddr) switching is avoided by: (1) introducing a half-cycle delay with respect to PGA
clocks Φ1 and Φ2 , and (2) nonoverlapping column decoder logic. The SAR ADC can
operate in 4, 6 and 8 bits-per-pixel modes. The PGA’s power is optimized by shortening
482 •Authorized
2021 IEEE International Solid-State Circuits Conference
licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 15,2021 at 06:58:51 UTC from IEEE Xplore. Restrictions apply.
the duration of Φ1 in 4- and 6-bit modes while still meeting the PGA settling accuracy.
The ADC sampling capacitor array embeds an analog offset subtraction technique to
calibrate accumulated offset errors and adjust the pixel black level prior to signal
quantization.
Figure 34.2.3 shows the imager characterization results. The core imager energy
consumption per frame remains relatively flat across a wide range of frame rates. This
is achieved by powering down the imager during idle periods between image captures.
The imager can also provide an accumulated pixel output per frame, either in conjunction
with progressive image readout or exclusively for ambient light sensing. The
accumulated output is used to compute an ambient light level within the imager field of
view, which is subsequently used to adjust the relative display brightness. For exclusive
ambient sensing operation, sparse array sampling is performed to reduce readout time
and hence power consumption. By utilizing a two-point correction across the three
illuminance ranges, a linear response to ambient brightness is obtained from 0.05 lux
to 260,000 lux.
Figure 34.2.4 shows the lens image-processing architecture. The raster-scanned raw
image output is passed through a linear contrast/brightness filter, followed by a
sequence of two non-linear filter operations. Each non-linear filter consists of two parallel
3×3 convolutions (Gx, Gy), followed by summation, thresholding, and digital gain.
Convolution coefficients, thresholds, and gains are all programmable. Low latency is
achieved by performing image processing on a per-row basis, while small area is
achieved by time-multiplexing the data processing, requiring only four rows of pixels in
memory per filter and sharing the memory across all buffers and filters. Low power is
achieved by minimizing the memory size and memory transactions, so each pixel is read
once from memory and simultaneously used in six convolution operations. Compared
to a single non-linear filter with 5×5 convolutions, two serial non-linear filters with 3×3
convolutions enable a wider variety of image-processing operations for a similar amount
of hardware. This approach, combined with thresholding and pixel binning (BIN) and
oversampling (OS) options, supports versatile image processing. Raw images taken
with the imager shown in Fig. 34.2.3 were fed to the image processor. Figure 34.2.5
shows the measured output of this image processing system as seen on our microLED
display. This figure shows a subset of the feasible image-processing operations,
underlining the versatility of the system.
Figure 34.2.6 shows the comparison with the state-of-the-art low-power image sensors.
The imager achieves 21pJ/frame/pixel and 13pJ/frame/pixel energy efficiency for 8-bit
and 4-bit modes, respectively. Figure 34.2.7 shows the imager die micrograph and
provides a performance summary of the imager and the image processor.
References:
[1] P. Sivakumar et al., “Barriers in Utilisation of Low Vision Assistive Products,” Eye,
vol. 34, pp. 344-351, 2019.
[2] P. Martin, “Mojo Vision nanoLEDs for Invisible Computing,” Proc. SPIE 11302, Light-
Emitting Devices, Materials, and Applications XXIV, Feb. 2020.
[3] N. Couniot et al., “A 65 nm 0.5 V DPS CMOS Image Sensor With 17pJ/Frame.Pixel
and 42 dB Dynamic Range for Ultra-Low-Power SoCs,” IEEE JSSC, vol. 50, no. 10, pp.
2419-2430, Oct. 2015.
[4] I. Cevik et al., “An Ultra-Low Power CMOS Image Sensor with On-Chip Energy
Harvesting and Power Management Capability,” Sensors (Basel), vol. 15, no. 3, pp.
5531-5554, Mar. 2015.
[5] J. Choi et al., “Always-On CMOS Image Sensor for Mobile and Wearable Devices,”
IEEE JSSC, vol. 51, no. 1, pp. 130-140, Jan. 2016.
[6] K. D. Choo et al., “Energy-Efficient Motion-Triggered IoT CMOS Image Sensor With
Capacitor Array-Assisted Charge-Injection SAR ADC,” IEEE JSSC, vol. 54, no. 11, pp.
2921-2931, Nov. 2019.
978-1-7281-9549-0/21/$31.00 ©2021 IEEE
 ISSCC 2021 / February 18, 2021 / 8:38 AM

Figure 34.2.1: Smart contact lens architecture and its principle of operation. Figure 34.2.2: Imager circuit architecture and its readout timing diagram.

Figure 34.2.3: Imager characterization results. Figure 34.2.4: Image-processing architecture and its principle of operation.

34

Figure 34.2.5: On-lens image-processing modes captured on microLED display. Figure 34.2.6: Comparison of state-of-the-art low-power imagers.

DIGEST OF TECHNICAL PAPERS •

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 15,2021 at 06:58:51 UTC from IEEE Xplore. Restrictions apply.
483
 ISSCC 2021 PAPER CONTINUATIONS

Figure 34.2.7: Imager and image processor summary and imager die photo.

• 2021 IEEE International Solid-State Circuits Conference 978-1-7281-9549-0/21/$31.00 ©2021 IEEE

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 15,2021 at 06:58:51 UTC from IEEE Xplore. Restrictions apply.