Information Extraction From Digital Light Microscopy in Life Sciences - 1647203947226001pdji

Information extraction from digital
light microscopy
Luciano Lucas, Constantin Kappel
GTC 2022
From Eye to Insight

Cellular analysis
Big data visualization
Neuron reconstruction Organism analysis
Leica Microsystems | October 2021

© Leica Microsystems GmbH. Registered Office: Wetzlar. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
33
Tackling challenges in microscopy and image analysis
Resolution
Blur
Noise and Motion Artifacts
Time-consuming image analysis with unstable

results
Feedback microscopy
Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

© Leica Microsystems GmbH. Registered Office: Wetzlar. All rights reserved, also regarding any disposal, exploitation, reprod uction, editing, distribution, as well as in the event of applications for industrial property rights.
Convolution of 2 objects with PSF 4
Convolution
 =
Object Optics Image

Reconstruction by deconvolution 5
Deconvolution
/ =
Image Optics Restored Image

Accessing Super-Resolution → Ultra-Fast
6
Confocal Data Lightning
Basal bodies
Paramecium
Image size:
▪ Fast resonant scan (5 frames/s) 1048×1048×18
GattaQuant Nanorulers [120nm]
▪ Extremely low photon count Voxel size:
▪ MAX projection
Leica Microsystems | Jürgen Reymann 0.035×0.035×0.119 µm
Accessing long time live cell Information in multicolor
7
Zebrafish posterior lateral

line primordium
• 1024 x 512
• xyz: 436 x 73 x 15 µm
• Duration: 8h 28min
Courtesy of Jonas Hartmann, Gilmour Group,
EMBL Heidelberg, Germany
Leica Microsystems | Jürgen Reymann

Seeing more in oncology model 8
7.5 h time-lapse recording of mammary epithelial micro spheroid cultured in 3D. Data courtesy of intelligent imaging group (B. Eismann/C. Conrad) at BioQuant/DKFZ Heidelberg

9
Tackling challenges in microscopy 9
Resolution
Blur

results
Feedback microscopy

Blur in fluorescence microscopy 12
Sample
Objective
FL
Image

Estimating out-of-focus blur 13

14
Computational clearing
feature length
scale >
Out-of-focus PSF
feature length
scale <
In-focus PSF

Thunder computational clearing to remove background blur 15
Based on minimization of
: position in pixel
: input data (image)
: estimated background
Desired signal
Background to remove
Under the hood: GPU Compute Service powered by CUDA 11 16
GPU architecture Thunder Lightning

Pascal (sm_60, sm_61) X X
Volta (sm_70) X X
Turing (sm_75) X X
Ampere (sm_8x) X X
Inference:
• Custom-built GPU Compute Service
• Toolchain: VS2019-VS2022/CMake/clang 11.0+ Exact configuration may vary

Introducing the Mica Microhub
Mica
Widefield THUNDER Confocal
-> Accessible for all
-> No constraints
-> Radically simplified workflows
100µm
500µm spheroid acquired with traditional SAME spheroid acquired with Mica Powered by:
widefield system
THUNDER & Patented Mica is an

Confocal FluoSync incubator
Resolution
Blur

results
Feedback microscopy

Dynamic Noise as a confounding signal
|Deal with noise in temporal dimension

21 21
Frame Averaging

AiviaMotion Step 1 – Video Interpolation 22
Video interpolation net

𝐼1
𝐼෪
12
𝐼2
• Architecture used: CAIN

• Choi et al. 2020. “Channel Attention Is All You Need for Video Frame Interpolation.” in AAAI.
• Combines channel attention with pixel shuffling
• Maintains large receiptive field throughout
• Fast computation

AiviaMotion Step 2 – Dynamic Signal Enhancement 23
Recorded
Interpolated by AiviaMotion
time
Streng vertraulich | Dr. Constantin Kappel | Software Engineering, R&D | 30.06.2021

© Leica Microsystems CMS GmbH mit Sitz in Wetzlar. Alle Rechte vorbehalten, auch bzgl. jeder Verfügung, Verwertung, Reproduktion, Bearbeitung, Weitergabe sowie für den Fall von Schutzrechtsanmeldungen.
Application 1: Fast sub-cellular dynamics
Application
Study sub-cellular dynamics.
Sample
Cell culture stained with MitoView Green,
SiR-lysosome and NIR750.
Imaging
Fast imaging (6.4 volumes per second) on
STELLARIS using the resonant scanner.
Image series further enhanced by
Dynamic Signal Enhancement (with
AiviaMotion) and LIGHTNING.
Insights
10 µm Fast, gentle imaging to capture sub-
cellular dynamics at high 3D resolution,
Raw DSE (with AiviaMotion) e.g., mitochondrial transport and lysosome
+ LIGHTNING maturation.

Under the hood: GPU Compute Service powered by CUDA 11 26
GPU architecture AIVIA Motion

Pascal (sm_60, sm_61) X
Volta (sm_70) X
Turing (sm_75) X
Ampere (sm_8x) X
Model Training:
• PyTorch 1.7
• Python 3.9
Inference:
• Custom-built GPU Compute Service
• Toolchain: VS2019-VS2022/CMake/clang 11.0+
Exact configuration may vary

7
Resolution
Blur

results
Feedback microscopy

Resolution
Blur

results
Feedback microscopy

33

34

35
Annually, the neurosciences

community spends
304
human work years on
tracing neurons
387 publications per year (only counting the top two commercial software tools for neuron tracing) x 2
(not all projects get published, for each published work, there are at least one other does not get
published) x 100 (workdays a researcher uses per year when they have an active project that needs
© Leica Microsystems GmbH. Registered neuron
Office: Wetzlar. Alltracing = regarding
77.5anyKdisposal,
workdays peruction,year = 303.8 as well aswork
in the eventyears)
Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24 2022
th
rights reserved, also exploitation, reprod editing, distribution, of applications for industrial property rights.
What is the scale of this tremendous challenge? 38
Note: assuming an 8 bit image (0 to 255 grey levels)
Min intensity
threshold
Dendrite Min branch
Partition diameter 3 length Intensity Mean intensity
threshold offset
Soma intensity (0 to 50 µm) (0 to 1000 µm)
Soma Diameter offset
(0 to 1000 µm)
Dendrite diameter
Min branch length Intensity Mean intensity
No Partition 3
(0 to 1000 µm) threshold offset
(0 to 50 µm)
Workflow with partition:

1000 x 255 x 255 x 50 x 1000 x 255 x 255 = 211,412,531,250,000,000 (211 quadrillion) of possible parameter combinations
Workflow without partition:

1000 x 50 x 1000 x 255 x 255 = 3,251,250,000,000 (3.2 trillion) of possible parameter combinations

39
10 thousand visible stars in the night sky
This Photo by Unknown Author is licensed under CC BY-NC

40
86 billion neurons in the human brain

41
37.2 trillion cells make up the human body

42
There are between 3.2 trillion to 211 quadrillion

possible parameter combinations.
Learn which ones are most likely to result in a

correctly analyzed neuron is the task!
1,000 trillion (1 quadrillion) synapses in the human brain

Aivia’s AI could learn from experts 44
Prediction
of analysis
parameters
This Photo by Unknown Author is licensed under CC BY-SA

AI would then predict best analysis parameters 45
In app
Prediction
usage data +
of analysis
image
parameters
fingerprint
This Photo by Unknown Author is licensed under CC BY-SA

Aivia That Learns (ATL) - training image data 46

47
Aivia That Learns (ATL) - training strategy
47
Raw Multi-head
GT
Predictions
model to
Telemetry Multi-head
predict
curation
Feature Map
model
Multi-head to
Data Multi-
model
parameters
predict
to
VGG16 parameters
head model
predictto
predict
parameters
multiple
parameters
Simonyan & Zisserman,

ICLR, 2015
What is predicted?
Concatenate
Sample critical Smoothing Filter Size
every 4 tiles into
region by intensity, Min Edge Intensity
one montage
variance Background removal
3D Image Fill Holes Size
2D Tiles
Dataset Object Radius
Mesh Smoothing Factor
Min Edge to Center Distance

ATL concept in brief 48
Gather raw
data from Deploy
individual models to all
users users
Telemetry,
meta-data, Global AI
image models
data
Train AI models

Human time needed from image to results 50
Processing
Human expert
4.4 mins
12.57 Minutes
ATL+
3.3 Minutes
ATL
0.2 Minutes
51
Annually, the neurosciences

community spends
304
80
human work years on
tracing neurons

Aivia that learns 56

7
Resolution
Blur

results
Feedback microscopy

Resolution
Blur

results
Feedback microscopy

Towards autonomous microscopy 60
Predict Predict
image
acquisition AI image
analysis
parameters parameters

Towards autonomous microscopy 61
Optimize acquisition parameters

Predict if
Acquire image is
“good
image enough” for
AI / analysis
analyze
Optimize
acquisition
parameters Detect
event of
interest

62
Final words and outlook 62
1 Improved spatial resolution
2 Improved temporal resolution
3 Automated image analysis
4 Moving towards smart microscopy

63

64

65

Thank You!
From Eye to Insight

Information Extraction From Digital Light Microscopy in Life Sciences - 1647203947226001pdji

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Information Extraction From Digital Light Microscopy in Life Sciences - 1647203947226001pdji

Uploaded by

Copyright:

Available Formats

Information extraction from digital

From Eye to Insight

Big data visualization

Neuron reconstruction Organism analysis

Leica Microsystems | October 2021

Noise and Motion Artifacts

Time-consuming image analysis with unstable

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Object Optics Image

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Image Optics Restored Image

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Confocal Data Lightning

Zebrafish posterior lateral

Leica Microsystems | Jürgen Reymann

Leica Microsystems | Jürgen Reymann

Tackling challenges in microscopy 9

Noise and Motion Artifacts

Time-consuming image analysis with unstable

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Leica Microsystems | Jürgen Reymann

Leica Microsystems | Jürgen Reymann

: input data (image)

GPU architecture Thunder Lightning

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

THUNDER & Patented Mica is an

Noise and Motion Artifacts

Time-consuming image analysis with unstable

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

|Deal with noise in temporal dimension

Leica Microsystems | October 2021

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Video interpolation net

• Architecture used: CAIN

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Streng vertraulich | Dr. Constantin Kappel | Software Engineering, R&D | 30.06.2021

Leica Microsystems | October 2021

GPU architecture AIVIA Motion

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Tackling challenges in microscopy 27

Noise and Motion Artifacts

Time-consuming image analysis with unstable

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Noise and Motion Artifacts

Time-consuming image analysis with unstable

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

Annually, the neurosciences

Note: assuming an 8 bit image (0 to 255 grey levels)

Workflow with partition:

Workflow without partition:

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

10 thousand visible stars in the night sky

This Photo by Unknown Author is licensed under CC BY-NC

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

86 billion neurons in the human brain

37.2 trillion cells make up the human body

Leica Microsystems | Luciano Lucas, Constantin Kappel| March 24th 2022

There are between 3.2 trillion to 211 quadrillion

Learn which ones are most likely to result in a

1,000 trillion (1 quadrillion) synapses in the human brain