See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/342515660

Towards Unaligned Guided Thermal Super-

Resolution

Presentation · June 2020

DOI: 10.13140/RG.2.2.33012.17289

CITATIONS READS

0 164

2 authors:

Honey Gupta Kaushik Mitra

Indian Institute of Technology Madras Indian Institute of Technology Madras
10 PUBLICATIONS   89 CITATIONS    86 PUBLICATIONS   937 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Solving inverse problems in computational imaging using deep learning View project

Fourier Ptychography, Computational imaging using deep learning View project

All content following this page was uploaded by Honey Gupta on 28 June 2020.

The user has requested enhancement of the downloaded file.

Unaligned Guided Thermal
Super-Resolution
Honey Gupta and Kaushik Mitra
Computational Imaging Lab
IIT Madras, India
 Thermography
Detects the infrared energy emitted from an object, converts it to temperature,
capture the temperature distribution.

Equipment monitoring All-day vision Health-care

Image courtesy: https://www.nttinc.com Image courtesy: https://youtu.be/aJGCAOTCXxw Image courtesy: https://www.faim.org

 Thermal cameras

FLIR AX8 FLIR E75

● Resolution: 60 x 80 ● Resolution: 180 x 240
● Cost: $999.99 ● Cost: $6999.99
Information courtesy: FLIR Systems (https://www.flir.com)
 Motivation

Visual-range ● Most of the thermal cameras have

RGB camera
visible-range cameras
Thermal camera
● Visible-range cameras have high
resolution

Therefore, Guided Super-resolution (GSR)

is a feasible solution which allows use
lower resolution thermal cameras
FLIR AX8
● Thermal res.: 60 x 80
● Visible-range res.: 640 x 480
Courtesy: https://www.flir.com
 Motivation

Visual-range
RGB camera
● Existing GSR methods assume pixel-to-pixel
alignment
Thermal camera

● Baseline between the cameras results in

misalignment due to disparity

● Wavelength difference makes is very

challenging to align the images
 Proposed method
Unaligned Guided Thermal Super-resolution (UGSR)

● Implicit alignment methods for guided super-resolution from unaligned

thermal and visible images

○ method for alignment in the feature-space

○ method for alignment in the input space

● Novel CNN model with dense-blocks and self-attention modules for GSR
Proposed network architecture
(without alignment correction)

UGSR-Base
Implicit alignment in feature-space

Misalignment in
feature-space
due to disparity

UGSR-Base
 Implicit alignment in feature-space

UGSR-FA
High resolution
guidance image
 Alignment in feature-space

● Minimise cosine-distance between the features

● Maximise correlation between the features

 Implicit alignment in input-space

UGSR-ME
High resolution
guidance image
 Implicit alignment in input-space

UGSR-ME
High resolution
guidance image
 Alignment in input-space
● We propose a misalignment estimation block
○ Classification approach
○ Aim: optimal translation-map estimation
● End-to-end optimisation

Misalignment-estimation block
Results
 Comparison
(CATS dataset)

Disparity between the input Interpolated input thermal

Input visible image
images image

MSF-STI-SR [6] 0.89/0.167) Deep-ISTA [5] (0.91/0.197) UGSR-ME(Ours)(0.91/0.156) UGSR-FA(Ours)(0.97/0.075) Ground-truth

 Conclusion

● To the best of our knowledge, ours is the first attempt towards unaligned GSR

● We proposed two methods to tackle the misalignment issue by

○ aligning in the feature-space

○ rectifying the misalignment in the input-space

● Among our two strategies, we found that aligning in the feature space is better
than aligning in the input-space.
Please post your questions on the Q&A feed at the CCD workshop page of the
CVPR website.

For updates, please follow: https://github.com/honeygupta/UGSR

Thank you!
References
[1] Hirschmuller, H., 2005, June. Accurate and efficient stereo processing by semi-global matching and mutual information. In 2005
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) (Vol. 2, pp. 807-814). IEEE.
[2] Cech, J. and Sara, R., 2007, June. Efficient sampling of disparity space for fast and accurate matching. In 2007 IEEE Conference
on Computer Vision and Pattern Recognition (pp. 1-8). IEEE.
[3] Yamaguchi, K., McAllester, D. and Urtasun, R., 2014, September. Efficient joint segmentation, occlusion labeling, stereo and flow
estimation. In European Conference on Computer Vision (pp. 756-771). Springer, Cham.
[4] Geiger, A., Lenz, P. and Urtasun, R., 2012, June. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012
IEEE Conference on Computer Vision and Pattern Recognition (pp. 3354-3361). IEEE.
[5] Deng, X. and Dragotti, P.L., 2019. Deep Coupled ISTA Network for Multi-Modal Image Super-Resolution. IEEE Transactions on
Image Processing, 29, pp.1683-1698.
[6] Almasri, F. and Debeir, O., 2018, December. Multimodal sensor fusion in single thermal image super-resolution. In Asian
Conference on Computer Vision (pp. 418-433). Springer, Cham.

View publication stats