Journal Pre-proof

Deep learning for scanning electron microscopy: synthetic data for

the nanoparticles detection

A. Yu. Kharin

PII: S0304-3991(20)30276-X
DOI: https://doi.org/10.1016/j.ultramic.2020.113125
Reference: ULTRAM 113125

To appear in: Ultramicroscopy

Received date: 18 June 2020

Revised date: 11 September 2020
Accepted date: 22 September 2020

Please cite this article as: A. Yu. Kharin , Deep learning for scanning electron mi-
croscopy: synthetic data for the nanoparticles detection, Ultramicroscopy (2020), doi:
https://doi.org/10.1016/j.ultramic.2020.113125

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.

© 2020 Elsevier B.V. All rights reserved.

Deep learning for scanning electron microscopy: synthetic data for the
nanoparticles detection
A. Yu. Kharin1
1
National Research Nuclear University “MEPhI”, 115409 Moscow, Russia

Highlights
 Training data for deep learning algorithms can be produced synthetically
 Synthetic data based on real SEM images as textures has been obtained
 RetinaNet Convolutional network trained on synthetic data can be successfully used on
real SEM images

Abstract. Deep learning algorithms are one of most rapid developing fields into the modern
computation technologies. One of the bottlenecks into the implementation of such advaced
algorithms is their requirement for a large amount of manually-labelled data for training. For the
general-purpose tasks, such as general purpose image classification/detection the huge images
datasets are already labelled and collected. For more subject specific tasks (such as electron
microscopy images treatment), no labelled data available. Here I demonstrate that a deep learning
network can be successfully trained for nanoparticles detection using semi-synthetic data. The real
SEM images were used as a textures for rendered nanoparticles at the surface. Training of
RetinaNet architecture using transfer learning can be helpful for the large-scale particle distribution
analysis. Beyond such applications, the presented approach might be applicable to other tasks, such
as image segmentation.

Keywords:
Image processing, scanning electron microscopy, synthetic data, convolutional neural
networks, deep learning, nanoparticles, detection.

1. Introduction
Progress in electron microscopy allows scanning the large areas and collecting the huge amount of
data that often should be processed in order to get the samples characteristics. One of the most
frequently appearing tasks is the calculation of objects and collecting the statistics of such objects`
size in order to get the size distribution. Classical solution of this task includes either manual or
semi-automatic approach, using intensity threshold and further connected components analysis of
pixels. Semi-automatic methods often needs the manual corrections on the background intensity
gradients and may be not applicable in case of brightness change due to elemental composition and
internal structure of particles. Neural network approach can be used for the microscopic images
enhancement [i], and can help solving the nanoparticles detection task, taking in account all possible
image imperfections and noises. The only existing dataset [ii] on synthetic particles do not include
any texture or background and seems insufficiently variable for the ML algorithms training. Other
available electron microscopy datasets [iii] include only TEM images, while the SEM have
significantly other output. Application of large, but poorly-labelled SEM images [iv] as a textures, a
large synthetic dataset can be obtained with large shape/texture/illumination variability [ v].
Using such data, a convolutional neural network can be used for nanoparticles detection task. Here I
use pre-trained RetinaNet architecture and retrain it on the synthetic set.
2 Materials and methods
2.1 Nanoparticles dataset creation
In order to create nanoparticles dataset, a Blender 2.79 software was used for rendering. Blender
Script based on [IV] was created. 9600 of high quality 512x512 pixels images of spherical particles
was created. During rendering, background and particles textures, light sourses position, camera
position, particle distribution and light sources brightness’s were randomized in order to achieve
better variability of dataset. The two-step particles image generation process can be seen on Figure
1

(a)

(b)
Figure 1. Scheme of the synthetic image formation. (a) 1st step is generation of JSON files with
data about each particle in image. (b) textured images are rendered. Textures are randomly formed
from real SEM images database [2]
After generation, the Gaussian blur filter (5x5 pixels) was applied to each image and histogram
normalization procedure was applied.
2.2 Convolutional neural network training
Neural network for the object detection usually consists of two parts: first is a backbone —
pretrained on large-scale well labelled and dataset for image classification task (i.e. on ImageNet
dataset [vi]). Backbone layer turns the image into the set of feature maps of lower resolution.
Usually used backbone net is ResNet [vii], which mainly consist of 3x3 image convolutions with
additional residual connections, allowing more effective training in case of large number of layers.
Here I used RetinaNet with ResNet-50 pre-trained on Imagenet as a backbone. RetinaNet is an
image detection network consisting on ResNet50 with Feature Pyramid Network (FPN) [ viii] to
catch the significant large-scale features as well as local ones and 2 separate networks on top of
FPN — one is used for class prediction, other for the bounding box prediction. Generally, during
training, neural network yield the result on training data during forward propagation, and loss
function is calculated. Further, the neural network parameters are changing at backpropagation
proportionally to the derivative of loss on those parameters. In case of RetinaNet, Focal loss [ ix] was
used for effective training. Neural network was trained on augmented images (each of images and
corresponding bounding boxes are randomly sheared, rotated and/or flipped before each passing to
neural network) on Kaggle cloud GPU (NVidia K80) for about 50h (48 epochs, batch size 16,
learning rate =0,001, Adam optimizer[ x]). Training process scheme is shown on Figure 2. Details
can be found at [xi].

Figure 2. Scheme of ResNet architecture used for nanoparticles’ detection

2.3 GUI and testing images from SEM
It was found, that neural network trained on such synthetic dataset can be successfully used on real
images (SEM TESCAN MAIA 3) of different particles, produced via laser ablation methods [xii].
Resulting trained network was implemented to GUI for its application and available at [xiii].
Inference pipeline, underlying GUI consists of image blurring, brightness histogram normalization
and optional image grid subdivision with RetinaNet passing to each cell (that may be helpful in case
of large number of NPs presented since maximal particles detection number on single image is
limited to 300 due to NN architecture). In case of tiff images saved by TESCAN software, scale is
picked automatically from *.hdr file. Otherwise it should be provided manually. GUI has 2 sliders
for the sensitivity of NN (changing threshold) and maximal size threshold.
It was found that in case of SEM images of both spherical (Figure 3a, b) and non-spherical particles
(Figure 3c), RetinaNet detector trained on synthetic data dives satisfactory results. The detection of
non-spherical particles (which are not presenting in training dataset) is possible both due to
augmentations and the initial usage of pre-trained for other tasks model.
Figure 3. Examples of algorithm work on real images and corresponding size distributions.
3 Conclusion

I showed that synthetic datasets are promising tool for the neural network training in case of
scanning electron microscopy images analysis. Synthetic images approach, where images are
generated using real SEM microphotographs in combination with transfer learning can be very
powerful approach and can significantly speed up the solution of large-scale analysis problem.
Developed software contains simple GUI that makes it easy to use without any programming
experience. Researchers can easily implement and share the best algorithms with any of the
scripting languages available.

References

i
Jain, V., Murray, J. F., Roth, F., Turaga, S., Zhigulin, V., Briggman, K. L., ... & Seung, H. S. (2007,
October). Supervised learning of image restoration with convolutional networks. In 2007 IEEE 11th
International Conference on Computer Vision (pp. 1-8). IEEE.
https://doi.org/10.1109/ICCV.2007.4408909
ii
DeCost, B. L., & Holm, E. A. (2016). A large dataset of synthetic SEM images of powder
materials and their ground truth 3D structures. Data in brief, 9, 727-731.
https://doi.org/10.1016/j.dib.2016.10.011
iii
Stagg, S. M., Lander, G. C., Pulokas, J., Fellmann, D., Cheng, A., Quispe, J. D., ... & Potter, C. S.
(2006). Automated cryoEM data acquisition and analysis of 284 742 particles of GroEL. Journal of
structural biology, 155(3), 470-481, https://doi.org/10.1016/j.jsb.2006.04.005
iv
Aversa, R., Modarres, M. H., Cozzini, S., Ciancio, R., & Chiusole, A. (2018). The first annotated
set of scanning electron microscopy images for nanoscience. Scientific data, 5, 180172.
https://doi.org/10.1038/sdata.2018.172 (2018).
v
https://www.kaggle.com/alexanderkhar/generated-nps (last viewed 18.06.2020)
vi
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-
scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern
recognition (pp. 248-255). Ieee. https://doi.org/10.1109/CVPR.2009.5206848
viiHe, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
https://doi.org/10.1109/CVPR.2016.90
viii Lin, T. Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature
pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision
and pattern recognition (pp. 2117-2125). https://doi.org/10.1109/CVPR.2017.106
ixLin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object
detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-
2988). https://doi.org/10.1109/ICCV.2017.324
xKingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980.
xiLin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object
detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-
2988). https://doi.org/10.1109/ICCV.2017.324
xiiKabashin, A. V., & Meunier, M. (2003). Synthesis of colloidal nanoparticles during femtosecond
laser ablation of gold in water. Journal of Applied Physics, 94(12), 7941-7943.
https://doi.org/10.1063/1.1626793
xiiihttps://github.com/Alexankharin/SEM_analysis (last viewed 18.06.2020)

Declaration of interests

☒ The authors declare that they have no known competing financial interests or personal
relationships that could have appeared to influence the work reported in this paper.