The magazine of the algorithm community

A publication by

July 2019
Women in Computer Vision: Autonomous Driving Applications:
Jelena Frtunikj - BMW • MSC Software
• AutoX

Research: Deep Image Reconstruction

from Human Brain Activity

Upcoming Events
Project Management:
Time Estimation for Your Projects
Exclusive Interview:
Andrew Fitzgibbon - Microsoft
Computer Vision Project:
Nuclei Detection and Segmentation
Computer Vision Tools:
Object Tracking in Python Using OpenCV - with codes!
 2 Contents Computer Vision News

04 14 32

Women in Computer Vision

Jelena Frtunikj - BMW
Project by RSIP Vision
Nuclei Detection and Segm. BEST OF CVPR 2019
38
06 20

Tool:
Object Tracking with OpenCV
Project Management Tip
Projects Time Estimation Guest
Andrew Fitzgibbon 42
08 24

Research Paper Review Autonomous Driving Autonomous Driving

Deep Image Reconstruction MSC Software AutoX

03 Editorial by Ralph Anzarouth 14 Best of CVPR 2019

Our articles from Long Beach, CA
04 Nuclei Detection and Segmentation 38 Computer Vision Tools
Project by RSIP Vision
Object Tracking in Python Using OpenCV
by Dorin Yael by Amnon Geifman
06 Project Management Tip
Time Estimation for Your Projects 42 Autonomous Driving App
AutoX with Jianxiong Xiao
08 Research Paper Review
Deep Image Reconstruction from…
47 Computer Vision Events
Upcoming Events July-Sept 2019
by Amnon Geifman
Computer Vision News
Welcome 3

Dear reader,
CVPR 2019 is just behind us, and we can
almost still hear the voices from CVPR that we
heard in Long Beach. It was a huge conference
with excellent scientific content, as confirmed
by those who were there and who read about
it in our CVPR Daily. For everyone, this
magazine includes a 24-page BEST OF CVPR
section, starting on page 14: be sure not miss
it!
This issue of Computer Vision News includes
Did you subscribe to several articles dedicated to the autonomous
Computer Vision News? driving area, one of our most popular subjects.
It’s free, click here! You'll find some in the CVPR section and one
more on page 42, where AutoX CEO and
founder Jianxiong Xiao (a.k.a. Professor X)
Computer Vision News
kindly shares with us how computer vision
Editor: helps them build a full AI platform for
Ralph Anzarouth applications such as logistics delivery services,
Engineering Editor: airport transportation and city driving in
Amnon Geifman smart, 5G-connected cities. You will find more
Publisher: articles related to autonomous driving on
RSIP Vision pages 24 and 32.
Contact us Finally, here you'll find the full recording (with
all the slides) of the webinar hosted just a few
Give us feedback
days ago by Moshe Safran and Miki Haimovich:
Free subscription “Project Management for AI Implementation
Read previous magazines in Medical Devices”. Stay tuned and don’t miss
Copyright: RSIP Vision our upcoming webinars on medical imaging
All rights reserved and more.
Unauthorized reproduction Enjoy reading!
is strictly forbidden.

Follow us: Ralph Anzarouth

Editor, Computer Vision News
Marketing Manager, RSIP Vision
 4 Nuclei Detection and Segmentation
Computer Vision News

by Dorin Yael

“… a smart solution was needed!”

Every month, Computer Vision News reviews a successful
project. Our main purpose is to show how diverse image
processing techniques contribute to solving technical challenges
and real world constraints. This month we review challenges
and solutions in a Medical Imaging project by RSIP Vision:
Challenges in Nuclei Detection and Segmentation.
Many diagnostic and research annotation for this project: this was a
processes require an accurate difficult task due to the large amounts
Project

evaluation of the types, shapes and
numbers of cells in a specific tissue
area. Thus, our goal in this project was
to detect and segment single cell
nuclei in slices of tissue. To achieve this
goal, we had to overcome multiple
challenges: the very high variability
between the properties of nuclei
belonging to different cell types
including shapes, sizes, color intensities
and textures which vary within and
between tissues. In addition,
sometimes cells are clustered and need
an expert eye to separate them into
single cells with definite borders.
Another difficulty is due to occlusion,
when one cell is partially hidden by
another cell.
These make it very challenging to build
a single process which detects them all
in every single slide. There is no single set
of parameters which covers them all.
Therefore, a smart solution was
needed. We separated the project into
two steps: the first step focused on
detecting the cell, while the second
step focused on the segmentation of
cells detected during the first step. To
do this, we used deep learning
techniques which specifically adapted
U-Net with different ground truths.
RSIP Vision also performed the
and high density of nuclei in every
single slice. Observations are very
subjective as well: specifically in
clusters of cells, inter- and intra-
observer variability is particularly high.
Of course, the accuracy of the model is
dependent on the accuracy, number
and richness of the annotations. For
this project we annotated a few
thousands of nuclei using different
approaches.
The first step of processing in our
solution was cell detection. Once the
nuclei are detected, the cell
segmentation task started, with the
goal of segmenting each cell separately
(instance segmentation). Once this is
done automatically, accurately and fast,
the output of this task can be used
both for diagnostic and for research
purposes. The standardization that we
were able to apply with our solution
solves the problem of variability in the
observations. The fact that this is done
automatically reduces the human time
needed and enables to answer more
questions. If the physician needs to
make a diagnostic based on cell
detection, classification or count, they
will be able to do it faster and better,
since it will be independent from
human individual observations.
 A project by RSIP Vision
Computer Vision News
Another complexity of this task
emerges from the multitude of
techniques for tissue staining. One of
the most advanced techniques used in
the study of cancer microenvironment
is multiplex, which allows dying the
same tissue with different specific
fluorescents. The stained slice is
photographed multiple times using
different wavelengths and the different
images are overlaid to produce a
merged picture. This allows a
classification of different cell types
within the same slide, while conserving
its spatial structure. Using automatic
analysis of these images, opens new
opportunities to shed more light on
multiple open research questions.
The main aspect of our solution was
the decision to separate the project in
two parts, enabling us to be specifically
precise in each one of them. The other
valuable choice that we made was the
5
use of non- classic computer vision
algorithms and techniques, in
particular the use of deep learning. In
contrast to classic algorithms, which
require a detailed definition of
conditions and heuristics which are
specific to tissue, staining technique
and even the microscope resolution,
deep learning is independent with all
of these and can fit different data
types. Thanks to deep learning, when
you train your model on many
examples it learns by itself to identify
and classify the different cells
Project
emerging from different tissues and
dyed by different staining techniques.
Read about more projects in medical
segmentation
Take us along for your next
Deep Learning project!
Request a call here
 6 Project Management Tip Computer Vision News

Time Estimation for Your Projects

RSIP Vision’s CEO Ron Soferman has launched a
series of lectures to provide a robust yet simple
overview of how to ensure that computer vision
projects respect goals, budget and deadlines.
This month he tells us about another aspect of
Project Management : Time Estimation for Your
Projects. Read more tips by Ron Soferman
about Project Management in Computer Vision.
Management

“…you will certainly obtain a reliable

time estimation, which you can present
to the other stakeholders in the organization!”
As a project manager, you frequently
have to estimate the development
time needed to develop a specific
solution, whether you initiate the idea
or you answer a specific need of the
organization to solve a problem.
Time estimation is a very tricky
task, since the research part is
always a big unknown: you don’t
know the solution yet and as a
result, you can’t know for sure
how long everything will take.
Still the organization expects from
you, the project manager, to
provide a reliable estimation of
the investment that is needed to
bring about the specific solution
that is requested.
A reliable time estimate is an
essential information for the
organization in order to decide
about a specific project. It starts
from your management, that
needs to decide whether to
launch this project. Other
stakeholders in the organization
might need to know and
understand the timetable in view
of entering a new market, train
the sales team on a new feature
and preparing the human
resources aspect of the project.
The first step will be to consult
with your team and get their input
about the length of time that will
be needed to solve the specific
problems. There might be several
unknown as to the success of
specific approach – so the answer
might be a range rather than one
definitive time length.
However, consulting with the
research team has a drawback:
often, their scale of time is biased
by their belief in their own ability
to solve the task. As soon as they
find the solution, their
assessment of the time will be
shorter. You will be well advised
to look back at previous projects
performed by this team: browsing
into the past logs of the
development team you will get a
broader insight into the different
stages that you anticipate from a
complete solutions.
Computer Vision News
Project Management Tip 7

The next step will be to inspect the
different threats that might appear
during the project and try to mitigate
the risks with specific solutions.
After this first estimation, you must
review and analyze this work plan in
terms of the project and product life
cycle for your company. It may start
with a first prototype, but then you
productization, which includes all the
testing and the fulfillment of
regulatory compliance requirements
(like FDA and others, when applicable).
Taking all those aspects into
consideration, my professional
experience says that you will certainly
obtain a reliable time estimation,
which you can present to the other

have to deliver a full working software,
including a GUI that will be checked
and approved by the marketing team
and also to take into account the
Management
stakeholders in the organization.
More articles on Project Management

“You will be well advised to look back

at previous projects performed by this team…
 8 Research Computer Vision News

by Amnon Geifman
Every month, Computer Vision News reviews a
research paper from our field. This month we
have chosen Deep Image Reconstruction from
Human Brain Activity. The authors (Guohua
Shen, Tomoyasu Horikawa, Kei Majima and
Yukiyasu Kamitani) published their article here.

…when will it be possible to read minds?

One of the most interesting questions today is: when it will be possible to read
minds? The recent success of deep neural nets enables to perform new and
fascinating tasks. One example for such task is the reconstruction of images from
brain signals. Today, many vision groups and brain science groups are devoting
their efforts to develop DNN based methods to interpret fMRI scans and
generate images using brain activity measurements.
The impact of succeeding in such task is huge, as it might help to understand the
brain structure, assist the blind, and more. In this article, we will describe one of
the best methods to perform this a task, review the method and show some of
its results.
Research

The frame-work of the task is simple: subjects are inserted into an fMRI
machine, then, a sequence of images is shown to them and the fMRI signal is
recorded. The images are divided into three classes: natural images, artificial
shapes, and alphabetical letters.
The fMRI data is represented in a 4-dimensional structure: one dimension is for
the time and the other three dimensions are a structural representation of the
activity of the brain, represented in voxels. Such dataset contains a set of
training images and a set of test images. While this paper performs
reconstruction of the images from the brain, also classification of the images
into classes is possible.
The authors use DNN visual features of a given image based on the VGG-19
network. These features are meant to approximate the hierarchical neural
representation of the human visual system. Then, the signal from the fMRI is
translated to fit into these visual features. At test time, given the translated
signal, an optimization is performed to predict the image which its DNN features
are best correlated with the fMRI signal. In this way, a reconstruction of the
image is generated. The figure in the next page demonstrates the main idea of
the method; we now dive into the details of the paper.
 Deep Image Reconstruction from … 9
Computer Vision News Computer Vision News

Method

DNN architecture for visual features representation

Research
In order to represent an image in an informative way, the authors used a pre-
trained classification model of the VGG-19 architecture. This architecture
was trained to classify ImageNet dataset with over 1000 classes of objects
category. A trained classification model (such as VGG-19) gives a
representation of an image in an hierarchical manner, which is invariant to
shifting, rotations and translation of the objects. In turn, this will produce an
invariance of the model to such transformations at inference time.
The VGG-19 model consists of 16 building blocks of convolution and 3 fully
connected layers. The outputs from each layer were taken immediately after
the weight multiplication (without rectification) and concatenated into a
vector. This vector is named the visual feature vector, that later will be used
in the training and reconstruction process.

fMRI decoding
The authors construct a decoding model that predicts the visual feature
vectors from brain signal using sparse linear regression (SLR). The logic is
that the SLR can automatically select the most important features for
prediction. In this way, they also cope with the high dimensionality of the
fMRI signal.
 10 Research Computer Vision News

The fMRI input is a sample 𝑋 = 𝑥1 , 𝑥2 , . . , 𝑥𝑑 where xi is a scalar value

specifying the signal amplitude of the i'th voxel in the brain. Using the
measurements of d voxels, the regression function is represented as:
𝑦 𝑥 = Σ𝑖 𝑤𝑖 𝑥𝑖 + 𝑤0
where wi are learnable weights and w0 is the bias term.
Given an input image, a forward pass to the DNN decoder (described in the
previous section) produces the visual feature vector with entries t1,.. tL where tl
is the l'th entry of this vector. Then, each tl is fitted using the regression function
above. The objective of the regression is to maximize the likelihood of tl given
the voxel amplitudes X, the weights W, and a fixed parameter 𝛽. This likelihood
function has the form of:
0.5
𝛽 1
𝑃 𝑡𝑙 𝑋, 𝑊, 𝛽 = ෑ ex p{ − 𝛽 𝑡𝑙 𝑛 − 𝑦𝑛 𝑥 ൡ
𝑛=1,..,𝑁 2𝜋 2

where N is the number of samples, and 𝑡𝑙 𝑛 is the n sample of the tl entry. In

order to get sparse estimator, the authors used Bayesian parameter estimation
and adopted the automatic relevance determination prior.

Image reconstruction from DNN layer

Research

Remember that, during test time, we don't have the input image nor the visual
feature vector. What we do have is the feature vector generated from the brain
signal, which is supposed to be a good approximation of the visual feature
vector (that was generated from the image). To reconstruct the image, the
authors use optimization in the image space that aims at finding the image, the
DNN features of which are the closest to the fMRI signal feature vector. In order
to do so, the authors formulated the optimization problem of the form:
1 2
𝑣 ∗ = 𝑎𝑟𝑔𝑚𝑖𝑛𝑣 { ෍ ቀΦ𝑙 𝑣 − 𝑦 𝑙 ൯ }
2 𝑖
where v is a vector in which elements are pixel values of an image (of size
224x224x3), v* is the reconstructed image, Φ𝑙 is the l'th visual feature vector
entry, and yl is the corresponding translated signal generated from the fMRI
signal. The goal of this optimization is to find the best image to fit the brain
signal.

DGN constraint
In order to improve the 'naturalness' of the reconstructed images, the authors
use a generative adversarial network (GAN). Using a pre-trained deep generator
network (DGN), denoted by G(z), they modified the optimization objective to be
of the form:
 Deep Image Reconstruction from … 11
Computer Vision News

1 2
𝑧 ∗ = 𝑎𝑟𝑔𝑚𝑖𝑛𝑧 { ෍ ቀΦ𝑙 𝐺(𝑧ሻ − 𝑦 𝑙 ൯ }
2 𝑖
When achieving minimum of the objective z*, the final image is then taken to be
v*=G(z*). Since GAN is trained to produce natural looking images, the result of
these optimization will generate more plausible results and improve the
appearance of the previous optimization.

Results
In the field of deep image reconstruction, the results are the most important
aspect of the model. While there are some quantitative measures for the
results, we will let you examine the visual results produced from the method
described above. We show here results from the paper on three different tasks:
Reconstruction of shapes and letters, reconstruction of natural images and
reconstruction of images from imagination. We start from the more
straightforward task, reconstruction of shape and letters. You can see the results
in the figure below:

Research
… these reconstructions
are relatively clear!

As you can see, these reconstructions are relatively clear: you can see the edges
of the shapes and read the word NEURON from the reconstructed images. These
results are the first of their kind which enable to understand the content of the
images. Note that the images above are coming from a relatively simple
distribution, hence nice results are expected.
The next set of results are the natural images reconstruction. Note that this is
much harder task since the images have much more complex distribution. It is
analogous to a classification network that achieves great results on Mnist and
 12 Research Computer Vision News

less accurate results on ImageNet. However, these results are also interpretable
and sometimes enable to identify the objects in the image. You can appreciate the
results in the following figure:

In the figure above, the number of iteration axes refer to the number of iterations
in the optimization scheme. Although the reconstructions are not entirely
identical to the source images, it can be seen that they capture the main object,
some details of the object and this gives some abstract representation of the image.
Research

The last set of results is the reconstruction from imagination. In this task, the
subjects were instructed to imagine an image that they saw in the experiment.
The reconstruction, in this case, has been done based on the imagined image. This
of course is much more challenging task as the signal to noise ratio is dramatically
increased. In the following figure you can see the results of this experiment:

Conclusion
Image reconstruction from brain signal is a fascinating task that several teams
around the globe are working hard to solve. The high signal to noise ratio, as well
as the high dimensionality of both the fMRI signal and the image, are the greatest
challenges in this field. The results presented above can be considered very good
results in this field. As you can see, there is a lot of research to do on this task,
many methods to try, which will hopefully bring us to better solutions for the
problem. There is no doubt that the ability to reconstruct images from the brain
will have a tremendous influence on humanity and on science in general.
 BEST OF CVPR 2019 13
Computer Vision News

BEST OF
CVPR 2019
24 PAGES !!!

14 Presentation
AutoAugment: Learning Augmentation Strategies from Data
Ekin Dogus Cubuk is a research
scientist at Google Brain. Before this,
he got his PhD in physics, but used to
apply machine learning to physical
systems. He did the residency at
Google Brain which is for people who
have an interest in deep learning but
are not experts. At Google Brain, he
has been working on AutoML
applications, mainly for data
augmentation. He is a first-timer at
CVPR and speaks to us ahead of his
first oral presentation.
Even as he was doing his PhD, Dogus
says it was clear that machine learning
had many interesting directions. He
says if you look at the research
divisions now, all the physics PhDs are
using their physics skills in machine
learning. They are also interested in
using machine learning to study
physics, and there’s still an active
research area at Google Brain and
Google Accelerated Science.
CVPR
TuesdayDAILY
Dogus finds that data augmentation is
an underutilized tool in deep learning.
Although there are many papers that
come up with new data augmentation
operations – like mixup, Cutout,
geometric operations – it’s not clear
how you would combine them and get
the optimum result. It’s interesting to
ask how far you can push the impact of
data augmentation on these models. In
this paper, they tried to combine the
already existing operations and get as
good a result as possible on the test
set.
“A lot of the work is
focused on architectural
improvements, rather than
processing data!”
During the process, they actually found
out some more fundamental and
important things that should be done
for data augmentation. One of them is
variability. Diversity in your data
augmentation policy, for example.
Most of the time when people apply
operations, they either apply it to
every mini-batch and every image, or
they don’t apply it. Cutout is very
helpful on some data sets, but how
you use it is you apply it to every single
image and every single mini-batch.
They found that instead of having one
strategy, having hundreds of strategies
and choosing one of them randomly
for each image in each mini-batch
actually gives you a huge
improvement. If you were to just do
that and not use any of AutoAugment’s
search capability, you already get a big
improvement.
CVPR
DAILYTuesday
He explains: “Accuracy with deep
learning models is important, and data
augmentation is an easy but impactful
way of improving the accuracy. A lot of
the work is focused on architectural
improvements, rather than processing
data. Since experts already have an idea
of what symmetries are, and it’s clear
how they can utilise those symmetries
to augment the data, it would be good
to increase the accuracy. Something
else we have been seeing recently that
we didn’t necessarily think about when
we were writing the paper is that it
seems that increased data
augmentation helps a lot with
robustness. Recently there was an ICML
workshop here on robustness for
machine learning models to common
corruptions and noise. What several
people found is that AutoAugment
leads to the best robustness. Although
it was actually only trained for
increased validation accuracy.”
15
AutoAugment
In terms of next steps, Dogus says that
they want to apply this idea to other
domains. Not just image classification,
but video and object detection, for
example. They recently had a paper,
SpecAugment, about applying
AutoAugment to speech to benefit
speech recognition, but they realised
that before they do that, there are
other improvements to be had just by
adding some new operations. In
general, they are trying to make the
method faster so that other researchers
can apply it quickly to other domains.
Since their work, several other groups
have used their search space and
augmentation strategy, but with a
different optimizer to reach as good
results, which is encouraging. They had
used a reinforcement learning approach
to optimize their augmentation policy,
which can sometimes be hard to get to
work. Since their work, other people
have reproduced it with Bayesian
optimization or with population-based
training or a few other methods, and
Dogus is encouraged by how easy it is
to get the same results with different
optimizers.
“Accuracy with
deep learning
models is
important, and
data augmentation
is an easy but
impactful way of
improving the
accuracy!”

16 Presentation
AdaGraph: Unifying Predictive and Continuous
Domain Adaptation Through Graphs
Massimiliano Mancini is a PhD student
at Sapienza University of Rome, in
collaboration with Fondazione Bruno
Kessler (FBK) and Istituto Italiano di
Tecnologia (IIT). His advisors are
Barbara Caputo from IIT and
Politecnico of Torino, and Elisa Ricci
from the University of Trento and FBK.
Samuel Rota Bulò from Mapillary is a
third unofficial advisor. Massimiliano
spoke to us ahead of his oral and
poster.
This work tackles predictive domain
adaptation. In standard domain
adaptation, what you have is a source
domain where you have a lot of labels
for the task that you want to address,
then another domain, which is
unlabelled, which is actually the
domain which you want to apply your
model. For instance, you have a lot of
images collected in daylight, so they
are clean and labelled, and then you
want to apply your model at night, but
you have only unlabelled images for
the night situation. You must figure out
how to go from daylight to night
without any label on the night.
You need the target data for doing this
CVPR
Wednesday DAILY
task because if you don’t see anything
you cannot forecast what night will
look like. This work tries to predict
what the night, or a situation, will look
like. If you are given a lot of domains –
one is labelled, a lot are not labelled –
for each of them, you are given an
attribute. For example, you see a front
view of a car, and then you see from
the side, and then someone asks you to
produce a model that will recognise
cars if you see them from the back.
Since you are seeing a lot of different
domains, either labelled or unlabelled,
you may try to figure out what the
model for the rear of the car will look
like. This is something that we do even
as humans. If someone tells you that it
will rain tonight, you might go out with
an umbrella because you adapt
yourself to the weather conditions. If
they don’t tell you it’s going to rain, you
might go out with nothing. As humans,
we adapt to what we know, and if we
have understood how the weather
changes with respect to some
attributes, we can try to adapt to that.
We hope that our algorithms are able
to do that too.
Massimiliano explains: “To solve this
task you must relate what the
parameters of a domain are with
respect to the attribute, or the
metadata, in our case. We have a first
phase where we train parameters
which are attribute or domain specific,
so they are specific for a certain
condition. While doing that, we
initialise what we call a graph. We have
a node for each of the domains and we
connect the nodes of the graph with
CVPR
DAILY Wednesday Massimiliano Mancini 17

the edges, where the weight of the you that it’s going to rain, you go out
edge basically says how much two with an umbrella, but if they don’t tell
domains are related. If I tell you I see a
you that and you go out and it starts
car from a side view and from a rear- raining, then you must figure out how
side view, a rear side and side are to react. Since obviously there will be
closer than a rear and front, so we nothing supervised, and so the
relate in this way. When we get the prediction of the target model can be
target attribute, at that point we wrong of course – and also, some
initialize a virtual node – a fake node –metadata may be received which are
because we didn’t have any data for not representative of the domain – this
that. No parameters are there, but if method unifies this kind of prediction
you assume that similar metadata, so with continuous domain adaptation. As
similar domains in our graph – similar the target data is received, the model
nodes – require similar parameters, we is continuously updated. This is
can just propagate the parameters of possible because the model is based
nearby nodes and obtain our model for on batch normalization. The different
the target domain which we never domains have different batch
see.” normalization statistics and scale and
bias parameters. For each domain,
"This is possible because there are different statistics, and so for
the target you basically predict just the
the model is based on statistics of the domains and the scale
batch normalization" and the bias. Once you have those, at
test time when you receive the target
He adds that there is another problem data, you use it to update the statistics
with that. Like before, if someone tells because it can be done easily. Then
 CVPR
18 Presentation Wednesday DAILY

you devise an unsupervised loss which
is just an entropy loss to update the
scale and the bias for the domain-
specific parameters.
Massimiliano tells us that they were
restricted in terms of the data they had
to test on. They tested on the data set
used in the literature for this problem,
CompCars, with different cars, with a
different viewpoint of cars year of
production. Also, another data set
which depicted different portraits
collected over 100 years in different
regions of America. This is not a real
application of this problem. He says
they would like to have a model, for
example, in autonomous driving. He
explains: “I tell you it’s evening, it
starts getting darker, so it adapts your
model for this kind of light. Obviously,
one can say that if I have a huge
amount of data which are balanced
among all the possible conditions. Our
algorithm does not do it. Nowadays,
the data sets are unbalanced, and we
must specialize our systems.”
Massimiliano says they assume the
metadata is representative of the
domain shift, which cannot be true, so
they must weight them. He says a next
step for this work would be to
understand which metadata are
important for the domain shift and
which are not. He also thinks they
could go from the metadata to a more
abstract representation, because
metadata is good if you have it and if
you can quantify the shift, but if
someone tells you it’s dark or it’s
darker, you can understand what the
environment will look like, so a
description rather than metadata
would be helpful.
Finally, Massimiliano is excited to tell
us that he is very proud of his heritage:
“I come from a very small village in the
middle of Italy in Umbria. I think we
are 30 people in the centre of the
village. It was a long journey to get
here and I’m very happy that at the
end, even starting from there, I can
represent my small village here.”

"Our graph is able to estimate target model

parameters with pretty good results in our
experiments. Receiving the target data and
using them to refine our model, allows to fill the
remaining gap with the upper bound, a
standard domain adaptation algorithm with
target data available beforehand."
CVPR 19
DAILYWednesday Orals

Top: Jiwoon Ahn presenting Weakly Supervised Learning of Instance

Segmentation With Inter-Pixel Relations
Below: Tatiana Tommasi presenting Domain Generalization by Solving
Jigsaw Puzzles. Tatiana explained how jigsaw puzzles help not only kids
but also machines to generalize better!

20 Guest
Andrew Fitzgibbon
Andrew Fitzgibbon leads the All Data
AI theme at Microsoft Research in
Cambridge and has worked for
Microsoft for the past 15 years. He also
teaches at Cambridge University and
has a PhD student there.
Andrew, do you feel more of a teacher,
or a business technologist?
I really enjoy trying to be clear about
things, so teaching is a chance to try to
very clearly explain something that’s
happening. I like teaching Fourier
transforms to undergraduates because
it’s fun to keep one’s hand on the
mathematics. The fantastic thing about
students, especially undergraduates, is
they have no idea what’s difficult and
what’s easy. They just try to understand
and if they don’t understand, they tell
you. They spot bugs in your reasoning,
CVPR
Wednesday DAILY
in your descriptions, very, very quickly.
Were you like this when you were in
their place?
Good question.
That’s what I’m here for!
I notice when something is a bit wrong,
and in fact, it’s almost a way I generate
research ideas. I find things that are
weird or annoying.
Is there anything that your generation
should learn from the current
generation of students?
Yes. Oh, that’s a great question.
Thank you, that’s the second time that
you’ve told me that. I will start to
believe it!
I like that they’re self-confident.
Well, they were trained to be more
self-confident than we were. We were
trained to accept discipline and
authority, while they’re actually used
to challenge it.
Yes. Which is obviously very useful. One
thing that’s good about being old,
because I’m just over 50, is I’ve just got
better at predicting what an algorithm
will do without having to run it.
So, it’s better intuition?
Is it intuition? I don’t know. It’s
definitely to do with experience. You’ve
just seen enough things tried in
different ways. I’ve definitely learned to
try to prove myself wrong.
I think it was Oscar Wilde who said:
experience is the name that people
give to their failures. Maybe the fact
that we did something many times
leads us to know where it will lead to.
That’s right, and easier to predict when
things will fail, or to predict how to
break things, which is just as important.
CVPR
DAILY Wednesday Andrew Fitzgibbon
Can you tell me something about
Microsoft that we, the community,
don’t know?
Something that I’m really proud of at
Microsoft is the way in which we bring
cutting-edge computer vision to real-
world products. We did that with
Kinect. No one knew how to do body
tracking before Kinect came out and it
just got done. With HoloLens, I’m super
proud of something that I didn’t work
on. The head tracking in HoloLens v1
done by my colleagues is just an
amazing piece of engineering. It’s kind
of like you had a moonshot and then
you ended up on Mars.
What is particularly revolutionary
about it?
If you ask George Klein, who was
responsible for large proportions of the
middle of it, he’ll say nothing. Just good
engineering. Just we put it together.
Sometimes in the research world, and I
think young people are maybe better at
this now, people say “It’s just
engineering”, as if just engineering is
not something to be proud of. I think
the distinction between research and
engineering is much closer than we
think. If we think about how deep
networks suddenly became popular, to
a large extent, it’s about the
engineering that Krizhevsky did in order
to make the networks run on the GPU.
In some sense, an engineering advance
has given us this amazing new tool and
capability. I have a title of researcher,
but I really think of myself as an
engineer. My group at Microsoft, and
across Microsoft, more and more we’re
trying to remove the boundaries
between roles that are called
researcher and roles that are called
engineer.
What is the most dramatic boundary
that you’ve been able to break?
21
In head tracking on HoloLens, we all
knew how to do SLAM, we all knew
how to do real-time SLAM, and we all
knew how to do structure from motion,
but no one knew how to do it for hours
and hours and hours. All the demos that
we would do as academics were on
reasonably short sequences, and no
one knew how to do it in incredibly
messy environments. The very messy
environments that we deal with most
often are offices.
Why do you think it’s Microsoft that
came out with this and not another
company? What is special about
Microsoft that made it the company
who will bring it out?
That is a good question.
Thank you. That is the third time. If I
get to 10, I’ll win a prize.
There are a few things. When we were
working on Kinect, we have this
visionary guy, Alex Kipman, and he
phoned up the lab in Cambridge and
said, “I’ve got this brilliant idea. We’re
going to do a machine learning system
which does body tracking and we’re
going to stand in front of the computer
and we’re going to play.” I said, “It’s
lucky you came to me because I am a
world expert in body tracking, and I can
tell you that what you’re saying is
impossible. It’s not going to happen.
You need to do it in 10 per cent of an
Xbox, which is a 2003 era piece of
hardware, and you need to do it real-
time, and it needs to run for hours, and
it’s absolutely impossible.” Kipman said,
“Oh, it’s funny you say that, because
look, I’ve done it.” He produced us a
demo which was doing a fantastically
good job of body tracking if you started
with your arms out in a T-shape. If you
started in the right pose, then it would
do a fantastic job for about 20 seconds
or about a minute.

22 Guest
Which was enough to impress you?
Which was enough to impress us, but
also, we knew that could never
succeed. There’s a compounding
probability of failure because the
system did the thing that all computer
vision engineers love which is, if I knew
what I was doing 30 milliseconds ago,
then surely I can use that information to
learn what I’m doing now. What we did
was we didn’t use that and that was the
crazy thing.
I now have a question for the engineer
that you feel you are. What is the next
boundary that you would love to break?
I would love us to be able to train much
smaller neural networks because I feel
very uncomfortable whenever I see two
floating-point numbers getting
multiplied together. If you make them
16-bit floats, it doesn’t make me much
more comfortable.
Well, that enlargement is today
considered acceptable.
Yes, but someday we’re going to have
to do it with much less computer.
Obviously, today the HoloLens is a
head-mounted device. We want to
think about the number of milliwatts it
CVPR
Wednesday DAILY
takes to compute an answer to some
question. It’s happening. One of my
colleagues, Manik Varma in India, has a
VGG net kind of architecture which is
one kilobyte in size, whereas you think
about neural networks as millions of
parameters.
In the medical world, there are
diseases which are so rare that you
never have enough samples in order to
put them into a large state-of-the-art
neural network.
Absolutely, and that’s one of the
focuses of my research group, All Data
AI. What that means is big data and
small data. You might have your
example of a medical scenario where
you have two training examples of this
important disease, and maybe you have
millions of examples of other situations.
How do you train using tiny amounts of
data? We know the answer to that is
essentially you use Bayesian methods.
Bayesian methods are great when you
lack training data. Sometimes you see a
world where there’s data everywhere.
It’s like with the phrase “Water
everywhere and not a drop to drink”.
You have data everywhere, but only
two training examples. There’s a whole
question there about what we can do
with semi-supervised learning, with
unsupervised learning, and I think our
strength in variational inference almost
looks like it’s being overshadowed by
this deep learning revolution, but it’s
crucially important for real-world
problems.
Now a question for the human behind
the engineer. What boundary would
you like to break in this community in
order to work better together?
Someday we’re going to have to figure
CVPR
DAILY Wednesday Andrew Fitzgibbon
out how to do these conferences
without everybody travelling to the
same place.
Do you mean that human science will
break time and space?
Well, the question is whether we can
live in a VR world or augmented-reality
world, where the experience of
travelling there is as good as the
experience of encountering it at home.
Here’s my vision for how it works with
the time zones because we’re not going
to change the time zones. The idea is
we might run local models of the
conference. Maybe in England we would
have a hotel somewhere and 200 people
would go to that place. Then they would
all switch to California time. Breakfast
would be served at 5pm. People would
attend talks until midnight.
Let’s see it from another point of view:
a significant share of the people that I
love the most in this world are here at
this conference and I meet them once
a year. Now you are telling me, as an
engineer, you would like to find a
solution so that I will never meet them
again! What do you think I will tell
about it?
I tell you this, nothing is stopping you
meeting them. You should meet them
more! You can’t shake their hand. We
don’t have a good solution for that. I
definitely want to be able to sit here
with a glass of beer and you sit there
with a glass of beer, but you’re 2,000
miles away. I was at a workshop
yesterday, Computer Vision for VR and
AR, and a few people – Facebook,
Christian Theobalt, Yaser Sheikh – were
showing us a vision of the future where
the process of me talking to you will
visually be exactly the same as being
together. I’ll have a full 3D view. Perfect
lighting. I think that is a boundary. I
23
don’t know where the good enough
boundary is, but once we go over it and
once it’s good enough, I think it will be
amazing. We will spend more time
together because it will be fun to phone
you up and spend an hour with you
some evening.
Okay, and you think that with this
solution you will be able to convince me?
I don’t know what we’re going to do
about the shaking hands. I wonder if we
will generate some alternative. One of
the things that I was surprised by in the
HoloLens was when we put hand
tracking in the device, I thought it
would be really important to have
haptic feedback when you touch things,
but it turns out that just watching the
virtual finger touch the virtual blob, and
the virtual slider move left and right,
maybe with a little bit of audio,
suddenly feels like haptic. It feels like
I’m touching it.
It’s almost the real thing.
It’s almost the real thing. Clearly
shaking hands, we’re not going to get to
the real thing.
But we’ll get to a proxy which is almost
as good as it.
We might invent a proxy which is we
put our hands in the virtual air and we
kind of like ding them, and if they make
a little noise when the fingers touch,
then maybe that’ll feel fine. I don’t
know. We’d have to try it. I would not
rule it out immediately. I would say that
might feel cool. Ding!
 CVPR
24 Expo: MSC Software Wednesday DAILY

miles between fatal crashes. That

means having to test for billions of
miles, and you cannot do that without
simulation. This product allows you to
mimic real life to make it as realistic as
possible, including simulating traffic.
There are sensor models and driver
models. It can also make a distribution
of scenarios so that rare events
happen more frequently.
Edward says his research interest is to
do with quantifying probabilistic
models for the density of different
crashes and adverse events. He’s
specifically interested in people who
want to look at the validation of the
car more so than the algorithms for
driving the car, because he would like a
safe vehicle quantitatively analysing
Edward Schwalb is a Lead Scientist at the safety of the vehicle on the road.
MSC Software, who are part of the He would like to use a simulation to
CVPR 2019 Expo. extend the physical testing to achieve
that statistical simulating of billions of
miles.
MSC Software has a product called a
Virtual Test Drive which is a simulation
product that provides simulation of
virtual worlds that match real worlds.
In order to test the software of
advanced driving assistance and self-
driving cars, they need a lot of virtual
miles driven because it is not practical
to test these vehicles in the physical
world.
Edwards tells us that they provide
hardware-in-the-loop, software-in-the-
loop, model-in-the-loop, vehicle-in-
the-loop, and driver-in-the-loop
simulation with different levels of
realism. They can record a realistic
scenario and generate variations of
that scenario.
One of the challenges in the industry is
that people drive about 100 million
CVPR
DAILY Wednesday
He explains how MSC Software
addresses these challenges: “We need
to create a population of scenarios
that represents the operational design
domain of the vehicles. There is safety
standard ISO 26262 and SOTIF that
specifically have requirements. SOTIF,
for example, would say that it
categorises all the scenarios between
known safe, known unsafe, unknown
safe, and unknown unsafe. What we
want to do is increase the number of
known safe scenarios at the expense of
the unknown unsafe scenarios. We
want to do probabilistic interpretation
based on scenario population, based
on the behaviour of the car. As a
scientist, I would like to build AI that
does this.”
He says another challenge is if you
have an AI sensor that is 99 per cent
25
MSC Software
accurate – which is very generous –
and you have to drive a billion miles
and every mile has multiple frames.
For example, 10 frames per second,
100 seconds per mile, that’s 1012
frames on a single camera. With
multiple cameras, you can easily have
1014 or more frames and you want to
have one fatal crash for every 1012
frames. How do you use an algorithm
that has an error rate of 10-2, and
build a system that is reliable to 10-14?
How do you build a system which is 10
billion times safer than the
components?
Edward tells us that MSC Software do
this better than anybody else. They can
generate scenarios and quantify the
probability density that certain adverse
events happen such as crashes or
going over a cliff. He concludes by
saying that it is very important to be
able to go from the physical road, to
the digital twin of the physical road, to
millions of simulations with a
population of scenarios, and to take
the result back to the physical world
and make it credible.
“How do you build
a system which is
10 billion times safer
than the components?”

26 Presentation
Data Augmentation Using Learned Transformations
for One-Shot Medical Image Segmentation
Amy Zhao is a graduate student at
MIT in the final year of her PhD. She
spoke to us ahead of her oral and
poster: Data Augmentation Using
Learned Transformations for One-Shot
Medical Image Segmentation, which
explores how to do smarter and
better data augmentation to help
improve medical image segmentation
when you don’t have a lot of labelled
data.
Amy says that deep learning has been
demonstrated to be so powerful for so
many image tasks, but it’s very hard to
do it on medical data. This work
explores segmentation when you only
have one labelled example. She says
this is a pretty common scenario. It’s
easy to get hundreds of unlabelled
examples if you need to – MRI scans
from patients, for example – but how
do we leverage the information in
these unlabelled scans to help us with
segmenting any scan?
These unlabelled scans have a lot of
information in them. They show you
CVPR
ThursdayDAILY
the anatomical variations in your
population. They’ll also have intensity
variations because they were taken
with different scanners and they’re of
different patients. This method can
learn to mimic all of these variations
and synthesise examples that look like
these unlabelled scans. These
examples can then be used to train a
segmentation CNN to leverage the
power of deep learning.
Amy tells us that there’s a more
common way of doing what they’re
trying to do: “Data augmentation.
Most people who are familiar with
computer vision know how to use it to
some extent. For medical imaging, it’s
common to do data augmentation
using random smooth flow fields.
These are fairly easy to code, but they
don’t always produce the best results
and you need to hand tune the
parameters. The challenge here is that
medical imaging is a pretty complex
space. These brain MRIs have very
complex variations. It’s difficult for
people to write functions that mimic
these variations. We are trying to learn
to do this instead of trying to do it by
hand.”
“These brain MRIs have
very complex variations.
It’s difficult for people to
write functions that
mimic these variations.
We are trying to learn to
do this instead of trying
to do it by hand.”
CVPR
DAILYThursday
To solve this, they are leveraging some
recent work in medical imaging: Adrian
V. Dalca and Guha Balakrishnan’s
VoxelMorph. The idea is that if you
have a method that can compute the
transformation from your labelled scan
to an unlabelled scan, that allows you
to compute a set of transformations.
Once you have the set of
transformations, you apply them back
to the labelled scan and use that to
synthesise more labelled examples.
Co-author Adrian Dalca comments that
"This work shows how leveraging
unlabeled data and machine learning
can lead to tangible and practical
improvements. This is important, since
the next step is to use this algorithm in
practical clinical analyses."
What happens if you have multiple labelled
scans available, or if you apply this approach to
different kinds of MRIs or CTs?
Amy Zhao 27
Co-author Guha Balakrishnan adds
that “The benefit of the method is that
it is interpretable and a simple idea
that can extend to various other data,
both in and out of medical imaging."
Thinking about how to develop this
work, Amy points out that they’ve
demonstrated it on a fairly limited
range of scans so far. They just look at
T1 MRIs and show what happens if you
assume that you have one labelled
scan and a hundred unlabelled scans.
What happens if you have multiple
labelled scans available, or if you apply
this approach to different kinds of
MRIs or CTs? They think it would work,
but don’t know how well, and are
really interested to find out.

28 Presentation
Online High Rank Matrix Completion
Madeleine Udell
Madeleine Udell is Assistant Professor
of Operations Research and
Information Engineering at Cornell
University. She spoke to us ahead of
her oral and poster, which she
presented alongside Postdoctoral
Associate Jicong Fan.
Madeleine tells us that this work starts
out with the problem of finding missing
entries in data vectors. A classic well-
understood approach to filling in
missing entries in data sets is low-rank
matrix completion. That assumes that
all of the vectors lie in a low-
dimensional subspace. If, for example,
the vectors lie in a two-dimensional
subspace, then knowing any two of
their coordinates allows you to figure
out all of the rest once you’ve learned
the subspace. This work looks at the
question, what happens if you want to
fill in missing entries, but your vectors
CVPR
Thursday DAILY
don’t live in a two-dimensional
subspace, they live in a low-
dimensional manifold?
She asks us to imagine a one-
dimensional manifold – just one curve
– and imagine it in a 3D space. If the
curve does anything remotely
interesting, then the span of all the
vectors on the curve is the complete 3D
space. It’s not low rank; it’s full
dimensional. You cannot use normal
low-rank techniques to fill in the
missing data, but there’s a clear
structure. You’re on a one-dimensional
manifold. It’s like a line. It’s very easy to
see this curve. The question is, how can
we see curves like this in higher-
dimensional space?
Madeleine explains: “The way that we
solve it is we take these vectors and we
blow them up. We map them through a
polynomial feature map. We take each
of the points and we compute many,
many polynomials in the coordinates
that we have observed. Now our data
lives in a much higher-dimensional
space, and in that much higher-
dimensional space, it is low
dimensional. So, it actually lives on a
low-dimensional subspace in this
higher-dimensional space. The
dimension of that subspace is probably
larger than the ambient dimension of
the original space, but it’s still low-
dimensional in the ambient dimension
of this blown-up space. The most
important trick is the fact that after
blowing up using this feature map, the
resulting set of points is a low-
dimensional subspace so that we can
use ideas from low-rank matrix
completion.”
CVPR Online High Rank Matrix Completion
DAILY Thursday
She points out that there was some
really nice work done by Laura
Balzano, Rebecca Willett, Rob Nowak
and Greg Ongie. They essentially had
the same idea, but it’s too slow, and
you can’t scale it to the kinds of sizes
of data sets that are interesting in
computer vision. To improve on that,
this method adds a couple of critical
components. One component that’s
really important is that they don’t
represent these vectors in this high-
dimensional space explicitly. Instead
they use a kernel map, because writing
them down in this high-dimensional
space would be too large. Then once
they’ve kernelized, they’re able to
rewrite the objective and rewrite all
the penalties in a way so that it
decomposes over the data points, and
it allows them to train it online.
“… this method adds a
couple of critical
components"
One of the most interesting things
about the online training is it means
that they can adapt to a changing
manifold. Imagine you’re watching
29
someone run, or they’re running and
they transition to walking and then
they transition to jumping jacks. Their
motion is quite complex. You might
imagine that all of human motion lives
in some low-dimensional manifold, in
the space of all possible coordinates of
every point in the body, but it’s a much
lower-dimensional manifold if you just
look at jumping jacks or if you just look
at running. By learning online, they’re
able to learn this transition and figure
out, for example, if they started
jumping jacks, they learn the jumping
jacks manifold, or if they’ve started
running, they learn the running
manifold. Because of this they can
learn much lower-dimensional
manifolds and learn it much faster and
get better accuracy than offline
methods. Normally online methods
don’t perform as well as offline
methods because they’re using the
information less efficiently, but for
them, the online method performs
better because they can adapt to quick
changes in the kind of motion.
How can this be applied practically in
the real world? Madeleine explains:
“Some of the things that we’ve
considered are using it to adapt to

"What this method gives you is a completion of

the points in the original space, but it can’t tell
you, for example, how close together two points
are in the natural coordinates of the manifold."
 CVPR
30 Presentation Thursday DAILY
missing sensor measurements. Like in Also, when thinking about a low-
this case of motion detection. I actually dimensional manifold, a natural thing
think it might be useful even for to think about is the internal
motion tracking – I’m wearing a Fitbit coordinates within that manifold. This
right now. We’ve also thought about method doesn’t give you those. What
cases where sensor measurements are this method gives you is a completion
corrupted, possibly in chemical system of the points in the original space, but
monitoring. It’s not clear, but I think it can’t tell you, for example, how close
that maybe the broader picture is that together two points are in the natural
any sufficiently complex phenomenon coordinates of the manifold. For
is probably not exactly a low- Madeleine, that would be extremely
dimensional subspace. That depends valuable to know, so it would be her
on the coordinates in which you’ve next target for this work.
measured it matching something
important about the structure of the
phenomenon. It’s not clear why that
should be true for sufficiently complex
phenomena, especially as you collect
more and more and more data, you
should be able to see that it’s not
actually a low-dimensional subspace.
It’s not flat. Our method allows you to
adapt. As you get more and more data,
then you’re allowed to start looking for
this non-linear structure.”
Thinking about next steps for this
work, Madeleine says she is interested
in the deep structure. They know that
there are ways that manifolds can
wrap back on each other that would
confuse this method in high
dimensions, but one thing they don’t
yet understand very well is when this
mapping succeeds and when it fails. Jicong Fan
CVPR
DAILYThursday Oral Presentation 31

Top: Aliaksandr Siarohin presenting his paper Animating Arbitrary

Objects via Deep Motion Transfer
Below: Supplementary video for this paper. Click to watch!
 CVPR
32 Women in Comp. Vision Tuesday DAILY
Jelena Frtunikj is an expert in
automated driving. Currently, she
works at BMW in Munich, Germany
as a researcher and engineer
developing safe (ISO 26262 & SOTIF)
End2End features of highly and fully
automated vehicles by using sensor
fusion and machine learning
algorithms. At CVPR 2019 she has
organized the workshop Dependable
Deep Detectors: Verification and
Certification of Neural Networks.
Read more interviews with
women scientists
Thanks a lot for the invitation. I was
very surprised!
Why were you surprised?
I don’t know how someone could
discover me for CVPR Daily. Maybe you
can tell me how! [laughs].
We want to learn about you. So tell us
about your work.
I work at BMW working on topics of
autonomous driving. I work on a topic
where we combine machine learning
and safety, which means I work on an
algorithm which does perception. I
bring my safety knowledge because
that’s the background that I have and
that I got during my PhD, in order to
understand how to make deep neural
networks safe.
It means that if autonomous cars do
not crash into another in the future,
we will owe it to you.
Exactly…[laughs] actually that’s my
goal. I wanted to stronger tackle the
topic of safety. It’s not that we are just
developing a prototype. We are
developing something that will go to
the market, and we have to make sure
that it’s safe. Since I have a background
also in computer science, it was not
difficult for me to do. I have learned
about neural networks and so on
before.
So you were the right person, at the
right moment, at the right place. How
long have you been with BMW?
On the first of July, it will be exactly
three years.
Congratulations!
Thank you.
Tell us something about BMW that we
don’t know.
Many people think that BMW is a very
traditional company. Many people
think that only Germans work there,
and it’s not so international. Actually,
that’s completely not the case for the
autonomous driving department. We
are so international. In my team, we
are ten people. I’m actually not
German, I am from Macedonia. We
have an Iranian. We have a guy from
India and another one from China. We
have a Spanish guy. Of course, we have
CVPR
DAILY Tuesday
some Germans. A lot of people think
there are many older people because
it’s very traditional. Especially in
autonomous driving, people who have
the knowledge are quite young. We
have a very young population.
The last thing that we heard about
BMW was the ad along the retirement
of the Mercedes boss. Did you see it?
When he took the i8? Yeah, of course I
saw it.
What did you think about it?
I liked it a lot. As far as I know, to show
mutual respect between both
companies, they do these kinds of
advertisements often. When BMW was
also celebrating 100 years a few years
back, they also did some kind of
advertisement.
So there is some kind of dynamic
discussion going on.
Exactly, it’s a public discussion.
I’m very proud of interviewing
someone from Macedonia for the first
time. Tell us something about
Macedonia that we don’t know.
Macedonia is a very small country.
Many people tell me that they often
confuse it with Barcelona or Estonia. I
don’t know why. Maybe it’s how I
pronounce it. Yes, there is a country
called Macedonia. It’s a very, very small
country north of Greece and south of
Serbia. East is Bulgaria, and west is
Albania… less than 2 million people. It’s
a beautiful country nature-wise.
Are you from Skopje?
Yes!
You are a city person.
Yeah, exactly. I moved to Germany for
my studies and chose Munich to do
Jelena Frtunikj 33
that.
And you do not regret it?
No, not at all.
Do you have a future in Macedonia?
From my perspective, no, unfortunately.
What would Macedonia need in order
to be more attractive for young
scientists?
The problem is that we don’t belong to
the EU. So we don’t have a lot of
funding projects. Also, we basically
have zero industry. We have a lot of
software companies, but those
companies are mainly outsourcing from
German or Swedish companies. It’s not
like you can perform any kind of
research there. That’s what is missing.
Macedonia gave me the basis also from
university because I finished it there. It
has a really good education system. . I
always think about how I can give back.
Until now, I haven’t found the time. It
has been a wish that I hope in the next
few years I can accomplish. Maybe I
can start a project with some of the
university professors. Last year, I also
went to visit them. They also invited
me for a talk, but a talk is one thing.
You just give it and go out. Then I go
 CVPR
34 Women in Comp. Vision Tuesday DAILY
back to Germany. Doing something
more is, for me, something that I have
to do. I will hopefully do it at some
point.
I’m sure that you will. When did you
discover that you had a passion for
science?
It was quite easy. My father is an
electrical engineer. He repairs, well
repaired, actually he is going to retire
this year. Sometimes he would bring
the medical devices home and
continue working. At home, we are
three women: my sister, me, and my
mother. He sometimes needed one of
us to help measure. Sometimes two
hands are not enough. He often asked
me and my sister as well. I was the one
who wanted to do it.
He always explained to me how the
whole machine works, which are the
sensors, how do they measure the
sugar in the blood. I knew since I was
in fourth or fifth grade that I would
study electrical engineering and then
computer science.
So you have one sister and your mom.
Who is the most scientific of the
three?
Me.
What would you have done if you
were not a scientist?
Honestly, I don’t know. It was so clear. I
was really good at math and physics. I
was bad at languages basically. It was
very obvious. My sister wanted to
study medicine, but then there were
discussions in the family about what is
the benefit of it or not. Then she
decided also to do electrical
engineering.
So if we need a medical doctor, we
need to go to another family, not to
yours.
Definitely… It’s funny. In Germany, my
passport says doctor. On an airplane,
they say, “Ah, a doctor.” I’m not a
doctor. I cannot help you if something
goes wrong on an airplane. This
actually happened once. On the
airplane, someone didn’t feel well. It
was a long flight overseas. Then they
said, “Can all the doctors please come
to the stewardess?” It says so officially,
but I am not! [both laugh]
Regarding the move to BMW, did it
come by chance?
It came by chance. I was doing my PhD
in the area of safety platforms for
future autonomous systems. I knew I
wanted to bring what I had done in my
PhD to a real product. In a PhD, you do
a prototype. It works in some cases. In
a conference, people present what
works or doesn’t work. I wanted to do
CVPR
DAILY Tuesday Jelena Frtunikj 35

I had to get good grades. I knew I had

to get good internships, some
extracurricular stuff. It’s not that easy!
I know it sounds really obvious and
very easy, but it wasn’t. I really work
hard at whatever I want to achieve.
There is a plan. Even now, if I think
about what I want to do, I know what I
have to learn as a basis, and what I
have to achieve before I can get there.
What is the most precious lesson that
you received from your teachers?
it for a real product. At that point in [laughs] I don’t know if it sounds very
time, there was a colleague at the cliché, but from my math teachers:
same university who moved to BMW. discipline and hard work. That’s also
He asked me, “Jelena, we are looking what my father told me every day. He
for people. You are perfect. You have also taught me that there are no easy
safety. You have computer science. Do ways in life. I can tell a story that
you want to apply?” I sent him my CV. basically no one knows except in my
It was really very spontaneous. I didn’t family and my boyfriend.
look for anything else. It looked quite
challenging because BMW wanted to So let’s reveal it!
implement autonomous driving, but I I finished elementary school, which is
can help!
From what you say, it seems like you
followed your career because it was
obvious. I’m sure this is not the
reality. I’m sure there were times
when it was not so obvious. What
were those times?
I understand your point, but it’s
actually my character. In whatever I do
in my life, I have some kind of vision of
what I want to do. For example, I knew
from the beginning of my studies that I
wanted to go and study abroad. In
Macedonia, the salary is very low. They
cannot sponsor, let’s say, going
somewhere. I knew I had some time. I
searched the internet on how I could
get scholarships and how I can get
some experience in between. I had
everything planned. I didn’t know if
everything would work out, but I knew
 CVPR
36 Women in Comp. Vision TuesdayDAILY
eight years in Macedonia.
Then I went to high school. It was not a
problem. I passed all the exams. I was
in a class which, on the first day, didn’t
look like nice people in the class. I
didn’t know anyone. I knew some
people in the other classes. It looked
like all nerds. I came back from school.
I went to my dad and said, “Please can
you find any solution to move me to a
different class?” He asked, “Why?” I
said, “You know, all of them are nerds.
It doesn’t look nice. I don’t want to be
there.” He said, “You know what, you
can’t get everything easily in life.
Either, you stay in that class or you
don’t go to school anymore. That’s my
decision. I won’t help you.” That’s what
he told me when I was 14.
And you didn’t take advantage of this
to quit school?
[laughs] No, that was not an option. I
always wanted to improve myself and
learn new stuff.
So one more time, you tell me that
you took the obvious choice.
Again, yeah! [both laugh]
Read more interviews with
women scientists
CVPR 37
DAILY Posters - Workshops

Top: Cecilia Xuaner Zhang presenting her poster Zoom to Learn, Learn to
Zoom. Hers was one of the most crowded posters in a very crowded
Tuesday afternoon poster session!
Below: runner-up best paper award for “SizeNet: Weakly Supervised
Learning of Visual Size and Fit in Fashion Images” in Understanding
Subjective Attributes of Data Focus on Fashion and Subjective Search
FFSS-USAD workshop. Author Nour Karessli (left, Zalando SE) is greeted
by Diane Larlus (Naver Labs Europe) and Nicu Sebe (University of Trento).
 38 Tool
Computer Vision News

Object Tracking in Python Using OpenCV

by Amnon Geifman
“…tracking and detection are not
the same tasks!”
In this article we will explain and implement the object tracking algorithms in
OpenCV library. For many computer vision applications, we are given a video that
contains an object, and we want to track the object's movement across different
frames. The problem is usually solved using a bounding box that contains the
object. The bounding box will be represented by four numbers: two for the
location of the upper left angle of the rectangle and two for the lower right.
Some of you may ask why we don't use YOLO or mask R-CNN on each frame and
get the object's location? The answer is that tracking and detection are not the
same tasks. Detection is a much harder task, which requires many more
resources. Hence tracking is much faster than detection. Moreover, tracking
usually preserves the location of the object; on the other hand, applying
detection on each frame separately might not consider the relationship between
the different frames.
When choosing an object tracker, we need to consider its three major
Focus on

properties:
Speed: in some application, we want our tracker to be able to process hundreds
of frames per seconds (FPS). In other applications, the tracking is done offline so
the speed is not very important.
Accuracy: of course, we want our tracker to be accurate. However, in general,
there is a trade-off between the accuracy and running time. A fast tracker might
be inaccurate and vice versa.
Occlusion robustness: when tracking a moving object, occlusion by other objects
might be a common phenomenon. In some application, we want our tracker to
cope with such occlusions and without moving to track the occluding object.
Luckily enough, OpenCV library contains a tracking API with 8 tracking
algorithms; Boosting, MIL tracker, kernelized correlation filters (KCF),
discriminative correlation filter, median flow tracker, TLD tracker, MOSSE tracker
and GOTRUN tracker. Each of those trackers has different ratio of speed to
accuracy: in this article we will use the KCF tracker since it is fast, relatively
accurate and good at failure detections.

Tracking with Kernelized Correlation Filters (KCF)

In tracking tasks, our goal is to find the object in the current frame, given its
location, shape and structure in the previous frames. The main idea of the KCF
Object Tracking in Python Using OpenCV 39
Computer Vision News

algorithm is to build a correlation filter (i.e. kernel) such that a convolution with
the input image gives the desired response. This desired response usually has a
Gaussian shape centered around the object and decreasing with the distance. To
calculate the optimal filter, the algorithm uses translated instances of the object
from the previous frames. The maximal filter response is taken to be the object
location.
In order to explain how the kernel trick works (like in SVM), we first start with
linear correlation filter. The optimal linear filter w is found by solving the
following least squares problem of the form:
2
min{ 𝑋𝑤 − 𝑦 + 𝜆| 𝑤 |ൠ
𝑤
Here, X is a circulant matrix containing all the possible cyclic image shifts, λ is a
regularization coefficient term and y is the response that we expect to receive.
The advantage of this formulation is that, given such circulant matrix X, we can
find the optimal weights in the Fourier domain w* using a closed form solution.
Like in SVM, the kernel trick allows us to perform a non-linear regression by
mapping the input using a non-linear mapping. In this way, the weights have the
form of 𝑤 = ∑𝛼𝑖 ϕ(xi ሻ and the minimization problem is of the form:
2

Focus on
min 𝐾𝛼 − 𝑦 + 𝜆𝛼 𝑇 𝐾𝛼
α

Here the matrix K is the kernel matrix with entries 𝑘𝑖𝑗 = 𝜙 𝑥𝑖 𝜙(𝑥𝑗 ൯ , thus
we can solve in a closed form in the Fourier domain as well. In this method,
usually an RBF Gaussian kernel is used.

Implementation
Before we begin to dive into the code, note that some of the tracker described
above are not available in old versions of OpenCV so make sure that you pip-
installed the latest version of OpenCV. Moreover, make sure you have opencv-
contib-python on your environment, we need it to create the tracking object.
We begin with importing the cv and sys libraries:

1 import cv2
2 import sys

Next, we define the tracker object. As mention above, we have eight possible
trackers to use and we used the KCF. For other trackers, you can take a look on
the OpenCV documentation. Defining the KCF tracker is simple as:
 40 Tool
Computer Vision News

1 tracker = cv2.TrackerKCF_create()

Now we are ready to preprocess the video. Essentially, we open the video and
read the first frame in the video; in this example, I'll also show you how to rotate
each frame in case your video is not aligned. This can all be done in the following
code:

1 video = cv2.VideoCapture("IMG_1003.MOV")
2 flag, frame = video.read()
3
4 (h, w) = frame.shape[:2]
5 center = (w /2 , h / 2)
6 M = cv2.getRotationMatrix2D(center, 270, 1)
7 frame = cv2.warpAffine(frame, M, (h, w))

In the above code, the first line loads the video from a file in my computer. The
second line is a standard extraction of the frame from a video. The flag
argument specifies if the frame extracted successfully. The last 4 rows just rotate
the given frame by 270 degrees.
Focus on

At this stage we are ready to begin. In this code, the region of interest, i.e. the
initial bounding box, will be defined manually. It is not very hard to use some
pre-trained network to perform the initial detection. For example, YOLO
network can extract a good bounding box for the object; however, we keep it for
future articles in order to explain detection methods in details. The ROI selector
of cv2 opens the frame in a dialog box and allows us to mark a rectangle over
the object. The output will be a tuple with four numbers, two for the upper left
side and two for the lower right. We use the ROI selector as:

1 box = cv2.selectROI(frame, False)

The last thing that we still need to do is to use the box that we defined. Then, we
will iterate through the frames and for each frame we update the tracker. If the
tracking succeeds, we draw it on the frame and show it. Otherwise we just show
the frame. At the end we make an option to break the while loop in the case
where esc is pressed or no frames are available. This code will look like this:
Object Tracking in Python Using OpenCV 41
Computer Vision News

2 flag = tracker.init(frame, box)

3 while True:
4 flag, frame = video.read()
5
6 flag, box = tracker.update(frame)
7
8 if flag:
9 p1 = (int(box[0]), int(box[1]))
10 p2 = (int(box[0] + box[2]), int(box[1] + box[3]))
11 cv2.rectangle(frame, p1, p2, (0,0,255), 2, 1)
12
13 cv2.imshow("Tracking", frame)
14
15 k = cv2.waitKey(1) & 0xff
16 if k == 27 : break
17

This is actually it. In my code, I added a few additional lines of code to measure
the number of frames per second and to make the visualization nicer, but this
code is enough to track any object! Let's see some examples:

Results

Focus on
We now demonstrate the performance of the tracker. To this end we track the
phrase FOCUS ON ME on the following video. You can see the results below.

Conclusion
In this article we have seen how to use tracking algorithms in OpenCV.
Specifically, we explained and implemented the KCF tracker which gave us the
desired speed and accuracy we wanted. As mentioned above, there are seven
more trackers implemented on OpenCV. Each of them has different properties
and different robustness. You can use our code as is and just replace the line
that define the tracker. This would allow you to check other trackers depending
on your application. Enjoy!
42 Autonomous Driving: AutoX
Computer Vision News

With Jianxiong Xiao

AutoX is a self-driving car company, focused on developing self-driving car
technology. Their core product is an AI platform that can be installed on
different kinds of vehicles for applications such as logistics delivery services,
airport transportation, and city driving in smart, 5G-connected cities. We talked
about it with Jianxiong Xiao, CEO and founder of AutoX.

Xiao tells us that there are many
automakers around the world making
cars, and there are many sensor
companies making Lidar and radar
cameras, but what is missing is a good
AI platform.
The self-driving AI platform has many
components. The first is high-definition
3D mapping and HD mapping. Next is
localisation. Localising where the
vehicle is in relation to the HD map.
Then it’s perception, including object
detection, object recognition, object
tracking, as well as object prediction to
predict what an object is going to do
next.
Then there is a decision-making
component called behaviour planning.
Given the road condition and the
traffic situation, the car has to make a
smart decision about whether it’s
going to stop, go, turn left, or turn
right. Next comes motion and speed
planning. After it has made that
decision, it has to plan the trajectory
and exact speed that it is going to drive.

“…they are the only company in the world with AI

sophisticated and smart enough to deal with
the very dense and challenging traffic there”

Computer Vision News
The next step is about controlling the
vehicle to drive autonomously. The
control panel has two parts – the
algorithm side and the hardware side.
The vehicle control unit has to send an
electrical signal to the vehicle in order
to control the steering and the
throttle.
This is a full-stack solution from the
very beginning to the end for self-
driving cars. It is backed up by a data
infrastructure built by gathering a huge
amount of data from global testing
fleets. Before AutoX cars hit the road,
they are tested heavily through digital
simulation.
Jianxiong emphasises how important it
is that they make no mistakes. Being a
self-driving car, they require 100 per
AutoX 43
cent accuracy. Not a single
compromise is allowed. He is very
proud that they have put together
such a robust and reliable system.
The solution covers many aspects of
computer vision. Jianxiong explains:
“For example, for perception, obviously
we need object detection. We need
semantic segmentation. Not only do
we detect the object, but we’re
segmenting out at instance-level. We
need to do this at a very fast speed.
Unlike other applications where you
can wait one second to get a picture;
for us, we need real-time performance.
When the image comes in, we need to
finish all the computation immediately.
We use a convolutional neural network
to do that.”
44 Autonomous Driving
To support object tracking, they have a
3D convolutional neural network
running in the physical space using a
LiDAR signal. They combine the LiDAR
and camera together in another neural
network. For object prediction,
because it’s a time sequence signal,
they use a recurrent neural network.
In three years, AutoX has grown to the
point that they currently have a large
Computer Vision News
testing fleet of self-driving cars in the
US and China. Jianxiong claims that
they are the only company in the
world with AI sophisticated and smart
enough to deal with the very dense
and challenging traffic there. They are
doing autonomous testing downtown
in Shenzhen, the city with the highest
population density in China, where
they are piloting a fleet of RoboTaxis.

Computer Vision News
In terms of next steps, Jianxiong tells
us:
“The next step is to further improve our
technology. To polish it. By the end of
the year, we will have more than 100
self-driving cars testing. We’re also
working with a lot of car
manufacturers. For example, we’re
working with Dongfeng Motor, the
second-largest Chinese car manufacturer.
“By the end of the year,
we will have more than 100 self-driving cars testing!”
AutoX 45
We’re also working with Shanghai
Motor. They are the largest Chinese car
manufacturer. They are both working
closely together for us to provide a self-
driving fleet to really push the
technology to commercialisation and
production.”
AutoX are currently hiring for various
positions. See their website for more
details.
 Computer
ComputerVision
VisionNews
News
FREE
SUBSCRIPTION
Dear reader,
Do you enjoy reading
Computer Vision News?
Would you like to receive it
for free in your mailbox?
Subscription Form
(click here, it’s free)
You will fill the Subscription
Form in less than 1 minute.
Join thousands of AI
professionals and receive all
issues of Computer Vision
News as soon as we publish
them. You can also read
Computer Vision News in
PDF format (though this
online view is way better)
and visit our archive to find
new and old issues as well.
We hate SPAM and
promise to keep your email
address safe, always.
Upcoming Events 47
Website and Registration
MIDL - Intern. Conf. on Medical Imaging with Deep Learning
London, UK Jul 8-10 Website and Registration
IEEE ICCI*CC Int. Conf. on Cognitive Informatics & Computing
Milano, Italy Jul 23-25 Website and Registration
MIUA - Medical Image Understanding and Analysis
Liverpool, UK Jul 24-26 Website and Registration
Medical Augmented Reality Summer School
Zurich, Switzerland Aug 5-16
Website and Registration
CMBBE Computer Methods in Biomechanics and Biomedical Eng.
New York City, NY Aug 14-16 Website and Registration
Federated Conference on Computer Science and Info Systems
Leipzig, Germany Sept. 1-4 Website and Registration
Int. Conference on Computer Analysis of Images and Patterns
Salerno, Italy Sept 2-5 Website and Registration
BMVC - British Machine Vision Conference
Cardiff, UK Sept 9-12 Website and Registration
Int. Conf. on Machine Learning, Optimization, and Data Science
Siena, Italy Sept 10-13 Website and Registration
Intelligent Health 2019
Basel, Switzerland Sept 11-12 Website and Registration
AI for Business Summit 2019
Sydney, Australia Sept 17-19 Website and Registration
RE•WORK - Deep Learning Summit
London, UK Sept 19-20 Website and Registration
The AI Summit 2019
S.Francisco, CA Sept 25-26 Website and Registration
Did we forget an important event?
Tell us: editor@ComputerVision.News
FEEDBACK
Dear reader,
If you like Computer Vision News (and also if you
don’t like it) we would love to hear from you:
Give us feedback, please (click here)
It will take you only 2 minutes. Please tell us and
we will do our best to improve. Thank you!
 IMPROVE YOUR VISION WITH

The only magazine covering all the fields of

the computer vision and image processing industry

SUBSCRIBE
CLICK HERE, IT’S FREE
Gauss Surgical

A PUBLICATION BY