Trainer

X-vision
 Trainer - X-vision

Manufacturer IMS Messsysteme GmbH

Dieselstraße 55
42579 Heiligenhaus

Tel.: +49 (0) 2056 / 975-0

Fax: +49 (0) 2056 / 975-140

E-Mail: info@ims-gmbh.de
www.ims-gmbh.de

© IMS Messsysteme GmbH. All rights reserved. This documentation is protected in all
parts by copyright laws. Distribution and reproduction without the prior written authorisation
of IMS Messsysteme GmbH are prohibited. All company and product names used in this
documentation may be registered trademarks.

2/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

Contents

1 Introduction ...................................................................................................................... 5

2 Fundamentals .................................................................................................................. 5

3 Classifier: Decision Tree ................................................................................................. 5

4 Learning Algorithms ........................................................................................................ 6

4.1 Peculiarities........................................................................................................................ 6
4.1.1 Interpretability .................................................................................................................... 7
4.1.2 Restriction of Separation Using Threshold Values ............................................................ 7
4.1.3 Feature Selection ............................................................................................................... 7
4.1.4 Performance ...................................................................................................................... 7
4.2 Variants .............................................................................................................................. 8
4.2.1 Decision Tree ..................................................................................................................... 8
4.2.2 Random Trees ................................................................................................................... 8
4.2.3 Extremely Randomized Trees ........................................................................................... 8

5 Tasks ................................................................................................................................. 9
5.1 Main View .......................................................................................................................... 9
5.2 Viewing the Training Data .................................................................................................. 9
5.3 Creating and Editing Classes .......................................................................................... 10
5.4 Reclassification of Defects ............................................................................................... 10
5.5 Structure of the Work Sets............................................................................................... 10
5.5.1 Training Set...................................................................................................................... 11
5.5.2 Reference Set .................................................................................................................. 11
5.5.3 Test Set............................................................................................................................ 11
5.6 Import of Defect Data into the Training Database ........................................................... 12
5.6.1 Synchronise Class Definitions ......................................................................................... 13
5.7 Training Database Backup .............................................................................................. 14
5.7.1 Backup ............................................................................................................................. 14
5.7.2 Restore ............................................................................................................................ 14
5.7.3 Delete............................................................................................................................... 15
5.8 Classifier Training ............................................................................................................ 15
5.8.1 Recalculation of the Features .......................................................................................... 15
5.8.2 Selection of Classes for Training ..................................................................................... 15
5.8.3 Selection of the Features ................................................................................................. 15
5.8.4 Selection of Classifier Variants ........................................................................................ 15
5.8.5 Training ............................................................................................................................ 16
5.9 Evaluation of a Classifier ................................................................................................. 17
5.10 Cross Validation ............................................................................................................... 18
5.11 Activating a Classifier ...................................................................................................... 18

SG_1206_02_en Vers.: 01.03.2019 3/21

 Trainer - X-vision

6 Diagnosis ........................................................................................................................ 19
6.1 Segmentation ................................................................................................................... 19
6.2 Features Diagnosis .......................................................................................................... 19
6.3 Probabilities ..................................................................................................................... 21

7 Reports ........................................................................................................................... 21

8 Settings ........................................................................................................................... 21

4/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

1 Introduction
This documentation describes the basic functions of the Trainer. The Trainer provides a complete range
of tools for the creation and evaluation of new classifiers. The defect data necessary for this can be
viewed and manipulated.

2 Fundamentals
These instructions presuppose that you already know the fundamental operating concepts as described
for the Inspector.

3 Classifier: Decision Tree

The system uses decision trees to classify the defects detected. They are described briefly below. You
can find a detailed description of the decision trees used in the respective reference documents.

A decision tree for classification is a binary tree in which the classification results are located at its leaf
nodes. All other nodes contain decision rules leading to a leaf node.

A prediction for an unknown defect begins at the root. The decision rules are evaluated recursively and
continued in the corresponding sub-trees.

Leave the House

Yes Raining? No

Umbrella
Rain
Predicted
Yes ? No

Umbrella No Umbrella

Figure 1) Example of a simple decision tree for the decision as to

whether we should take an umbrella with us when leaving home

SG_1206_02_en Vers.: 01.03.2019 5/21

 Trainer - X-vision

4 Learning Algorithms
The decision trees used here are trained by monitored learning. The learning algorithm is presented a
correctly classified training set, whereupon it generates decision and result nodes independently.

The tree is basically generated recursively in the following steps:

1. Selection of a separation criterion so that the result subsets are as “pure” as possible. All feature
values available can be used for this.
2. Once useful separation of the subsets is no longer possible, cancel and assign the result nodes
a class.
3. Otherwise insert decision nodes and continue with 1 for both resultant subsets.
4.

Figure 2) Possible separation of a simple training set

To avoid overfitting, a tree obtained in this way can subsequently be pruned.

Defect

X < A?

Y < B? Y<C

Class Blue Class Red Class Green

X<D

Class Yellow Class Green

Figure 3) Example of a resultant decision tree for classification

4.1 Peculiarities
A decision tree has a number of peculiarities that distinguish it from other classifier technologies and
which must be taken into consideration when using such a tree.

6/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

4.1.1 Interpretability
A decision tree generates numerous understandable rules so that the plausibility of the learning success
can be checked manually on the basis of the generated trees.

4.1.2 Restriction of Separation Using Threshold Values

In the real features area the classification is based solely on a separation of the feature values according
to the threshold values. This separation always only takes place in one dimension. More complex
correlations can therefore not be represented. This type of separation already fails in simple linear
correlations.

To remedy the situation, it is necessary to generate features that depict such correlations in separate
form. For this reasons a multitude of features and features derived from them are often used.

Not separable Separable

Figure 4) Example of a non-separable problem that can be turned into a

separable problem through simple axis rotation

4.1.3 Feature Selection

Based on the separation criteria selected, the decision tree independently selects a subset of features
available for classification. This makes the final result relatively robust against the use of features that
do not help to solve the problem.

The automatic feature selection can therefore also be used for diagnostic purposes. If, for example, the
classifier uses obviously irrelevant features, this can suggest a poorly selected training set.

4.1.4 Performance
Efficient learning algorithms allow acceptable training times of only a few seconds or minutes even when
the training set is very big or many features are used.

Due to the simple rule structure, decision trees can be evaluated online very quickly. Even if several
classifiers are processed at the same time, the computing time for this is insignificant compared to prior
image processing.

SG_1206_02_en Vers.: 01.03.2019 7/21

 Trainer - X-vision

4.2 Variants
The system allows selection of various classifier variants.

4.2.1 Decision Tree

This tree is a simple decision tree according to Breiman, Friedman, Olshen and Stone “Classification
and Regression Trees”, Wadsworth (1984). This basic variant essentially reflects the above-mentioned
concepts.

4.2.2 Random Trees

A variant presented by Breimann and Cutler that does not use only one decision tree, but several
simultaneously. To this end, the training set is divided into random subsets, which are used
independently of training of a decision tree. The forecast depends on majority decision of all trees trained
in this way.

All trees are generated with the same parameters, but different training sets. A random subset of the
features is used additionally at every node for separation purposes.

4.2.3 Extremely Randomized Trees

This is a further variant that evaluates several trees at the same time, by Geurts, Ernst and Wehenkel
“Extremely randomized trees” (2006). In contrast to Random Trees, the same training set is used for all
trees. The selection of separation criteria for this is extremely randomised.

Like Random Trees, this variant is a voting procedure along all decision trees evaluated simultaneously.

8/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

5 Tasks
5.1 Main View
The main view provides access to a few basic functions at all times.

- The current view and main filter can be configured in the tool bar.
- Task-specific views can be opened with the page selector.

Menu

Toolbar

Page Selection

Page

Figure 5) Breakdown of the main view

5.2 Viewing the Training Data

All data accessible to the Trainer can be viewed in the view “Browse”. The class filter and status filter
can narrow down the current selection.

Class Filter Image Display

Image List Defect List

Figure 6) Screen breakdown in the view “Browse”

Every defect can be marked for a specific use via its status. Defects generated automatically by the
system first always have an undefined status.

The Trainer only shows defects that have at least a status of “Interesting”. Please use the Inspector to
mark new defects interesting for training.

The defects that are to be shown can be configured with the filters in the tool bar.

Figure 7 Defect status filter in the tool bar

SG_1206_02_en Vers.: 01.03.2019 9/21

 Trainer - X-vision

Selected statuses are stored. If no status has been selected in the filter, the defect display is empty.

A defect can be classified multiple times and can also be reclassified. The results are stored in separate
fields, thus allowing a certain degree of transparency.

The current classification results shown can be configured via the tool bar. At some points the Trainer
can show two results at the same time (defects list, confusion view).

Figure 8) Display of configure classification results

This is set by default for the most common use, namely comparison of currently predicted and actual
results. Thus the labels “Predicted” and “Actual”. In principle, however, any combination is allowed here.

The word “Actual” is used for labelling in the image, the tool bar and the histogram.

5.3 Creating and Editing Classes

The class filter can, presuming the user has the necessary rights, be used to create and edit classes.
Please see the documentation for the Inspector for a detailed description.

5.4 Reclassification of Defects

Reclassification of defects works in the same way as in the Inspector. It is described in the documen-
tation for the Inspector.

5.5 Structure of the Work Sets

Defects marked as interesting with the Inspector are available to the Trainer. They are shown when the
corresponding filter is set in the tool bar. The defects are shown on the page “Browse”.

Figure 9) Work set filter

It is recommended that you see the defect status “Interesting” as an inbox for new defects and that you
regularly look through it for new candidates for the work sets.

10/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

The work sets should fundamentally only contain validated defects. The quality of training and evaluation
depends decisively on the reliability of the underlying data.
To give defects a new status or to move them between work sets, the context menu can be opened on
selected defects by right-clicking. You can alternatively also use the function keys to set the status of
the currently selected defects.

F7 Mark as undefined
F8 Mark as interesting
F9 Move to reserve set
F10 Move to test set
F11 Move to training set
F12 Move to reference set

Table 1 Function keys to change the defect status

Please note that it is possible to set the status back to “Undefined”. The defect is then no longer shown
in the Trainer. This can be used to remove defects permanently from the work sets.
5.5.1 Training Set
The defects of the training set are used for classifier training. The training set used has a decisive
influence on the quality of the resultant classifier.

The following principles can help to create a successful training set:

- uniform distribution of the defects over all classes in the training set
- homogenous defects / noticeable similarities within a class
- good segmentation of the trained defects
- no mixed defects (e.g. water and scratches within the same segment).

5.5.2 Reference Set

In order to compare classification results objectively, a reference set must be referred to. The reference
set should be large enough and the performance reflect production. It should fulfil the same
requirements regarding segmentation quality and distribution as the training set.

New classifiers are optimised against this set. The elements from the reference set may not exist in the
training set.

If an optimised classifier performs badly in production, it can be extended successively. Previous

classifiers can be tested against the reference set again should a comparison be wanted.

5.5.3 Test Set

The test set can be adapted to current requirements. You are free to put the test set together as you
like. To obtain generally meaningful tests, however, they should fulfil the same requirements as the
reference set.

SG_1206_02_en Vers.: 01.03.2019 11/21

 Trainer - X-vision

5.6 Import of Defect Data into the Training Database

The Trainer works with a database that is separate of production.

- Changes to and experiments with the training data do not change the production data.
- The training data can easily be saved separately.

That means, however, that new data first have to be imported before they can be used in the Trainer.

When the Trainer is launched, the defect data are compared in the background. If new defects are found
in the production database, this is shown in the tool bar.

Figure 10) Indication in the tool bar that there are new defects available for import

A comparison can also be started manually with the menu item “Import - Defect import”.

The import dialogue shows the defects that have been found. They can then be imported or rejected. A
rejection causes the defect status to be reset to “Undefined” in the production database.

Figure 11) Import dialogue for defect data from production

The import can then be set to run in the background. The current status can be checked at any time
with the menu “Import - Defect import”.

When the import is finished, the view must be updated with the refresh function in order to be able to
work with the new defects.

12/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

5.6.1 Synchronise Class Definitions

The class definitions can be edited in both the production and the training database. New elements are
synchronised automatically. Subsequent changes are marked as conflict and must be resolved
manually.
When the Trainer is launched, the class definitions are compared in the background. If new defects are
found in the production database, this is shown in the tool bar.

Figure 12) Indication in the tool bar that there is a class conflict

A comparison can also be started manually with the menu item “Import - Class synchronization”.

Figure 13) Dialogue for synchronisation of class definitions

The definitions can be applied from production to training and vice-versa with the arrow buttons. This
can be done for all conflicts at once or individually.

Changes are only written back to the databases after final confirmation.

SG_1206_02_en Vers.: 01.03.2019 13/21

 Trainer - X-vision

5.7 Training Database Backup

The backup dialogue for the training database can be opened with the menu “Database Backup/restore”.
It is used to create new backup points or restore backups.

Figure 14) Backup dialogue for the training database

5.7.1 Backup

Creates a new backup file of the current training database. The file name is counted up automatically.
The comment should describe the backup point effectively.

Figure 15) Dialogue for creating a backup file of the training database

5.7.2 Restore
Restores a selected backup. The current state is lost as a result. It is therefore advisable to make a
backup of the current state before restoring an older file.

14/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

5.7.3 Delete
Deletes the selected backup file irretrievably.

5.8 Classifier Training

Classifier training can be started on the page “Classifier”. Here you can experiment with various settings.
Only when a classifier has been enabled explicitly is it also used by the inspection system.

Prerequisite for training is a full training set.

5.8.1 Recalculation of the Features

If you are not certain, it is recommended that you select this option at least for the first training run.

5.8.2 Selection of Classes for Training

Here you can select the classes that are actually to be used for training. It can be useful to exclude
some classes from training test-wise or permanently, for example when there are not a sufficient number
of trainable defects available. The class list on this page offers the possibility for this.

5.8.3 Selection of the Features

It is not necessary to use all features for training. It is possible that an irrelevant feature or a feature that
is not distributed uniformly over the training set will mislead the classifier. Disabling certain features can
therefore have a positive effect on the achievable classification rates.

The features that are to be used for training can be set on the page “Training”. The training page shows
how many features will be used.

Figure 16) Feature selection

Selection boxes show which features will be used. Filter conditions and the context menu can be used
to add several features for training or exclude several features from training at the same time. The
current selection can be saved for later use.

5.8.4 Selection of Classifier Variants

The Trainer offers a choice of classifier variants (see 0

SG_1206_02_en Vers.: 01.03.2019 15/21

 Trainer - X-vision

Variants). Please see the selection dialogue of the page “Classifier” for the current choice available.

5.8.5 Training
The button “Train” starts training. A progress bar is shown during processing. After training the view
changes to the confusion matrix for evaluation (see 4.9)

16/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

5.9 Evaluation of a Classifier

One tool for evaluation of a classifier is the confusion matrix. In this view the actual classes (Actual,
columns) and the predicted results of the classifier to be tested (Predicted, lines) are compared.

The diagonals contain the number of correctly detected defects for every class. Deviations from the
diagonals were classified incorrectly.

Figure 17) Confusion matrix

The current confusion matrix can be saved for a later comparison. It can also be exported as Excel-
readable CSV (comma separated value) file.

In addition to this, this view presents a number of values from which the performance of a classifier can
be read. They can be switched on and off via the context menu (right-click).

Definition Description
Accuracy Proportion of correctly classified defects in the
total set examined.
Treffer-
genauigkeit
Precision Proportion of the correctly classified results of
a class in the predicted results of this class.
Präzision
A high precision means good reliability
→ when the classifier predicts class A for a
defect, this is probably correct.

Recall Proportion of the correctly classified results of

a class in the actual number of this class.
Erinnerung
A high recall means a good detection rate for
this class
→ when the classifier examines a defect in
class A, it will probably detect it.

SG_1206_02_en Vers.: 01.03.2019 17/21

 Trainer - X-vision

F-score The F-score combines the two values

Precision and Recall in an overall performance
F-Maß value.

5.10 Cross Validation

The system uses a stratified cross validation that generates a uniform distribution in the subsets.

The cumulated individual results are shown together in the confusion matrix.

5.11 Activating a Classifier

The main view of the page “Classifier” contains the button “Activate” to activate the current or a
previously trained classifier.

Before a classifier can be activated, it must first be trained and saved. Should a currently selected
classifier have already been trained, it is saved automatically before activation. It is then shown as
preselection. However, any previously saved classifier can be selected with the selection dialogue.

Figure 18) Activation dialogue for classifiers

When a file has been selected, the classifier can be activated with “Activate”. It is reloaded by the
inspection system automatically at the next possible opportunity and is used from then on, i.e. from the
next inspection.

18/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

6 Diagnosis
The Trainer provides various diagnosis possibilities. They are described below.

6.1 Segmentation
You can open a further window with additional information on the segmentation of the current selected
image with the menu Diagnostic - Segmentation.

This function has already been described in the documentation for the Inspector.

6.2 Features Diagnosis

A further window containing all feature values for the current filter setting can be opened with the menu
Diagnostic - Features.

Figure 19) Features diagnosis

The selection is synchronised automatically with the defect selection on the page “Browse”.

Apart from the possibility of selecting features for training, the page “Features” also presents a few
statistics that can also be useful. Apart from Minimum, Maximum and Mean, they also include a
histogram of the currently selected feature.

SG_1206_02_en Vers.: 01.03.2019 19/21

 Trainer - X-vision

Figure 20) Feature distribution

Two features can be compared with each other with a comparison function. To do so, check the box
“Compare”. When a new feature is selected, the statistical display changes to comparison mode. In this
way any dependencies between the features can be made visible.

Figure 21) Relative feature distribution of two features

Removing the check mark switches the system back to the single feature view.

20/21 Vers.: 01.03.2019 SG_1206_02_en

 Trainer - X-vision

6.3 Probabilities

Some classifiers allow output of probabilities for individual classifications. There must be a trained
classifier available for this. It can be displayed via Diagnostic - Probabilities.

Figure 22) Probabilities of a classifier result

7 Reports
The Trainer allows reports to be generated on the current selected defects. They can be adjusted to
specific requirements with the Report Manager.

8 Settings
The Trainer allows setting of the connection to the data source. The Trainer should usually be configured
such that it uses the data service set up.

Figure 23) Settings

In exceptional cases, however, the Trainer can also communicate directly with a test database. In this
case it should be ensured that the current Windows login has sufficient rights in the database.

SG_1206_02_en Vers.: 01.03.2019 21/21