6/24/2020

Modern Scientific Method

Ir. Noor Akhmad Setiawan, Ph.D., IPM.
APTIKOM, FORTEI, IEEE Indonesia Section, IEEE SMCS Indonesia Chapter
Department of Electrical and Information Engineering
Universitas Gadjah Mada

CV
Education:
Bachelor in Electrical Engineering, Universitas Gadjah Mada (1998).
Master in Electrical Engineering, Universitas Gadjah Mada (2003).
PhD in Electrical and Electronics Engineering, Universiti Teknologi PETRONAS (2009).
Professional Engineer in Electrical Engineering, Universitas Gadjah Mada (2018).
Certification:
Insinyur Profesional Madya (Senior Professional Engineer), Persatuan Insinyur Indonesia.
Professional Activity:
Chair, Bachelor in Biomedical Engineering Program, Universitas Gadjah Mada.
Head, High Voltage Engineering Laboratory, Universitas Gadjah Mada.
Secretary, IEEE Indonesia Section.
Treasurer, IEEE SMCS Indonesia Chapter.
Member, IEEE SMCS Technical Committee on Soft Computing.

1
 6/24/2020

CV
Journal Editing Activity:
Associate Editor – Communications in Science and Technology
Associate Editor – International Journal of Business Intelligence and Data Mining
Editorial Review Board Member – International Journal of Rough Sets and Data Analysis
Dewan Redaksi – Jurnal Nasional Teknik Elektro dan Teknologi Informasi
88 reviews of 61 journal manuscripts, mostly WoS CC (83 reviews of 56 journal
manuscripts).
Detailed CV: https://noorwewe.staff.ugm.ac.id/cv/

Research Peer Review Profile

2
 6/24/2020

Research and Publication Profile

Research and Publication Profile

3
 6/24/2020

Research and Publication Profile

What is Research and Development?

• Research is a process that acquires new knowledge.
• Development is a process that applies knowledge to create new
devices or effects.

Research seeks truth, while development seek utility.

4
 6/24/2020

Criteria of Valid Knowledge

• The criterion of reproducibility requires that all independent
attempts to accomplish a task under the same conditions yield the
same results.
• The criterion of completeness requires that the propositions and
methods used to accomplish a task be completely and clearly
disclosed using standard terminology.
• The criterion of objectivity requires that the governing propositions
for a task be free of personal bias and universally acceptable as
reasonable and valid.

Categories and Types of Knowledge

5
 6/24/2020

Conclusive Knowledge
• Conclusive propositions express knowledge obtained by statistical
inference or formal proof.

Conclusive Knowledge
• A result is an intermediate measure of performance for a task, e.g.
“Two average speeds are different.”
• A conclusion is the final result of a task that states the extent to
which the task objective has been achieved, “The first highway traffic
is faster.”
• A theorem (or lemma) is a conclusion that has been formally proven,
e.g. Pythagorean theorem.

6
 6/24/2020

Scientific Method
• The Scientific Method comprises four sequential phases – Analysis,
Hypothesis, Synthesis, and Validation – which are applied to task
iteratively and recursively to achieve the objective of the task.

Scientific Method
• Analysis
• Hypothesis
• Synthesis
• Validation

7
 6/24/2020

Analysis
• Describe Problem
• Set Performance Criteria
• Investigate Related Work
• State Objective

Hypothesis
• Specify Solution
• Set Goals
• Define Factor
• Postulate Performance Metrics

8
 6/24/2020

Synthesis
• Implement Solution
• Design Experiments
• Conduct Experiments
• Reduce Results

Validation
• Compute Performance
• Draw Conclusions
• Prepare Documentation
• Solicit (obtain) Peer Review

9
 6/24/2020

Analysis Phase
• The objective of the Analysis Phase of the Scientific Method is to gain
a thorough understanding of the components of the problem
domain, leading to the formulation of a single specific and reasonable
task objective.

Describe Problem
• A problem statement is expressed as an interrogative sentence, a
declarative sentence, or an imperative sentence that summarizes a
question, complaint, or requirement, respectively.

10
 6/24/2020

Describe Problem(2) – Simple Examples

Describe Problem(3) - Examples

Problem Description:
The same material surface can appear to have different intensity and contrast due
to varying environmental illumination conditions. The same material surface can
also appear to have different scale and orientation due to varying viewpoints. The
appearance of a material surface changes drastically with varying imaging
conditions where viewpoint and illumination are changed. In natural images it is
common that the color and texture of adjacent material classes are visually very
similar, and the texture in a class region can vary gradually in scale and orientation.
In addition, it is common that there is visual color variation within the same
material class. Most available material (or object) surface classification methods
tend to work well with a few classes of selected materials over a limited range of
imaging conditions, but often fail when applied to unconstrained natural images,
where viewpoint and illumination vary and the number of test classes is large.

11
 6/24/2020

Describe Problem(4) - Examples

Problem Statement:
Most available material (or object) surface classification methods tend
to work well with a few classes of selected materials over a limited
range of imaging conditions, but often fail when applied to
unconstrained natural images, where viewpoint and illumination vary
and the number of test classes is large.

Set Performance Criteria

• Performance criteria are requirements that any proposed solution to
the problem must fulfill.
• Criteria are stipulative knowledge propositions, which may be
characterized as facts, accompanied by appropriate explanations and
justifications.

12
 6/24/2020

Set Performance Criteria(2) - Examples

The proposed performance criteria for the viewpoint-invariant and
illumination invariant classification of natural surfaces using general-
purpose color and texture features with the ALISA δCRC classifier are:
• Must meet or exceed the classification accuracy of state-of-the-art
methods
• Must provide a lower computation cost than existing methods
• Must provide illumination-invariance
• Must provide viewpoint-invariance
• Must be useful for a broad range of application domains
• Must apply to applications with a large number of classes

Investigate Related Work

• Once the problem has been fully described, including a precise formal
problem statement, and all the a priori performance criteria have
imposed, the project team may begin,
• the process of finding out as much as possible about what has been
done to address the problem and achieve similar objectives in the
past.

13
 6/24/2020

Investigate Related Work(2)

Investigate Related Work(3) - Examples

• Investigate material surface classification and similar types of applications
• Investigate illumination-invariant color feature representations
• Investigate orientation and scale invariant texture features
• Investigate feature selection methodologies
• Investigate probability density function estimation methodologies
• Investigate classification methodologies
• Investigate available color and texture image databases for classification
performance evaluation
• Investigate classification performance estimation methodologies

14
 6/24/2020

Investigate Related Work(4) - Examples

Most existing classification methods in image processing and computer
vision use a few classical or specially designed features to discriminate
between different classes of materials (or objects). In the training phase,
these methods accumulate feature vectors of several known classes and
compute the parameter statistics (e.g., mean and variance) of the feature
vector cluster for each known class. During the testing phase, they extract
selected feature vectors from each test sample and assign it the label of the
trained class whose feature vector has the shortest distance from the feature
vector of the test sample.
These parametric or non-parametric approaches tend to work well over a
limited range of imaging conditions with very few material classes, but often
fail when applied to natural material surface classification, where viewpoint
and environment illumination vary and the number of material classes
involved increases (Graf et al., 1996; Yang et al., 1998b).

Investigate Related Work(5) - Examples

There are many texture feature extraction algorithms which have been
proposed and are available in related literature. Some texture extraction
algorithms work well in small analysis tokens (e.g., 3×3, 5×5, or 7×7);
however, some of them only perform well in large tokens (e.g., 64×64,
128×128, or the entire image). Texture features extracted using small tokens
tend to capture roughness (or randomness) characteristics from an image
surface, but are not capable of capturing the spatial-frequency information
of a texture pattern. On the other hand, some techniques, e.g., Gabor
Wavelet filtering, capture spatial-frequency characteristics of a texture
pattern, and work very well for homogeneous textures using large token
sizes.

15
 6/24/2020

Investigate Related Work(6) – Examples

The intra-class variations increase as the used token size decreases. Color
and roughness alone are not sufficient to discriminate between all selected
classes when the total number of selected test classes is large. The observed
feature distributions of different material classes of visual images are often
far from uniform and not well separated. The situation becomes even more
challenging as the number of test material classes increases, resulting in
increased classification confusion. There have been many proposed
classification approaches based on the principle of statistical modeling of
texture images, but each has been limited by either the complexity of the
models or the complexity of the images (De Bonet & Viola, 1997).

State Objective
• Once the problem has been fully described, including a precise formal
problem statement, and all the a priori performance criteria have
imposed, and the related work has been thoroughly investigated,
• then presumably the task domain has been appropriately and
sufficiently focused down and constrained.
• A this point, a specific, clear, and detailed task objective may be
formulated.

16
 6/24/2020

State Objective(2)
• Task objective is a statement of what the task is intended to achieve,
expressed as an infinitive phrase.

State Objective(3) - Examples

Objective:
To classify different natural material surface classes in color images, for
which color and texture are a reasonable basis for classification, using a
set of general-purpose color and texture features and the novel ALISA
δCRC classifier regardless of the viewpoint and illumination of the
materials to be classified, subject to the stated performance criteria.

17
 6/24/2020

Hypothesis Phase
• The objective of the Hypothesis Phase is to propose a solution to
achieve the task objective, a set of goals and hypotheses for this
solution, and the factors and performance metrics for testing the
validity of the solution.

Specify Solution
• The solution used to achieve the task objective is one to the two
components of the task method (which produces knowledge, device
or effect).
• The other component is the experiments, which measure the
effectiveness of the solution when it is applied to the task unit. The
experiments

18
 6/24/2020

Specify Solution (2) – Examples

• Postulate a set of general-purpose color and texture features
• Postulate a dynamically allocated ALISA texture module (STM)
classifier for pixel level classification using general-purpose color and
texture features
• Postulate the ALISA δCRC classifier for region/image level
classification
• Compile image databases (corpuses) for performance evaluation
• Design and implement the ALISA δCRC classifier
• Conduct classification feasibility pilots (QD pilots)

Specify Solution (3) - Examples

The proposed research solution is a new ALISA δCRC color and texture
image classifier that bases its classifications on the comparison results of the
Classification and the ReClassification distributions (δCRC) for a set of
general-purpose color and texture features. Figures 1-2 and 1-3 provide a
high-level illustration of the process steps involved in building and applying
the proposed ALISA δCRC classifier. General-purpose color and texture
features capture both color and texture information from color planes (or
representations) which are less sensitive to illumination variation. They are
defined as those features which are less sensitive to illumination variation
and can be used in a broad range of applications for which color and texture
are a reasonable basis for classification. These features are extracted from
each image pixel and its surrounding neighborhood using a small analysis
token.

19
 6/24/2020

Set Goals
• One a solution for a problem has been specified in complete detail,
the next step is to set the goals for the task, which specify how
response of the task unit will be determined when the solution is
applied to it.
• Every task has at least one goal to determine the response of the task
unit to the application of the solution, expressed as an infinitive
phrase.
• A research hypothesis is a declarative sentence that asserts a desired,
expected, or possible conclusion of a goal.

Define Factors
• From the parameters and conditions that could be varied as part of
the experiment protocol, the investigators select a subset that will be
varied to observe their effects on the performance of the task unit.
These are called factors.
• The remaining parameters and conditions are given constant levels or
settings and become the governing propositions.

20
 6/24/2020

Define Factors (2)

• From the every beginning of a project, start building a list of
knowledge propositions. Update it whenever a new proposition is
established or a previous propositions is modified.

Define Factors (3)

21
 6/24/2020

Postulate Performance Metrics

• A performance metrics is a postulate that transforms the results of
the task into measures of performance for drawing conclusions about
the task objective.

Synthesis Phase
• The objective of the Synthesis Phase of the Scientific Method is to
implement the task method (solution and experiments) to
accomplish the goals and validate the hypotheses of the task.

22
 6/24/2020

Implement Solution
• Before the goals and hypotheses for the specified solution can be
tested, it must be implemented.
• Very little specific advice can be offered for implementing solutions;
usually scientists and engineers on the project team already have the
necessary skills to achieve their specialized task.
• It can be manufactured or acquired.

Implement Solution (2) - Examples

Hardware and software: The proposed solution was prototyped and
implemented on an HP workstation. This workstation was an IBM compatible
PC with a Pentium 4 2.80 GHz processor and 2 Gbytes of memory. All related
source code for this research was written using C/C++ and MATLAB. The
C/C++ programs were coded and compiled using the Microsoft Visual C++ 6.0
Integrated Development Environment or above. The MATLAB programs were
implemented using MATLAB 7.0 or above. The required amount of memory
increases exponentially with the total number of selected features and their
precisions in the ALISA classifier implementation. The initial system had 2
Gbytes of memory with the ability to be upgraded to 4 Gbytes of memory if
necessary. The system also had a total of 250 Gbytes of disk storage, which
was sufficient for storing all training and test images.

23
 6/24/2020

Implement Solution (3) - Examples

Data collection and processing: The MATLAB programming language
and its existing support tools were used to do the proof-of-concept and
assisted with the data presentation. Necessary graphical user interfaces
(GUI) that reside in or interact between C++, MATLAB and/or Microsoft
Excel were designed and implemented to adjust the values of factors,
gather data, process data, and present the results when conducting the
designed experiments.

Design Experiments
• The purpose of this step is to design a series of experiments whose
results will eventually be used to estimate how well the task objective
has been achieved.
• An experiment acquires data to measure the performance of the
solution under controlled conditions in a laboratory.

24
 6/24/2020

Design Experiments (2)

• Before beginning the design of the experiments, carefully inventory
all the components of the task.

Design Experiments (3) - Examples

25
 6/24/2020

Design Experiments (4) - Examples

Design Experiments (5)

Experiment Laboratory
• Laboratory for a task is the set of physical regions containing the task
unit and all the task resources required during the experiments.

26
 6/24/2020

Design Experiments (6)

Block Design
• The design of the laboratory must be closely coordinated with the
design of the processes that take place in the laboratory during the
experiments.

Design Experiments (7)

Block Design - Examples

27
 6/24/2020

Conduct Experiments
• The time to put all the plans to the test.
• The only guideline: follow the plans.
• If an experiment does not run smoothly, stop it, and reenter the
Scientific Method to prepare a new and improved set of plans that
solve or avoid the problem.
• Regard a failed experiment as a pilot.

Reduce Results
• Often the performance values from which the conclusions of the task
will be drawn cannot be computed directly from the raw results of
the experiments.
• The results must be reduced, combining and/or transforming them in
some way to generate values suitable for the domains of the
performance metrics.

28
 6/24/2020

Reduce Results (2) - Examples

Validation Phase
• The objective of Validation Phase of the Scientific Method is to decide
whether the objective of the task has been achieved, based on
formal conclusion about its goal and hypotheses and a rigorous peer
review of the task methodology.

29
 6/24/2020

Compute Performance
• By definition, the reduced results files recorded in the last step of the
Synthesis Phase are the basis for computing the performance of the
task unit using the performance metrics postulated in the Hypothesis
Phase.
• Because stepwise refinement is not only allowed, but expected as
part of the application of the Scientific Method to the task, these
performance metrics may have to be revised.

Compute Performance (2)

• They may turn out to be unsuitable in the face of the actual results
acquired from the experiments, or because the goals and/or
hypotheses cannot be validated and must be redefined.
• Both outcomes may require postulating a different or modified set of
performance metrics.

30
 6/24/2020

Compute Performance (3)

• Before any performance measures are actually computed, it is
important to take a good long look at the reduced results to get a feel
for the behavior of the data.
• It may even be necessary to apply some preliminary statistical tests as
QD pilots to determine whether the postulated metrics are
appropriate, and if not, what metrics might be more appropriate.

Compute Performance (4)

31
 6/24/2020

Compute Performance (5)

Tip:
• Before computing performance, conduct a QD pilot to analyze the
raw or reduced results to detect any unusual artifacts or unexpected
trends stemming from biases in the task methods and to confirm the
appropriateness of the postulated performance metrics.

Compute Performance (6)

• The null hypothesis is the hypothesis that there is no significant
difference between specified populations, any observed difference
being due to sampling or experimental error.
• The alternative hypothesis is the hypothesis used in hypothesis
testing that is contrary to the null hypothesis.

32
 6/24/2020

Compute Performance (7) - Examples

Compute Performance (8) - Examples

33
 6/24/2020

Draw Conclusions
• Once results for a task have been assembled and the required
measures of performance have been computed and summarized in
an appropriate set of tables, charts, and graphs, the conclusions for
the task may be drawn.

Draw Conclusions (2)

• Formal conclusions are those conclusions that are required to achieve
the objective of the task and that are based on demonstrably rigorous
methods for acquiring and reducing the results of the task.
• As required by the task objective, each conclusion may be a decision
or an estimate, derived from tests of a set of explicit hypotheses or
from the results of a set of explicit goals, respectively.

34
 6/24/2020

Draw Conclusions (3)

• Once all the required conclusions have been drawn, the project team
is free (in fact, encouraged) to draw additional conclusions. Some may
be based on rigorous and formal methods, but are only identified
after inspecting the results and are often somewhat tangential to the
stated objective.
• Others, however, may simply be speculations based on insufficient
evidence.

Prepare Documentation
• If the methods and knowledge of the task remain locked in the
research notebooks and heads of the project team, all the planning
and hard work go for naught.
• Failure to create comprehensive documentation for the project
violates the completeness criterion for the provisional acceptance of
knowledge, and threatens the criterion of reproducibility as well.

35
 6/24/2020

Prepare Documentation (2) - Examples

Prepare Documentation (3) - Examples

36
 6/24/2020

Prepare Documentation (4) - Examples

Ko, T.C.T., 2008. The ALISA deltaCRC classifier for natural surfaces:
Viewpoint-invariant and illumination-invariant classification of natural
surfaces using general-purpose color and texture features with the
ALISA deltaCRC classifier (Doctoral dissertation, THE GEORGE
WASHINGTON UNIVERSITY).

Prepare Documentation (5) - Examples

Thesis Abstract:
The objective of this research is to develop a classifier that can reliably and accurately
discriminate among a large number of different natural-surfaces in color images using only
general-purpose color and texture features. The general-purpose color and texture
features are those which exhibit the least sensitivity to illumination and viewpoint variation
in a broad range of applications for which color and texture are a reasonable basis for
classification. The feature probability density function (PDF) distributions of natural-surface
classes are most definitely not disjoint, which instantly obviates the appropriate use of
Bayes' Decision Rule, which is profoundly confused by classes that are heavily overlapped
in feature space. Therefore, it was also necessary to develop a new ALISA deltaCRC
classifier that would not be confused by the many different and often highly similar
natural-surface classes. A series of experiments have been designed and conducted using
the CUReT image database, Caltech facial photo images, and natural images. An ALISA
delta CRC classifier was trained with up to 61 classes in the CUReT image database, which
presents each class in 205 different and carefully controlled viewpoint and illumination
conditions. The results with images not in the training set yielded classification accuracies
well above 95%. This research then extended the classifier feasibility tests and accuracy
measurement using mosaic images made up from different CUReT material surface classes.
Next, a skin classification experiment was conducted using Caltech facial photo images that
had a single class of interest with complex backgrounds. Finally, a natural image-content
classification experiment was performed using natural scenery images that had several
classes of interest and irregular class boundaries.

37
 6/24/2020

Prepare Documentation (6)

Ko, T., 2006, October. Viewpoint-Invariant and Illumination-Invariant
Classification of Natural Surfaces Using General-Purpose Color and
Texture Features with the ALISA dCRC Classifier. In Applied Imagery and
Pattern Recognition Workshop, 2006. AIPR 2006. 35th IEEE (pp. 26-26).
IEEE.

Prepare Documentation (7)

Paper Abstract:
The paper reports the development of a classifier that can accurately and reliably
discriminate among a large number of different natural surfaces in canonical and
natural color images regardless of the viewpoint and illumination conditions. To
achieve this objective, a set of general-purpose color and texture features were
identified as the input to an ALISA statistical learning engine. These general-
purpose color and texture features are those which exhibit the least sensitivity to
illumination and viewpoint variation in a broad range of applications. To overcome
the Bayesian confusion while a large number of test classes are involved, an ALISA
deltaCRC classification method is developed. The classifier selects the trained class
which has a known reclassification distribution histogram of a training image patch
that is most closely matched with the unknown classification distribution of the test
image patch. Preliminary results using the CUReT color texture dataset with test
images not in the training set yields average classification accuracies well above
95% with no significant associated cost in computation time.

38
 6/24/2020

Prepare Documentation (8)

Solicit (Request) Peer Review

• Knowledge offered without formal peer review is unacceptable, at
least to scientists and engineers.
• Acceptance requires an explicit and unbiased confirmation that the
three criteria have been met: reproducibility (repeatable and yields
same result), completeness (clearly and completely disclosed), and
objectivity (no bias, reasonable and valid).

39
 6/24/2020

Thank You

40