Research Process

• Steps in research
• Data sets preparation
• Experimental research
• Performance evaluation

1
 Steps in research
• Research is an extremely cyclic process.
– Later stages might necessitate a review of earlier work.
• This isn’t a weakness of the
process but is part of the built-in
error correction mechanisms.
• Because of the cyclic nature of
research, sometimes it might be
difficult to determine where to
start and when to stop.
 Here are some of the steps to
take into account in choosing
starting & stopping points during
your research work.
2
 Step 1: A Problem Raised for
Research
• A question/problem/issue occurs to or is posed to the
researcher for which that researcher has no answer.
– This doesn’t mean that someone else some where doesn’t
already have an answer.
• The question needs to be converted to an appropriate
problem statement as documented in a research
proposal.
– It should show clearly the gap between the expected and
actual situations
– Whether any research has been done to solve it and further
research direction recommended
3
 Step 2: Formulate research questions
• At the end of problem statement, the researcher is
expected to :
– formulate research questions that should be answered in the
course of the research work.
• This is at best a temporary questions to be answered since
there is as yet no evidence to support either the acceptance
or rejection of the research questions.
• Examples for research questions:
– What attributes will be properly predicting the status of …
– Which algorithm is suitable for ….
– How do we select the “optimal” parameter(s) or model for a
given problem?
– To what extent the performance will be improved by using …4
 Step 3: Literature Review
• The available literature is reviewed to determine if
there is already a solution to the problem.
– Existing solutions do not always explain new observations.
– The existing solution might require some revision or even be
discarded.

• It’s possible that the literature review has yielded a

solution to the proposed problem.
– This means that you haven’t really done research.

• On the other hand, if the literature review turns up

nothing, then additional research activities are justified.
5
 Step 4: System development
• Design the system architecture
– It is preferred to come up with a generic architecture
which may or may not be answered by the study

• Identify and Select two or more possible algorithms

for each module/task such that the best one is
selected through experiment for implementation
• Select suitable tools (open source software) for
training;
– or select programming languages for writing source code

6
 Step 5: Dataset preparation
• The researcher now begins to gather data relating to the
research problem.
– The means of data acquisition will often change based on the
type of the research problem.
• Before using the data we have to make sure the quality
of data?
– Most of the data in the real world are: incomplete,
Inconsistent, Noisy, redundant, …
• A well-accepted multidimensional data quality measures
are the following:
– Accuracy
– Completeness
– Consistency
– Timeliness
– Believability
– Interpretability 7
 Split Data into a training set & a test set
– Training dataset
• is used for Model selection (to decide about the learning
parameters and weights).
• A training set is a set of problem instances (described as a
set of properties and their values), together with a
classification of the instance.

– Test dataset (which is never seen during the training

phase)
• is used to estimate the performance of the model created
during the training phase
• Test set evaluates the accuracy of the model/hypothesis in
predicting the categorization of unseen examples.
• A set of instances and their classifications used to test the
accuracy of a learned hypothesis.
8
The Learning Scenario: classification

Results Known
+ Model
Data + Training set Builder
-
-
+

Model Builder

+
- Evaluation
+
Test Set Final Model -
9
 Methods to Split the data into training set
& test set
• There are various ways in which to separate the data into
training and test sets.
– The holdout method
• Split dataset into two groups with some proportion
• In problems where we have a sparse dataset we may
not be able to afford the “luxury” of setting aside a
portion of the dataset for testing
– K-fold Cross-validation
• Each split randomly selects a (fixed) number of examples without
replacement
• The advantage of K-Fold Cross validation is that all the examples in
the dataset are eventually used for both training and testing
– The bootstrap
• Each split randomly selects a (fixed) number of examples with
replacement
10
 Step 6: Experimentation
• The dataset that were gathered and prepared in the
previous step are used for experimenting algorithms/
techniques selected for system development.

• Experimental setup
– Decide about how many experiments needs to be
conducted, with what parameters, weights, algorithms,
datasets, etc.
• Record the result using performance measurement
techniques selected
– The performance measurement various from system to
system or from technique to technique
11
 Step 7: Analysis & Interpretation
• The researcher analyze using statistical techniques and
interprets the newly analyzed data and suggests a conclusion.
– This can be one of the difficult task in research, and of course
the crucial one to come up with research findings.
– Keep in mind that data analysis that suggests a correlation
between two variables can’t automatically be interpreted as
suggesting causality between those variables.
• Depending on whether the research questions are answered
or not, the researcher may be forced to cycle back to an
earlier step in the process and begin again with a new
research questions formulation.
– This is one of the self-correcting mechanisms associated with
the scientific method.
12
Experimental
Research

13
 Mode of Enquiry: Observational or
Experimental?
Observational
or non-experimental Experimental

• In an observational study…
– The aim is to gather data or information about the world as it
is.
– So you hope the act of studying doesn't substantially modify
the thing you are interested in.
• In an Experimental study…
– You do something to the world and see what happens.
– You gather data or information almost always before and after
the intervention, use it for training and testing, and then look
for changes.
14
 Experimental Research in Computer
Science
• Experimentation in computer science aims to answer
questions like :
– Is program A really more efficient than program B, in
practice? (Note that this is different from theoretical
questions about asymptotic bounds on runtime and
space that can sometimes be answered using static
analysis.)
– Does A's runtime vary more widely than B's on different
inputs?
– Is program A really more accurate than program B, when
applied to some task?
– What are the best parameters for compiling or running a
particular program?
15
 Why should we experiment?
• Experiments helps to measure
the performance of system,
tools, algorithms;
– though it cannot prove anything
with absolute certainty
• In general, they are good for:
– Reducing uncertainty about
theories, models, and tools
– Leading to new insights and
whole new areas of
investigation
– Quickly eliminating fruitless
approaches
16
 Challenges: Technology changes too
fast
• “The rate of change in
computing is so great
that by the time
results are confirmed
they may no longer be
of any relevance”

• Better look to
fundamental long term
problems
17
How do you do Experimental Research?
• Once a topic is selected and problem statement is
well defined, Experimentation in Computer Sciences
pass through the following research process:
– Prepare dataset
• Split into training and testing set
– Select proper tools
• Write source code, if no tools available that is suitable
for the experiment
– Designing an experimental setup
• Specification needs to be complete and explicit.
– Run different experiments
• Record results and take notes of the challenges
– Measure performance
• Analyze and interpret results
18
How do you do Experimental Research?
• Designing an Experimental setup
– How many experiments needs to be conducted? It
depends on:
• The number of algorithms used in the study
• Types of dataset collected & splitting techniques used

– Requirements:
• Controlled – define dependent and independent
variables, and other factors must be kept constant
• Quantitative - provide numbers (in percentage) to show
the performance of the system
• Coverage - are tests representative of the full range to
answer the research questions

19
How do you do Experimental Research?
• Running different experiments
– Keep a note book!
• Record results, parameter/threshold adjustments,
error fixed
• Most importantly, keep challenges encountered
during implementation of the system and testing its
performance
– Keep even apparently those seems insignificant
aspects of challenges you encounter during
implementation and testing
– Data analysis requires (some/lots) of statistics
• Try to register all the experimental results and
reasoning for achieving such results
20
How do you do Experimental Research?

Datasets
estimation; regression; learning; training

Model
classification; decision
pattern classification
machine learning
statistical inference Evaluation
...
21
How do you do Experimental Research?
• Reporting and Analyzing Results
– Analyze the performance achieved:
• using performance measures such as accuracy, recall,
precision, etc.
– Interpret
• Relate performance registered with the algorithms vs.
datasets used
• Show clearly the strength and weakness of the study,
algorithms selected
– Be Careful:
• Better to admit to flaws in your methodology
• Don’t generalize without adequate support
– Report everything:
• Procedures followed, results achieved and conclusions
• So that others can replicate the experiment, and build on
your conclusions 22
 Performance evaluation
• Confusion matrix
• performance
Measurements

23
 Confusion matrix
Condition as determined by
Gold standard
True False
→ Positive
True True positive False positive predictive value
Test or Precision
outcome
False → Negative
False True negative
negative predictive value
↓
↓
Specificity (or its
Sensitivity or Accuracy
complement,
recall
Fall-Out)
24
 Performance Indicators
•Test efficiency
– Algorithm analysis in terms of running time and memory space
requirement
• Accuracy
– is a measure of how well a system correctly identifies. That is,
the accuracy is the proportion of true results (both true positives
and true negatives) in the population.

• Precision (or positive predictive value)

– is defined as the proportion of the true positives against all the
positive results (both true positives and false positives)

• Recall
– precision is the fraction of retrieved instances that are
relevant/correctly classified, while recall is the fraction of
relevant/correct instances that are retrieved.
25
 Performance Indicators
• Sensitivity and specificity
–Sensitivity (also called recall rate in some fields) measures the
proportion of actual positives which are correctly identified as
such (e.g. the percentage of sick people who are correctly
identified as having the condition).
–Specificity measures the proportion of negatives which are
correctly identified (e.g. the percentage of healthy people who are
correctly identified as not having the condition).
• Another useful performance measure is the balanced accuracy
which avoids inflated performance estimates on imbalanced
datasets. It is defined as the arithmetic mean of sensitivity and
specificity, or the average accuracy obtained on either class

26
 Performance Indicators

Sensitivity = True Positive

X 100%
True Positive + False Negative

Specificity = True Negative

X 100%
True Negative + False Positive

Precision = True Positive X 100%

True Positive + False Positive
Recall = True Positive X 100%
True Positive + False Negative
27