Professional Documents
Culture Documents
Scientific Research Methodologies and Techniques: Unit 2: Scientific Method Unit 2: Scientific Method
Scientific Research Methodologies and Techniques: Unit 2: Scientific Method Unit 2: Scientific Method
Luis M. Camarinha-Matos
cam@uninova.pt
1. BASE TERMINOLOGY
© L. M. Camarinha-Matos, 2009-2012 2
Terms
© L. M. Camarinha-Matos, 2009-2012 4
2. OVERVIEW OF
RESEARCH METHODS
© L. M. Camarinha-Matos, 2009-2012 5
Classical phases
2 • Background / Observation
3 • Formulate hypothesis
4 • Design experiment
7 • Publish findings
© L. M. Camarinha-Matos, 2009-2012 6
Classical phases ...
© L. M. Camarinha-Matos, 2009-2012 7
Other variants
[Wikipedia]
www.sciencebuddies.org/mentoring/project_scientific_method.shtml
© L. M. Camarinha-Matos, 2009-2012 8
Other variants
Other variants
1. Observe an event.
2. Develop a model (or Ask Fred To Act Dramatically Cool
hypothesis) which
makes a prediction. • A- ask
3. Test the prediction. • F- form a hypothesis
4. Observe the result. • T- test hypothesis
5. Revise the hypothesis. • A- analyze the results
6. Repeat as needed. • D- draw conclusions
7. A successful • C- community
hypothesis becomes a
Scientific Theory.
www.gallimorelearning.com/index_files/Powerpoint%
[Nordgren, 2004]
20for%20website/Science%20PP/scientificmethod.ppt
© L. M. Camarinha-Matos, 2009-2012 10
Other variants
Idea
The Scientific Method Made Easy
Literature review
Problem and
hypotheses
Experiments/ System
analysis (prototype)
http://www.youtube.com/watch?v=zcavPAFiG14
Theory/paper
(new knowledge)
[Mämmelä, 2006]
© L. M. Camarinha-Matos, 2009-2012
In practice !
© L. M. Camarinha-Matos, 2009-2012 12
Errors of experts who did not follow the Scientific Method
© L. M. Camarinha-Matos, 2009-2012 13
3. STEPS OF THE
SCIENTIFIC METHOD
© L. M. Camarinha-Matos, 2009-2012 14
Step 1: Formulate Research question / Problem
© L. M. Camarinha-Matos, 2009-2012 15
“Q1: What are the main components of logistics costs that determine
the logistics and transport network design?
Q2: To what extent are the existing network design and evaluation
models sufficient and how can collaboration be incorporated in the
network design methodology?
Q3: How can economies of scale and scope, present in the newtork,
be taken into account in the network design?
Q4: Is it possible to set boundaries to the development path of the
network, and search for a feasible path instead of searching solely
for a feasible solution? “
© L. M. Camarinha-Matos, 2009-2012
Research question / Problem - Examples
“The main research questions which have guided this
EXAMPLES WITH SOME research work are:
PROBLEMS: Q1: Which are the main characteristics of a collaborative
network and of a collaborative networked environment?
“The main objective of this Q2: How can be assessed the performance of a CN?
Q3: Which are the most relevant conceptual frameworks,
work is to contribute to the
architectures, reference models, independent and industry-
development of elements of a specific initiatives, ICT platforms and their underlying
formal theory for technologies, targeting interoperability in a collaborative
manufacturing systems in networked environment?
order to allow the Q4: Which are the main requirements for interoperability in
establishment of a formal a networked environment?
methodology for the design Q5: How can seamless interoperability be achieved?
Q6: Which are the main differences and similarities between
and analysis of
existing conceptual frameworks?
manufacturing systems” Q7: How can conceptual frameworks be compared, and
which are the criteria to support such an analysis and
evaluation?
It states the “idea” ... but is Q8: Do the conceptual frameworks and the technological
solutions compete or complement each other?
not formulated as a research
Q9: Which is the path to be followed to allow heterogeneous
question ... and sounds and geographically distributed organizations to naturally
vague. inter-operate?
© L. M. Camarinha-Matos, 2009-2012 19
Characteristics of a hypothesis
Should be operationalisable
... i.e. Expressed in terms that can be measured.
© L. M. Camarinha-Matos, 2009-2012 20
Hypothesis example
© L. M. Camarinha-Matos, 2009-2012
“If skin cancer is related to ultraviolet light, then people with a high exposure to UV light will
have a higher frequency of skin cancer.”
“If temperature affects leaf color change, then exposing the plant to low temperatures will result
in changes in leaf color.”
© L. M. Camarinha-Matos, 2009-2012 22
Step 4: Design experiment
Doing it !
© L. M. Camarinha-Matos, 2009-2012 24
Test hypothesis – perform experiments
© L. M. Camarinha-Matos, 2009-2012
Discussion regarding
Literature
Research objectives
Research questions.
© L. M. Camarinha-Matos, 2009-2012 26
Interpret / Analyze results
Intermediate results
Conferences “Publish or perish !”
Collect feedback
Write dissertation
© L. M. Camarinha-Matos, 2009-2012 28
Attributes of a good thesis
Is it a good thesis ?
© L. M. Camarinha-Matos, 2009-2012 30
Proof of concept
Presentation languages
© L. M. Camarinha-Matos, 2009-2012 32
Role of simulation
© L. M. Camarinha-Matos, 2009-2012 33
But there are many cases in which such experimental procedures can
lead to interesting research results even in computer science. “
[Feitelson, 2006]
This situation quite frequently affects the “policies” of research funding agencies !
© L. M. Camarinha-Matos, 2009-2012
Fallacy #1: Traditional scientific method isn’t applicable Fallacy #2: Current levels of experimentation are enough
• In a study of CS papers requiring empirical backup,
• Subject of inquiry is information unlike traditional 40-50% had none
sciences which study matter or energy • Compared to <15% in non-CS papers
• Example: • The youth of CS as a discipline is not sufficient
– Object-oriented programming, is it genuinely better? justification
© L. M. Camarinha-Matos, 2009-2012
ECS – Fallacies ... [Gain, 2008]
Fallacy #3: Experiments cost too much Fallacy #4: Demonstration will suffice
• Experiments can be expensive, but:
– Often cheaper than the alternative • Demos allow proof of concept and
– The cost may be worthwhile for important illustrate potential
questions (general relativity) • But they cannot provide solid evidence
– Explore cheaper options (benchmarking)
© L. M. Camarinha-Matos, 2009-2012
Fallacy #5: There’s too much noise in the way Fallacy #6: Experimentation will slow progress
© L. M. Camarinha-Matos, 2009-2012
ECS – Fallacies ... [Gain, 2008]
Fallacy #7: Technology changes too fast Fallacy #8: There are substitutes
• “The rate of change in computing is so
great that by the time results are
• Theory
confirmed they may no longer be of any
– Can be contradicted in practice by
relevance” [Mudge]
incorrect simplifying assumptions
• Look to fundamental long term problems
• Intuition
rather
– Fails in the face of counterintuitive
results
– E.g., productivity is NOT
necessarily improved by
typechecking
• Experts
– Science must always be backed up
by evidence
– E.g., claims about cold fusion
© L. M. Camarinha-Matos, 2009-2012
Many proposals are rejected because a reviewer said: "That's already been
done." Many others have never been submitted because the proposer
feared such a response.
© L. M. Camarinha-Matos, 2009-2012
ECS – Misconceptions ... [Denning, 1980]
But the whole point of science is to discover which ideas are important.
Experiments are essential:
to understand ideas and
to convince others of their value.
Once an idea is assimilated by the community, the experiments behind it
may be forgotten.
(We use the word "hacking," rather than "tinkering," in our field.)
Risks:
- that funds being allocated for experimental research will be used merely for hacking.
- of discouraging conceptual work. Tinkering is no substitute for thinking.
© L. M. Camarinha-Matos, 2009-2012
© L. M. Camarinha-Matos, 2009-2012
4. ENGINEERING RESEARCH
© L. M. Camarinha-Matos, 2009-2012 44
Scientist vs Engineer
© L. M. Camarinha-Matos, 2009-2012 45
[Hong 2005]
© L. M. Camarinha-Matos, 2009-2012 46
Research methods in engineering ...
[Muller, 2008]
http://www.gaudisite.nl/SEresearchValidationSlides.pdf
© L. M. Camarinha-Matos, 2009-2012 47
© L. M. Camarinha-Matos, 2009-2012 49
Feitelson, D.G. (2006). Experimental Computer Science: The Need for a Cultural Change.
http://www.cs.huji.ac.il/~feit/papers/exp05.pdf
McNeese, M. D. (2003). New visions of human-computer interaction: Making affect compute. International Journal of
Human-Computer Studies, 59 (1), 33-53.
Pedersen, K.; Emblemsvåg, J.; Allen, J.; Mistree, F. (2000). THE ‘VALIDATION SQUARE’ - VALIDATING DESIGN
METHODS, 2000 ASME Design Theory and Methodology Conference, http://dbd.eng.buffalo.edu/9th_meet/9-vsq.pdf
Tichy, W. F. (1998). Should Computer Scientists Experiment More? Computer (IEEE), Volume 31 , Issue 5
© L. M. Camarinha-Matos, 2009-2012 50