Trust, Evidence and Reports

Problem areas …
Who sponsored the research?
Did this lead to a bias in the results?

Image from http://thesocietypages.org/socimages/2010/11/27/vintage-cigarette-ads-reach-for-a-lucky-instead-of-a-sweet/ and

http://transform-drugs.blogspot.com/2008/10/pseudoscience-tobacco-advertsing-from.html
 Problem areas …

• Problems of use
– CFCs
– Tobacco
– Storage of incorrect data

• Problems of abuse
– Nuclear fission
– Diamorphine
– The Internet
 What is Evidence
• Things we have learnt ourselves
• Things other people have learnt
• Examples from which we can generalise
• It is not an opinion without reasoning as to
why this is true and reasonable
 What information supports the
evidence
• The context in which it was obtained
• A clear definition of what the data was and
how it was
– Collected
– Analysed
• Any sources of threats to the validity of the
conclusions it draws
• Something that gives us confidence that the
work has been independently reviewed
 For Selecting a component

Evidence

Assertive Inductive

Certified Component Constituent Components Component run in same Strongly Weakly

internal/third party All certified System/operational environment

Certified Process Run in similar Produced by Self Certified Constituent Components Run in dis-similar
operational profile Compliant Process Component from compliant process System/operation

CMM ISO Other E.G. IEC61508

For Selecting a paper
Quantitaive
Results
Revieved in
Journal
Data Available
As above but in a
Assertive
different context

As above but
only qualitative
Evidence

Produced by well
Strongly respected
authors
Inductive
Opinion and no
Weakly
past history
 Creating a Trust Scale
• Based on the context you need to be able to
describe to anyone using your results why you
gave importance to some information over
others.
• Simplest form is to use the concepts from the
previous two slides to create a scale and then
for each piece of research place it on such a
scale.
 Building the report

Define the Search for Evaluate and

Report Results
problem space Evidence Rank Evidence
 Why is it relevant?
• You are faced with something new, for your
firm or meeting a client need
• You need to demonstrate that your
recommendation is the right one
 Widely held beliefs vs evidence: in the
medical domain
Vitamin C & the common cold
– It was commonly believed that very large doses of
Vitamin C prevented the common cold - belief derived
from a non-systematic review of literature by Nobel
Laureate Linus Pauling (1986)

– An exhaustive systematic review of the literature by

Knipschild & colleagues concluded that even mega
doses of Vitamin C cannot prevent a cold

– Pauling’s review missed 5 of the ‘top 15’ published

studies (even though he was an expert)

School of Computing and

Mathematics, Keele University
 Widely held beliefs vs evidence: software
projects
IT project failure rates

– In 1994, The Standish Group reported that average cost overrun of

IT projects was ~ 189%. Group claims that their surveys are the
most widely-quoted statistics in the IT industry – this figure (189%)
is widely quoted in governmental reports.

– Group unwilling to disclose their research method

– A large scale survey in 2003 (using interviews) suggests average

overruns are ~ 40% - details of method were published

– A comparison with previous published surveys (again with full

details of methods used) suggests little change over last 10-20
years.

School of Computing and

Mathematics, Keele University
 Evidence-Based Practice
• The procedure is:
1. Convert need for information (about a technique, method, etc.)
into an answerable question
2. Find the best evidence with which to answer the question
3. Critically appraise the evidence for its validity (closeness to the
truth), its impact (size of effect) and its applicability (usefulness)
4. Integrate the critical appraisal with domain expertise and with
stakeholders’ values and circumstances
5. Evaluate the effectiveness and efficiency of the process in steps 1-
-4 and seek ways to improve them
• Steps 1--3 are essentially the process of systematic
literature review and indeed also most other research

13 CSC-40037
 Conclusions
• The talk has highlighted
– Trust
– Evidence
– Processes
• Need to apply these if we are to have a sound
basis for recommendations