07 Evaluation

Management of Information Systems
Prof. Dr. Christof Weinhardt – Ewa Lux
Institute of Information Systems and Marketing (IISM), Karlsruhe Service Research Institute (KSRI)
KIT – University of the federal state Baden-Württemberg

and national research institute of the Helmholtz-association www.kit.edu
economy
utilization
technology
law
Acquisition Storing Transformation Evaluation Commercialization
society
2 Management of Information Systems – Prof. Christof Weinhardt, Ewa Lux Institute of Information Systems and Marketing (IISM)
Outline of the lecture
Information, Measuring/Observation, Experiments, Forecasting,

Acquisition Simulation, Survey, Interviews
Storing Databases, SQL, Pivoting, Semantics and Ontologies
Basics, Filtering
Transformation Regression, Cluster Analysis
Utility Analysis, AHP, Decision Rules, Information Value,

Evaluation Page Rank
Internet Economics, Digital Goods, Network Effects,

Marketing Standardization Networks, Pricing, Bundling
What is the value of an information?
Basics
Criteria to evaluate information
Decisions in uncertainty
Methods and modelling
Search for information
Criteria of information evaluation
Recomm
ender
Meaning, Importance, Relevance system
Use y
Testing b
experts
Reliability
t a ti s tical
S
ds
metho
Integrity, Correctness
Timeliness …
Evaluation models in practice
•Utility analysis
• AHP
2% Andere Methoden •…
Other methods
(Hedonistisches
(Hedonistic Model,
32% Modell, Real Options)
Real Options)
0% Nutzwert-
Utility analysis •End value
5% 20% Analysen
11%
•Capital value
Dynamische Methoden
Dynamical Methods •Present value
der
of investment
71% calculation
Investitionsrechnung •Internet interest
68%
Statische
Statistical Methoden
Methods rate
der
of investment
calculation
Investitionsrechnung •…
Middle class
Mittelstand Big companies
Großunternehmen •Comparison of costs
(Multiple answers or [Bernroider und Koch 2000] •Comparison of revenue
abstentions possible) •Rentability
•Amortisation period
•…
Music is
in the air
Utility analysis
Aim Zj
Legal Capital value Changes in Appearance in Profit per

instructions personell public Alternative
Investment
A1
Alternative Investment
Ai A2
Investment
A3
Utility analysis
Aim Zj

instructions w1 = 6 personell public Alternative
w2 = 3 w3 = 1
Investment
A1
Alternative Investment
Ai A2
Investment
A3
Goal weighting factor:

w1 = 6
w2 = 3
w3 = 1
Utility analysis
Aim Zj

instructions w1 = 6 personell public Alternative
w2 = 3 w3 = 1
30 Mio. +20
Investment satisfying good satisfying
A1 fulfilled 2 * 6 = 12 4 * 3 = 12 2*1=2 26
38Mio. -50
Alternative Investment fulfilled Very good sufficient sufficient
A2 6 * 6 = 36 0*3=0 0*1=0 36
Ai
35 Mio. +40
Investment fulfilled good Very good good
A3 4 * 6 = 24 6 * 3 = 18 4*1=4 46
Point evaluation: Goal weighting factor: But how to find

very good =6 w1 = 6 • Alternatives
good =4 w2 = 3 • Aims
satisfying =2 w3 = 1 • Weighting factor?
sufficient =0
[Hahn 1996, S. 67]
Utility analysis
[Dilbert.com]
AHP – Analytical Hierarchy Process [Saaty 1980]
Set up of evaluation hierarchy Structured process to

Comparison of elements , Reduce the complexity,
Every hierarchy level in relation to (i) Comfortablility of the desicion,
superordinate elements
Exclusion of subjective
Calculation of priorities AHP influencing factors (as far as
(ii) possible),
Consistency (iii) Disclosure of inconsistency in
test the weighting of the criteria
Determination of the evaluation for And liquidation of
every alternative Inconsistency.
Decision
AHP – Analytical Hierarchy Process
Calculation of the relative weighting factor by comparing and
evaluating with the following scale
Value aij
1 Factors i and j have the same importance
3 Factor i is slightly more important than factor j
5 Factor i is more important than factor j
7 Factor i is significantly more important thanfactor j
9 Factor i is definitely more important than factor j
Aim Zj
Capital value Change in personell Appearance in public
Capital value 1 2 6
Change in
Aim Zj personell 1/2 1 3
Appearance in
public 1/6 1/3 1
The weighting factors are between 0 and 1 which means that a
normation is necessary – therefore every comparison value is divided
by the sum of the columns:
Aim Zj
Capital value 1 2 6
Change in
Aim Zj
personell
1/2 1 3
Appearance in
public
1/6 1/3 1
Sum 1,67 3,33 10
The weighting factors are between 0 and 1 which means that a
normation is necessary – therefore every comparison value is divided
by the sum of the columns:
Aim Zj
Capital value Change in personell Apperance in public
Capital value 0,6 0,6 0,6
Change in
Aim Zj
personell
0,3 0,3 0,3
Appearance in
public
0,1 0,1 0,1
The weighting factors are the mean of every row:
Aim Zj
Capital value Change in Appearance in Mean

personell public
Capital value 0,6 0,6 0,6 0,6
Change in
Aim Zj
personell
0,3 0,3 0,3 0,3
Appearance in
public
0,1 0,1 0,1 0,1
Result: The capital value has a relative importance of 60% for the
investment decision, while the change in personell has a relevance of
30% and the appearance in public only 10%.
Consistency testing of the comparison
Max. value
Consisteny index C.I.
Random index R.I.
Consistency value C.R. Rechecking when

[Saaty 2000]
Consistency testing of comparisons
Aim Zj
Capital value 1 2 x
Change in
Aim Zj
personell 1/2 1 3
Appearance in
public 1/x 1/3 1
Practical example: Evaluation of Cloud-Computing for
operational applications (1/3)
Flow chart of application portfolio assessment for cloud
[http://www.ibm.com/developerworks/cloud/library/cl-assessport/]
Illustrative criteria hierarchy
Schematic representation of AHP for evaluating cloud technical fitment
Basics
Criteria to evaluate information
Decisions in uncertainty
Methods and modelling
Basic model of the economical decision
making theory
Object system
Field of decision making
Information Events
Information Decision making

system logic
 battery 
Ziele Zj Ziele Zj
Target system
Gesetzliche Kapitalwert Veränderung Erscheinungs- Nutzgröße je Gesetzliche Kapitalwert Veränderung Erscheinungs- Nutzgröß e je
Vorschriften w1 = 6 der Belegschaft bild in der Alternative Vorschriften w1 = 6 der Belegschaft bild in der Alternative
ergonomics 
w2 = 3 Öffentlichkeit w2 = 3 Öffentlichkeit
w3 = 1 w3 = 1
 
30 Mio. +20 30 Mio. +20
Investition befriedigend gut befriedigend Investition befriedigend gut befriedigend
erfüllt 26 erfüllt 26
A1 2 * 6 = 12 4 * 3 = 12 2* 1=2 A1 2 * 6 = 12 4 * 3 = 12 2*1=2
Alternativen
Ai
Investition
A2
erfüllt
38Mio.
sehr gut
6 * 6 = 36
-50
ausreichend
0*3=0
ausreichend
0* 1=0
36  power  Alternativen
Ai
Investition
A2
erfüllt
38Mio.
sehr gut
6 * 6 = 36
-50
ausreichend
0*3=0
ausreichend
0*1=0
36
Investition
35 Mio.
gut
+40
sehr gut gut
 weight  Subject system Investition
35 Mio.
gut
+40
sehr gut gut
 
erfüllt erfüllt
A3 46 A3 46
4 * 6 = 24 6 * 3 = 18 4* 1=4 4 * 6 = 24 6 * 3 = 18 4*1=4
 extras  [nach Bamberg, Coenenberg, Krapp 2012]
Decision making process
Target recognition Information

Selection process
process process
Kind of Demand of Information

Expectation Selection method
target target acquisition
nach [Laux, 1998]
Decision making in uncertainty
Risk
Current or future states are not known
Probabilities are known
Methodical basis: Demand-use-maximisation
Uncertainty
Probabilities are not known
Methods: Minimax, Maximin etc.
Choice of method also depends on personal risk preferences
Example: Expectancy value vs. Minimax-rule
50% 50% ? ?
E1 E2  E1 E2 min
A1 42 32 37 A1 42 32 32 max
A2 63 21 42 max
A2 63 21 21
Expectancy value Minimax-rule

uncertainty uncertainty
Basics
Methods and Modelling
Evaluation of information systems
Evaluation of IT systems
Basic model for the evaluation of information
Assumption
Information is not for free
To solve a problem or to make a decision you need to find proper information
To evaluate on this problem you can compare costs and benefits
The evaluation of benefits is complicated because the content of information is

unknown before
Draft of the model
Decision making problem:
Alternatives Ai (i = 1,...,n), one of them should be chosen
Environment circumstances Sj (j = 1,..., m) with probabilities w(Sj)
 w S j  = 1
m
j=1
Return Gij depends on the choice of the alternatives Ai and the

environment status Sj
First decision making without information
Example:
Return Matrix of the decision maker before information acquisition
Environment state
Expectation of profits
S1 S2 ... Sm
a priori
m
A1 G11 G12 ... G1m  w(S )  G

i=1
i 1i
Alternatives
A2 G21 G22 ... G2m  w(S )  G

i=1
i 2i
.. .. .. .. ..
. . . . .
m
An Gn1 Gn2 ... Gnm  w(S )  G

i=1
i ni
Example: Sale of an estate

Investor receives 2 offers. One of the estates could be supported by special
amortization possibilities. A proposal was already given to the legislator.
Chances are 50:50.
Amortization happens No amortization Expectation of profits

w(S1) = 0,5 w(S2) = 0,5 a priori
A1 120 90 105
A2 100 100 100
(Risk neutral) decision maker chooses the alternative with the highest
expectation of profits m
E(G)oI = max  w(S j )  Gij
i
j=1
Information acquisition can influence the return matrix

New alternatives
New environment states
Better estimation of the different alternatives
Better estimations of the probabilities of the environment states
Under the condition of risk neutrality the information value can be

evaluated if
„The value of expected returns of a decision after the information acquisition

is compared to the value before the information acquisition.“
Assumption: There is a known number of possible information.

w(Ii | Sj): probability that information result Ii happens, under the
condition that environment state Sj happens
 is assumed to be known
w(Sj | Ii) is the probability that the environment state Sj occurs under the
condition that information result Ii occurs
Determination of w(Sj | Ii) with the BayesTheorem
w(Ii | S j )  w(S j )
w(S j |Ii ) = m
(j=1,...,m) (i=1,...,n)
 w(I | S )  w(S )
j=1
i j j
For every information result Ii an alternative with the highest expectation of

return can be determined
m
E(G |Ii ) = max  w(S j |Ii )  Gij (i=1,...,n)
i
j=1
Not known which I is happening. The expected value of return:
n m  m 
E(G)mI =  w(Ii )  max  w(S j |Ii )  Gij  i  i j
w(I ) = w(I | S )  w(S )
j 
i=1
i
j=1  j=1 
Example: Sale of an Estate (Extension)
Information can lead to two results:
I1: Probability that the amortization possibility will be realized

I2: Probability that the amortization possibility will not be realized
Assumption:
Bayes
w(I1 | S1)=0,7 w(S1 | I1)=0,7778
Theorem
w(I2 | S1)=0,3 w(S1 | I2)=0,2778
w(I1 | S2)=0,2 w(S
w(S1) = 0,52 | I1)=0,2222
w(I2 | S2)=0,8 w(S2) = 0,52 | I2)=0,7222
w(S
Return matrix for I1

Expected return value for I1
w(S1 | I1)=0,7778 w(S2 | I1)=0,2222
A1 120 90 113,3333
A2 100 100 100
Return matrix for I2

Expected return value for I2
w(S1 | I2)=0,2778 w(S2 | I2)=0,7222
A1 120 90 98,18
A2 100 100 100
Both matrices result in an expected outcome with information,

before information is given.
Probability of a information result
w(I1) = 0,7 * 0,5 + 0,2 * 0,5 = 0,45

w(I2) = 0,3 * 0,5 + 0,8 * 0,5 = 0,55
Expected result with information
E(G)mI = 0,45 * 113,3 + 0,55 * 100 = 106
Information value IW
Difference between E(G)mI and E(G)oI
IW = 106 – 105 = 1
The information value is 1 .
Implication:
It is more profitable to acquire information

if the acquisition costs less then 1.
Extension of the basic model
Dynamical information value(1)
Modelling of the delay during information acquisition
Due to that alternatives can drop out
p defines the drop out probability of the alternatives
The evidence is the same for all alternatives
Alternatives can be put in a ranking depending on their return expectation and

the outcome of the information results
E(G<1> | Ii) – The return expectation value of the best (<1>) alternative with the
information result Ii
Dynamical information value (2)
The drop out probability p reduces the expected value of the alternative
with the highest return expectation value: (1-p) * E(G<1>| Ii)
In case that the first alternative drops out ( probability p) only the second
best alternative with expected value p*(1-p)* E(G<2>| Ii) can be realized
Expected return with delay:
E(G | Ii) = (1-p)*E(G<1>| Ii) + p*(1-p)* E(G<2>| Ii) + p²* (1-p)* E(G<3>| Ii)
+ p³ *(1-p)* E(G<4>| Ii) +... =
E (G | I i )  k 1 (1  p ) p k 1 E (G k  | I i )
n
Example: Sale of an Estate (Extension)
The estates can be bought by someone else
p=0,05 results in following expected return values:
 E(G | I1) = 0,95 * 113,3 + 0,05 * 0,95 * 100 = 112,39

 E(G | I2) = 0,95 * 100 + 0,05 * 0,95 * 98,18 = 99,66
w(I1)=0,45 and w(I2)=0,55 result in:

E(G)mI = 0,45 * 112,39 + 0,55 * 99,66 = 105,39
WI(t=1) = 105,39 - 105 = 0,39
Summary
Opportunity costs occur because of delays in the life cycle due to information
aggregation
The opportunity costs result in missed returns due to the drop out of an
alternative
If evaluation takes to much time other market participants can be faster
Literature
Bamberg, Coenenberg, Krapp 2012: Betriebswirtschaftliche Entscheidungslehre.
Vahlen Verlag.
Saaty, T.L. (1980). The Analytic Hierarchy Process. McGraw-Hill, New York.
Saaty, T.L. (2000). Fundamentals of Decision Making and Priority Theory with the Analytic
Hiearchy Process. 2. Auflage, Pittsburgh.
Sassone, P.G. (1987). ACM Transactions on Office Information Systems. Vol. 5,
Cost-Benefit Methodology for Office Systems. 3, S. 273-289.
Benroider, E. und Koch, S.: Entscheidungsfindung bei der Auswahl betriebswirtschaftlicher
Standardsoftware – Ergebnisse einer empirischen Untersuchung in österreichischen
Unternehmen. Wirtschaftsinformatik. 42(4), S. 329-338.
Buxmann, P. (2001). Informationsmanagement in vernetzten Unternehmen. Gabler Verlag.
Laux, H. (1998). Entscheidungstheorie. Springer Verlag.

07 Evaluation

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

07 Evaluation

Uploaded by

Copyright:

Available Formats

Management of Information Systems

Prof. Dr. Christof Weinhardt – Ewa Lux

KIT – University of the federal state Baden-Württemberg

Acquisition Storing Transformation Evaluation Commercialization

Information, Measuring/Observation, Experiments, Forecasting,

Storing Databases, SQL, Pivoting, Semantics and Ontologies

Utility Analysis, AHP, Decision Rules, Information Value,

Internet Economics, Digital Goods, Network Effects,

Legal Capital value Changes in Appearance in Profit per

Legal Capital value Changes in Appearance in Profit per

Goal weighting factor:

Legal Capital value Changes in Appearance in Profit per

Point evaluation: Goal weighting factor: But how to find

Set up of evaluation hierarchy Structured process to

Capital value Change in personell Appearance in public

Capital value Change in personell Appearance in public

Sum 1,67 3,33 10

Capital value Change in personell Apperance in public

Capital value 0,6 0,6 0,6

Capital value Change in Appearance in Mean

Capital value 0,6 0,6 0,6 0,6

Consisteny index C.I.

Random index R.I.

Consistency value C.R. Rechecking when

Capital value Change in personell Appearance in public

Field of decision making

Information Decision making

 extras  [nach Bamberg, Coenenberg, Krapp 2012]

Target recognition Information

Kind of Demand of Information

nach [Laux, 1998]

Choice of method also depends on personal risk preferences

Expectancy value Minimax-rule

To solve a problem or to make a decision you need to find proper information

To evaluate on this problem you can compare costs and benefits

The evaluation of benefits is complicated because the content of information is

Draft of the model

Decision making problem:

Alternatives Ai (i = 1,...,n), one of them should be chosen

Environment circumstances Sj (j = 1,..., m) with probabilities w(Sj)

Return Gij depends on the choice of the alternatives Ai and the

First decision making without information

A1 G11 G12 ... G1m  w(S )  G

A2 G21 G22 ... G2m  w(S )  G

An Gn1 Gn2 ... Gnm  w(S )  G

Example: Sale of an estate

Amortization happens No amortization Expectation of profits

A2 100 100 100

Information acquisition can influence the return matrix

New environment states

Better estimation of the different alternatives

Better estimations of the probabilities of the environment states

Under the condition of risk neutrality the information value can be

„The value of expected returns of a decision after the information acquisition

Assumption: There is a known number of possible information.

For every information result Ii an alternative with the highest expectation of

Not known which I is happening. The expected value of return:

Example: Sale of an Estate (Extension)

Information can lead to two results:

I1: Probability that the amortization possibility will be realized

Return matrix for I1

A2 100 100 100

Return matrix for I2