TOPIC 9: EVALUATION

9.1 INTRODUCTION
9.2 SORTING
9.3 EVALUATION METHODS
9.3.1 Advantage-Disadvantage Tables
9.3.2 PMI (Plus/Minus/Interesting)
9.3.3 Castle Technique
9.3.4 Sticking Dots
9.3.5 Creative Evaluation
9.3.6 Force-Field Analysis
9.3.7 Weighting Systems
9.3.8 The Process of Choosing
9.3.9 Qualitative Evaluation: Reverse Brainstorming
9.3.10 Financial/mathematical Evaluations
9.3.11 Pay-Off Tables
9.3.12 Decision Trees
TOPIC 9: EVALUATION
9.1 INTRODUCTION
Methods of evaluation range from simple checklists to
complex weighted scoring systems. First, however, we look
at sorting methods and then we go on to look at
evaluation methods. Many of the ideation methods we
have examined in the previous chapters produce a large
quantity of ideas.

Before we can evaluate these ideas we need to sort them

into categories or themes. This facilitates the process of
making comparisons and evaluations.

Note: Capital Budgeting is a Financial evaluation method:

Pay-Back Period, NPV, IRR, MIRR etc..
9.2 SORTING
An individual was asked to brainstorm ideas for
the use of aluminium foil. These were first
written down on a sheet of paper. Ideas were
then examined for those that looked interesting
(highlighting), and the remainder were
discarded. The remaining ideas were then
grouped according to different hotspots or
themes. An idea could be appended to more
than one hotspot or theme (indicated by the
entries in brackets).
1 Cooking
1.1 covering roast chicken
1.2 wrapping baked potato
1.3 covering food for storage
1.4 milk-bottle tops
1.5 wrapping food for storage w/o fridge
1.6 keeping food warm
1.7 lining a grill pan
1.8 wrapping sandwiches to keep fresh
1.9 lining cake storage tins
1.10 (cooking food on campfire)
1.11 reflecting heat source
1.12 keep food cold
2 Fun
2.1 babys rattle
2.2 screw up to make ball
2.3 making a cats toy
2.4 stepping stones
2.5 flapping fish game
2.6 (water in Brownie pool)
2.7 (making childrens jewellery)
3 Practical
3.1 making a cup
3.2 making a plate
3.3 smooth to make a mirror
3.4 lining wall behind radiator
3.5 glue/paint dish
3.6 temporary curtains
3.7 (wrapping buttonhole stem)
3.8 (cutting strip to make bookmark)
3.9 (make milk-bottle tops)
4 Camping/out of doors
4.1 reflecting sunlight for SOS
4.2 lining under sleeping-bag
4.3 wrapping food on campfire
4.4 keeping matches dry
4.5 (keep food warm)
5 Decorative
5.1 wrapping buttonhole stem
5.2 Christmas decorations
5.3 use in collage
5.4 water in Brownie pool
5.5 making childrens jewellery
5.6 cutting strip to make bookmark
5.7 gift-wrap 25th-anniversary presents
5.8 (smooth to make mirror)
9.3 EVALUATION METHODS
The following methods may be used to evaluate all the ideas that have been generated along
a particular theme or grouping, or to evaluate/eliminate particular themes or groupings.

9.3.1 ADVANTAGEDISADVANTAGE TABLES

Perhaps the simplest method of evaluating ideas makes use of tables which permit the
comparison of the advantages and disadvantages of various ideas. For example, suppose
there are two ideas about how we should reorganize the office method A and method B.
First we list the criteria against which we want to compare and evaluate the ideas. The same
criteria are used for both of the options, and space is left to indicate whether the idea was
rated as having predominantly advantages or disadvantages when considered against those
criteria.
The technique is useful as a rough evaluation tool, and in the example in Table 10.1 it will be
noted that method B seems to have the better rating. However, except for possible use as a
preliminary screening device, this approach has too many limitations. Its main weakness is of
course that it assumes that all the criteria carry equal weight and that it is the overall score
that is important (it overlooks the fact that some of the criteria may be critical i.e. they must
be satisfied).
More elaborate screening methods have been suggested by Hamilton (1974). The methods
involve culling ideas which fail to satisfy key criteria, and rating and scoring ideas against
desirable criteria.
9.3.2 PMI (Plus/Minus/Interesting)
9.3.3 CASTLE TECHNIQUE
9.3.4 STICKING DOTS
This is a useful method for a small group of individuals to employ
when trying to make an evaluation (see Geschka, 1979). Each group
member has a fixed number of votes (usually these are physically
represented in the form of self-adhesive colored dots).Group
members can then vote in any way they wish. The procedure is as
follows:
1) Each member of the group has a card on which there are
adhesive dots of a color unique to that person. The number of dots
corresponds to 10 per cent of the total number of ideas being
considered.
2) Ideas are listed on cards or flip charts and pinned to a wallboard.
3) Ideas are evaluated by group members sticking dots next to the
ideas they prefer. Dots can be allocated in any manner. An
individual can place all his or her allocated dots against one idea if
he or she so desires.
4) The ideas receiving the largest number of votes are selected for
further analysis or implementation.
9.3.5 CREATIVE EVALUATION
This also is a method which is useful for dealing with a
large number of ideas. It attempts to present ideas in a
format that will reduce the amount of time required for
evaluation. All ideas are evaluated in terms of time and
financial requirements (Moore, 1962).The procedure is
as follows:
1) List the ideas.
2) Categorize the ideas into simple, hard and difficult.
Note that simple ideas are those which can be put into
action with a minimum of expenditure of time and
money. Hard ideas require more expenditure, while
difficult ideas require the most expenditure.
Like the advantagesdisadvantages method, this
approach is most suitable for a cursory examination of a
large number of ideas.
9.3.6 FORCE-FIELD ANALYSIS
This is a method used to get a whole view of all the forces for or against a plan so that a
decision can be made which takes into account all interests. In effect this is a specialized
method of weighing pros and cons. Where a plan has been decided on, force-field analysis
allows you to look at all the forces for or against the plan. It helps you to plan or reduce the
impact of the opposing forces, and strengthen and reinforce the supporting forces. To carry
out a force-field analysis, take the following steps:
List all forces for change in one column, and all forces against change in another column.
Assign a score to each force, from 1 (weak) to 5 (strong).
Draw a diagram showing the forces for and against, and the size of the forces (see Figure
10.8).
Once you have carried out an analysis, you can decide on the viability of the project.
Where you have decided to carry out a project, it can help you to analyze how you
can push through a project that may be in difficulty. Here you have two choices:
to reduce the strength of the forces opposing a project
to increase the forces pushing a project
Often the most elegant solution is the first: just trying to force change through may
cause its own problems (e.g. staff can be annoyed into active opposition to a plan
instead of merely not welcoming it).
If you were faced with the task of pushing through the project in the example above,
the analysis might suggest a number of points:
By looking for a strategic alliance, loss of management control could be reduced
(reduce loss of management control by 2).
Coping with uncertainty is necessary for business survival (new force in favor, +2).
More work will mean a more productive workforce (new force, +1).
More sales will increase morale of sales force (new force, +1).
More profit will increase satisfaction of shareholders (new force, +1).
These changes swing the balance from 9:9 (neither for nor against the plan) to 14:7
(in favor of the plan).
Force-field analysis is an effective method of getting a picture of all the forces for
and against a plan. It helps you to weigh the importance of these factors and to
assess whether a plan is worth pursuing. Where you have decided to proceed with
a plan, carrying out a force-field analysis helps you identify changes that might be
made to improve the plan.
9.3.7 WEIGHTING SYSTEMS
The first step is to generate evaluative criteria. Next, one assigns different weighting for each criterion
reflecting its importance with respect to the problem under consideration. Next, each idea is rated on
the degree to which it satisfies each criterion. The lower the number, the less the criterion is satisfied.
Finally, one multiplies the criteria-importance rating one has assigned to obtain the rated score.
Weighted scores are then summed to obtain an overall weighted score for each idea or theme. An
example of a weighted decision matrix is shown in Table 10.3, in which case weights and scores are out
of a maximum of 5 where 5 is considered very important or high-scoring and 1 is relatively unimportant
or low-scoring.

Generally, ideas with the highest overall scores are considered the best. However, one must take
account of the fact that there may be some critical criteria that have to be satisfied, and even if an
idea has the best score of those available it may still not be considered adoptable simply because it
has failed to satisfy the requirements of particular criteria. In addition one might specify that an idea
must obtain a certain overall score before it can be considered adoptable. If the best idea available
does not satisfy this criteria, it may not be adopted.
9.3.8 THE PROCESS OF CHOOSING
As can be seen from the foregoing, when exercising choice it is usual to have a set of
alternatives and a set of evaluation criteria. Evaluating a list of alternatives involves
measuring, trading off or scoring them in terms of the specified criteria and
determining the relative importance of the criteria. This may involve several
complexities:
multiple criteria and multiple alternatives;
a large number of criteria and sub-criteria;
criteria which are not all equally important to the decision maker;
some criteria may be qualitative while other criteria are quantitative.
Some of the typical suggestions for analyzing such data involve:
discussing the pros and cons of each alternative;
analyzing the costs and benefits, or weaknesses and strengths, of each alternative;
ensuring the effective utilization of financial and other quantitative information in
evaluating alternatives;
sifting back through evidence provided in situation analysis to help reach a
conclusion;
assessing whether a chosen alternative solves a problem without creating new
problems;
justifying why an alternative has been selected and outlining why others may have
been rejected.
The general approach to adopt at the choice phase is as follows:
1) Use quantitative analysis on objective data:
(a) acquire data on the anticipated outcomes of each alternative with respect to each criterion;
(b) perform descriptive or experimental research to measure each alternatives performance on the
criteria;
(c) consider whether quantitative tools such as simulation and optimization methods might be usefully
applied;
(d) conduct a sensitivity analysis with the quantitative data.
2) Use qualitative analysis with subjective criteria.
(a) conduct a pro/con analysis;
(b) make use of expert judgement in dealing with the evaluation of alternatives with respect to the
qualitative criteria.
3) Merge quantitative and qualitative analysis.
4) Perform a synthesis of the data generated to rank alternatives.
5) Conduct a sensitivity analysis to show how sensitive the final priorities for decision
alternatives are to possible changes in scores assigned to alternatives and weights assigned to
decision criteria.
6) Check analytical results against intuition. If they agree, you can be more assured that the
decision is a good one. If they do not agree, find out why this is the case. Check, in particular,
to ascertain that all relevant factors have been included in the decision analysis and that they
have appropriate scores or weights.
The merging and evaluating of quantitative and qualitative data presents considerable
choices. Moreover, synthesis and sensitivity analysis of such data presents a daunting task for
all but relatively simple decisions. One sophisticated tool that can be used to get to grips with
this process is the Analytic Hierarchy Process {AHP} (Saaty, 1980).
The analytic hierarchy process (AHP) enables the decision makers to set priorities and make
choices on the basis of their objectives and knowledge and experience in a way that is
consistent with their intuitive thought process. It has substantial theoretical and empirical
support, overcomes problems associated with pro/con analysis and the weights and scores
technique by using a hierarchical structure of the decision problem, pair-wise relative
comparison of the elements in the hierarchy and a series of redundant judgements. The
approach reduces error and encourages consistency in judgements. The use of redundancy
allows accurate priorities to be derived from qualitative judgements, even though the wording
may not be very precise. This means that words can be used to compare qualitative factors
and derive ratio scale priorities that can be combined with quantitative factors.
Expert Choice helps a decision maker examine and resolve problems involving multiple
evaluation criteria. The software uses the AHP methodology to model a decision problem
and evaluate the relative desirability of alternatives.
9.3.9 QUALITATIVE EVALUATION: REVERSE BRAINSTORMING

The technique was developed at the Hotpoint company (Whiting, 1958) as a

group method for discussing all possible weaknesses of an idea, or what
might go wrong with an idea when it is implemented. It is almost identical to
classical brainstorming except that criticisms rather than ideas are
generated. (Devils Advocate: Roman Catholic sainthood process?)
Imagine the problem being how to counteract declining sales and that the
following potential ideas for solutions were generated by classical
brainstorming or some other ideation method:
new advertising strategy change or improve packaging
offer discounts find new markets
door-to-door sales
The first step in reverse brainstorming is to suggest criticisms for the first of
these ideas new advertising strategy. Criticisms developed might be:
too expensive
unable to target the specific areas required
After exhausting criticisms for the first idea the group begins criticizing the
second idea, and the process continues until all the ideas have been
criticized.
Using classical brainstorming the group then re-examines the ideas to generate possible
solutions for each weakness that has been identified. For example, in the case of the second
idea, offer discounts, the criticisms might be that people might perceive the quality of the
product not to be good as a result of offering discounts. In the case of door-to-door sales, it
might be the fact that unacceptable training and added costs will be incurred because of the
need to employ more sales staff. Other criticisms will no doubt be found for the other ideas.
As far as solutions to these criticisms are concerned, it may not be felt that there are any in,
say, the case of a new advertising strategy; however, in the case of door-to-door sales it
may be felt possible to employ part-time workers in order to lower the cost.
The idea that possesses the lowest number of weaknesses, and that will be most likely to
solve the problem, is usually selected for implementation. Of course, one does also have to
bear in mind the comparative seriousness of any unresolved criticisms.
Example

PROBLEM: getting people to have a positive attitude towards adopting new ways of working.
IDEAS:
1) Rewards associated with adopting new methods.
2) Firing those who do not co-operate and hiring new staff.
3) Training people and giving them the right kind of skills to do the new tasks.
CRITICISMS:
1 (a) May be too costly.
(b) May not believe they will receive rewards, or seen as further manipulation by management.
2 (a) Will cause even more hostility and resentment.
(b) Difficult to identify best method of recruitment may still hire inappropriate people.
(c) Effort and time need to be spent on recruitment and interviews.
3 (a) Training requires additional time and cost.
(b) Not possible to provide training for every situation.
SOLUTIONS TO WEAKNESSES:
1 (a) Link the new methods with productivity increases.
(b) Provide written agreements to show commitment.
2 (a) No solution.
(b) Agree that all new appointees be on probation for a fixed period.
(c) Hire recruitment consultants.
3 (a) Provide training on the job.
(b) As (a), plus make sure first line managers can provide proper guidance and support to workers.

Although both ideas 1 and 3 seem to have resolved all the difficulties associated with them, 3 might well be the
preferable alternative. This is because the problem at the core of the matter lies in a decrease in productivity
which is incurred as each new method is adopted.
9.3.10 FINANCIAL/MATHEMATICAL EVALUATIONS

Mathematics can be used effectively to aid/evaluate

possible solutions. In essence one is interested in
assessing the potential pay-off of problem
solutions. One way to achieve this is to consider the
probabilities of certain pay-offs in the light of
previous experience. Of course, this method can
only be used where there is enough experience to
calculate the odds with some degree of accuracy;
and, of course, probabilities of this kind are of no
use in a case where it has never been necessary to
consider a similar instance before.
9.3.11 PAY-OFF TABLES
Alternative solutions to a problem can be laid out in a pay-off table (Table 10.4). Three
probabilities are shown with respect to both costs and benefits associated with the ideas
(.25, .50 and .25). Thus, for option A, for example, it is estimated that there is a 25 per cent
chance the costs will be 2,500, a 50 per cent chance it will be 3,000 and a further 25 per
cent chance that it will be 4,000. Benefits are calculated in a similar way.
Using the expected pay-off concept we can see that option A appears to have the best
expected pay-off.
9.3.12 DECISION TREES
While pay-off tables are adequate for most purposes, decision trees enable one to see what
effect a decision made at a particular moment will have on the options to be faced in the future.
For example, suppose that a firm is operating at full capacity and demand for its products is rising.
A 20 per cent rise in demand is expected, and this can be met by adopting one of two new
methods of working (A and B).The net cash flow in the case of a 20 per cent rise in demand will be
$460,000 if new method A is used, or $440,000 if the extra production is obtained by method B.
However, there is a body of opinion in the firm that sales will not rise and that they will actually
decline by 5 per cent, even though the rise is more likely. There is general agreement that there is
a 60 per cent probability of a rise in sales of 20 per cent and a 40 per cent probability of a 5 per
cent drop in sales.
If there were a 5 per cent sales-drop, the cash flow would be
$340,000 if the company were to use method A and $380,000 if it
were to opt for method B. The alternatives are presented in the
decision tree (Figure 10.11).
In order to take account of the probabilities, one has to make the
following calculations:

Method A: $460,000x0.6 + $340,000x0.4 = $412,000

Method B: $440,000x0.6 + $380,000x0.4 = $416,000

Method B seems preferable. However, one does have to take

account of future years sales; and, assuming predictions can be
made for a further couple of years, this would introduce another
two decision points and extend the branches of the decision tree.
Care, however, has to be exercised in terms of extending a decision
tree into the future as uncertainty increases the further ahead one
looks.