Professional Documents
Culture Documents
Abstract
The recent upsurge in the need for monitoring and evaluation of development
interventions by governments and non-governmental organizations around the
world has sparked a debate among monitoring and evaluation practitioners and
“experts” on the need to professionalize monitoring and evaluation. This has
resulted in a number of proposals being put forward by some practitioners and
“experts” about how to do it. If one looks at some fields of knowledge that are
now professionalised, one finds that these fields of knowledge got to the level of
professionalization by first developing a central theory which came about after
studies of certain behavioural patterns overtime. In the monitoring and
evaluation field there is not enough evidence that this is the approach among
those that are involved in the field’s professionalization effort.
The following key things need to be pulled out of the foregoing two paragraphs,
one, a theory is borne out of studying a certain phenomenon and it is a model of
a real life situation and it has explanatory and predictability power about the
recurrence of certain aspects of the phenomenon. Two, in order to
professionalize any human activity there has to be at least a universal majority
consensus about validity of certain human activity behavioural patterns’
occurrence and common meanings of terms that are used by those concerned.
Going back to the above example to substantiate the point further, majority of
economists are on a common denominator that economics is a study of human
behaviour in the process of production, distribution and consumption of goods
and services, making the demand theory one tool that can be used to explain and
predict human behaviour in the consumption activity anywhere wherever it
occurs. Analogously therefore, for one to develop a theory of monitoring and
evaluation, one has to understand the necessary human behavioural patterns in
the human activities on which monitoring and evaluation are applied and single
out those that will always be true and recurring as is the case in the statement
that “the demand for a good will in majority of cases drop as its price
increases”.
Taking the analogy further, if economics deals with studying human behaviour
in production, distribution and consumption of goods and services, one first has
to define what monitoring and evaluation deal with in order for one to be able to
develop a theory which can then be the basis for monitoring and evaluation
professionalization. Mastery of theory, along with mastery of practical skills of
the field, is a hallmark of professionals. Indeed, according to Fugate and Knapp,
reliance on the theoretical is the single most important factor distinguishing a
profession from a craft.
Taking the example of economics again, the theory that there is an inverse
relationship between the price of a good or service and its quantity demanded
can be used to study human behaviour in the manner of explaining it and thus
understanding it. It can predict future behaviour and tools can be developed out
of it for analysing and even controlling human behaviour. The validity of the
theory has thus been tested in practice. The theory has empowered novices to do
the things that formerly only experts could do. Therefore, it is not craft.
Theories of change have been largely used as a tool for evaluation, long before
ActKnowledge or the Roundtable began refining the process. It is probably impossible to
pinpoint the first use of the term "Theory of Change," but a hint at its origins can be found in
the evaluation community among the work of notable methodologists, such as Huey Chen,
Peter Rossi, Michael Quinn Patton, and Carol Weiss. These methodologists, along with a host
of others, have been thinking about how to apply program theories to evaluation for at least
20 years.
The “theory” was popularised by Weiss who described it “as a way to describe
the set of assumptions that explain both the mini-steps that lead to the long term
goal of interest and the connections between program activities and outcomes
that occur at each step of the way”.
According to ActKnowledge
She challenged designers of complex community-based initiatives to be specific about the
theories of change guiding their work and suggested that doing so would improve their
overall evaluation plans and would strengthen their ability to claim credit for outcomes that
were predicted in their theory.
While the “theory” seems to be logically plausible because it says you need a
plan to achieve a goal by suggesting mini-steps to follow, other disciplines like
project management have adopted the same approach by insisting that every
project should have a work breakdown structure. However, experience has
shown that even the projects with the best work breakdown structures still fail.
According to Duignan,
Outcomes systems are the range of related systems used in various sectors, disciplines and
professions which attempt to specify or measure outcomes (also known as results, goals,
objectives, targets etc); to attribute changes in such outcomes to parties (individuals, projects,
programs, organizations, coalitions, joint ventures, governments etc); to contract or delegate
the achievement of outcomes; and to hold parties to account (reward or punish them) for
changes in outcomes. Outcomes systems are known by names such as results management,
performance management, monitoring, evaluation, evidence-based practice, contracting and
strategy. Outcomes theory provides a rigorous set of definitions and principles for analyzing
and improving such systems.
The foregoing discussion confirms one key thing, that most efforts in the field
of monitoring and evaluation are still trying to put down what might distinguish
monitoring and evaluation from other fields and these efforts are diverse in their
approaches and fundamental meanings of concepts that they use to explain
scenarios. The key fundamental thing to note is that the developers of these
theories do not seem to subscribe to Fugate and Knapp’s thinking that theory is
the single most important factor distinguishing a profession from a craft, hence
making what the attempts are suggesting seem more as crafts than theories
because they do not meet Koskela’s criteria of what a theory should entail and
be able to do.
With all the above in place and natural factors within their control it is possible
for them to realise the output and achieve their goal.
Proofs
Consider the following hypothetical data collected from building the house by
the group of individuals alluded to above.
Table 1: Observed Data on Construction of a House
Construcion of a House
12 11
Amount of Output Realised
10
(Wall HIeght in Metres)
8.5
8
8
5.7
6
3.7
4
2 2.2
2
0
500 600 1000 1500 2100 2700 3100
Quantity of Inputs Used
(Number of Brikcs)
Looking at Figure 1 above it can be seen that as more bricks were used the
height of the wall also increased steadily signifying a directly proportional
relationship between inputs utilisation and the output realised.
Consider the hypothetical data below that has been collected by observing the
price fluctuation of bricks purchased and the quantity fluctuation of bricks
demanded in building the house at different price levels.
Table 2: Observed Data on Construction of a House
As was the case in with the data in Table 1, the data in Table 2 was used to
develop the graph depicted in Figure 2 below.
230
(Rands)
250 220
200
150 120
100
50
0
500 600 1000 900 2100 2000 3100
Quantity Demanded
(Number of Bricks)
Looking at Figure 2 above it can be seen that as the price of bricks increased
overtime the quantities of bricks demanded declined steadily overtime. If one
concentrates on the price pattern between R320 and R220, it can be seen that as
the price increased from R320 to R330 the quantity demanded declined from
1000 units to 900 units. As soon as the price declined to R220 the quantity
demanded increased to R2100.
Table 3 below is the same as table 1. The only difference is that a hypothetical
technology factor of 1 has been applied which has resulted in the height of the
wall increase by a factor of 1 assuming that a technology factor of 1 has an
output increase factor of 1.
Table 3: Observed Data on Construction of a House with Technology
The graph is Figure 3 below has been derived from the data in Table 3.
Figure 3: The exponentially proportional relationship between technology use and
output
Construction of a House
14
12
Amount of Output Reallised
12 11
(Wall Hieght in Metres)
10 9.5
9
8.5
8
8 6.7
5.7
6 4.7 Without Technology
3.7
4 3.2
2.12.2 2.2 With Technology
2
0
500 600 1000 1500 2100 2700 3100
Quantity of Inputs Used
(Number of Bricks)
Holding other factors constant, the above relationships can be tested with
empirical data and the behaviour they present will always hold creating a model
that can be used to explain and predict behaviour in other forms of activities
other than building a house where resources are used to achieve a goal. The
above relationships are important because they advise on what results have to be
expected out of certain behaviours during acquisition of inputs by those
involved in interventions that are geared towards realisation of certain results.
As such they become an integral basis for developing a monitoring and
evaluation theory.
Monitoring can only be deemed to have taken place well when an intervention
that was being implemented has led to the realisation of results that were
planned before hand. The conventional way of monitoring is to routinely track
key elements of an intervention such as inputs and outputs. However, looking at
this way of monitoring very closely is it really useful to monitor inputs? Use of
inputs implies progress but does not necessarily mean expected progress.
Realising that half of a stack of bricks has been used up says nothing about the
height of the wall that has been built except if certain things had been held
constant. Some of the bricks could have been thrown away as waste.
However, realising that half of the expected height of the wall has been built
immediately tells one that the house is halfway to completion and most
probably half of construction material such as bricks has been used up. The
same logic goes about time. Realising that one has taken 5 weeks to build a
portion of a house says nothing about how far one has gone in terms of the
amount of house that has been built or remains to be built to finish it. However,
by looking at the amount of house built one can tell how far one is towards
completing the house. What this means is that monitoring of time and other
resources is not important in telling anything about progress of implementation
of an intervention because they tell little about the amount of progress made in
realising the expected result. They are better as indicators of amount of usage as
opposed to indicators of progress. Therefore, spending a lot of effort monitoring
time spent and amount of resources used has a limitation in telling progress and
quality of results, which are critical success factors.
One logical way of looking at time and money or resources money buys is as
delimiters. This means that the best way of monitoring an intervention is to
concentrate on the degree of progression as evidenced in the expected results.
Concentrating on progress made in working towards realising the expected
results is the one that dictates how much more of inputs in the form of time and
resources are still needed to reach the final form of the expected result. This
implies that what should be done is simply to insure that there are enough
inputs.
However, given the fact that both time and money are not in limitless supply
due to competing issues that require human effort and resources in other spheres
of life, estimating them on the basis of past experience should always be
common practice. This should happen at planning and they should be
understood as intervention delimiters and not indicators of progress made.
Another factor that has to be taken into account at planning is to identify what I
call “Success Limiting Factors” or SLFs. These are factors outside an
intervention but which are critical in determining its success. For example, what
are possible SLFs in building a house? The following are the key ones; drying
up of resources supply, negative factors to staff working on building the house,
unfavourable weather, country political instability and escalation in prices of
input resources. Constant monitoring of these is sure to create certainty in
achieving anticipated progress and the ultimate expected result in the form of a
house. It is important to note that the fewer SLFs are the better. Therefore,
every effort has to be made during planning or before implementation starts that
those SLFs that can be eliminated are eliminated. It needs to be revealed though
that some of the SLFs may be force majeure. In the case of these, alternative
ways of doing the work under such SLFs should be decided a priori so as to
hype the mood when the SLFs do take effect.
Therefore, while the mother in the above example is like the bricklayer by being
close to the action, the father or other members of the family should be there to
contribute in manners diverse such as making sure that nothing external
disrupts, endanger or even cause death to the mother and the child (the
intervention) in its process of growth. The father or other members of the family
could be looked at as the monitoring expert or the intervention manager.
After the plan has been developed, make a detailed list of all the possible SLFs
to the intervention, identify their sources in terms of data available on them and
determine their behaviour and frequency of fluctuation. Eliminate all those that
can be eliminated before the intervention starts by revising the intervention
plan. Develop possible remedies for those SLFs that are force majeure. Have
SLFs plan outside the intervention plan.
Goal (SMART)
Output SLF 1
Developmental Objective (SMART)
Output SLF 2
Expected Output
Output SLF 3
Performance Indicators Output SLF 4
Human and Financial Resources (Budget) Output SLF n
Output Milestone 1
Output Milestone 2
Output Milestone 3
Output Milestone 4
Output Milestone n
IMPLEMENTATION STAGE
Output SLF 2
Output SLF 1 Output SLF n
The point where the lines intersect indicates that one SLF can affect realisation
of more than one milestone of the expected result. The overlap of the blue
circles in the results realisation chain below indicate that monitoring is a
continues process and there will be less of it at early stages towards results
realisation milestone and more of it closer to the result realisation and lesser of
it after the milestone has been hit, following an erratic pattern with troughs and
crests as realisation of results milestones progresses, with the trough being at
the beginning from the left of the results chain where the circumference of the
first circle begins and where the circles intersect and crests being at the areas
that give the diameter of each circle.
References
ActKnowledge, www.theoryofchange.org/background/basics.html
Duignan, Paul. Introduction to Outcomes Theory.
http://knol.google.com/k/paul-duignan-phd/introduction-to-outcomes
theory/2m7zd68aaz774/3.
Fugate M. & Knapp J. 1998. The development of bodies of knowledge in the
professions (Study for the Project Management Institute). Princeton, New
Jersey, US.
Koskela, Lauri & Gregory Howell. 2002. The underlying theory of project
management is obsolete. Http://www.usir.salford.ac.uk/.../2002,pdf.
Koskela, Lauri. 2000. An exploration towards a production theory and its
application to construction. Espoo, VTT Building Technology. 296 p. VTT
Publications; 408. WWW: http://www.inf.vtt.fi/pdf/publications/2000/P408.pdf
Koskela, Lauri & Gregory A. Howell. 2002. The theory of project management
- problem and opportunity. Working paper. VTT Technical Research Centre of
Finland & Lean Construction Insitute.