Probabilistic Judgement Studies - The Evidence

In many accounting contexts and especially in auditing, there is no ‘correct’ solution with which
judgements can be compared in order to assess their accuracy. One way of coping with this lack
of a benchmark criterion against which to judge performance is to examine the degree of
consensus regarding a particular decision across a number of decision makers. Another way is to
use a mathematical or statistical model. As discussed in an earlier section, probabilistic
judgement research is based on analysis of whether humans revise their beliefs in line with
Bayes’s theorem once new evidence becomes available. HJT research within this model has
consistently demonstrated that humans possessing a variety of skill levels and, observed over a
variety of tasks, revise their prior probabilities to a lesser extent than Bayes’s theorem prescribes.
This conservatism has been attributed to the use of rules of thumb and biases which are adopted
as a means of simplifying complex judgements in order for humans to cope. 
Three rules of thumb are defined in the literature as follows:
a. Representativeness
This rule of thumb states that when judging the probability that a particular item comes from a
particular population of items, people’s judgement will be determined by the extent to which
the item is representative of the population. Items or events that are viewed by the decision
maker as being more representative will be assessed as having a higher probability of
occurrence than those that are less representative. For instance, a bank loan officer may judge
the likelihood that a company will default on its loan by how similar it is to a stereotypical
failed firm. As described later, research has shown that the use of this rule of thumb can lead
to poor decisions because decision makers who use it often ignore other relevant data that are
not part of the representative stereotype. One example of these errors relates to the rate at
which the event of interest actually occurs in the total population (this is called the ‘base
rate’). The population rate of loan default might be, say, 5 percent but the decision maker
might ignore this important information and consequently overstate the rate of default when
evaluating a sample of companies mainly because he or she is looking only for companies that
have characteristics that correspond to the stereotype.
b. Availability
The availability rule of thumb refers to the assessment of the probability of an event based on
the ease with which instances of that event come to mind. The consequence of use of this rule
of thumb is that probabilities related to ‘sensational’ events are likely to be overestimated.
c. Anchoring and Adjustment
This rule of thumb refers to a general judgement process in which an initially generated or
given response serves as an anchor, and other information is used to adjust that response. The
consequence of this rule of thumb is the possibility of insufficient adjustment in the light of
changing circumstances.
As previously stated, most of the research into probabilistic reasoning has used auditors as
subjects. Auditors are ideal subjects for the study of human judgement, and probabilistic
reasoning in particular, since many audit judgements entail the need to revise assessments in
the light of additional evidence. In addition, in the United States the larger public accounting
firms have been extremely cooperative in making available both funds and practising auditors
for research. A brief overview of research related to all three rules of thumb follows.
Representativeness: The Evidence
Kahneman and Tversky first reported the existence of representativeness and the tendency to
ignore base rates. Since then, research in both the psychological and accounting fields has
investigated the phenomenon. The evidence is inconclusive in that it shows base-rate information
is sometimes neglected and sometimes used appropriately in assessing the probability of an
event. The use of base-rate information seems to be highly sensitive to a variety of tasks and
contexts, and this has led to the hypothesis that probabilistic reasoning involves contingent
processing.
Joyce and Biddle used an accounting adaptation of the employee theft/lie detector example
previously given in this chapter to illustrate the application of Bayes’s theorem. In this example
related to management fraud, again a very low base rate was used. It was expected that, as in the
previous example, subjects would pay insufficient attention to the low base rate and, therefore,
have too high an expectation of fraudulent activity. Practising auditors were the subjects, and
results indicated that, although the auditors did not ignore the base rate given, they also did not
consider it sufficiently. Since audit judgements often involve low-base-rate, high-impact events
(e.g. fraud), the authors were concerned about the implications of this result. However, Holt cast
doubt on the findings of Joyce and Biddle, arguing that it was the wording of the problems,
leading to a framing effect, which had driven their results. Framing effects are defined as a
cognitive perspective elicited by the task characteristics. They are often illustrated by use of the
half-empty/half-full glass analogy where a pessimist regards a glass as half empty and an
optimist regards the same glass as half full.

Availability: The Evidence

The basis of this rule of thumb is that likelihood judgements are based on retrieval from memory
of relevant instances or construction of plausible scenarios. The more instances recalled, or the
greater the ease with which one can recall instances or generate plausible explanations for an
event, the higher will be the judged probability of occurrence of the event. However, this
requires a large sample of probabilities to improve prediction accuracy.

Representativeness: The Evidence

Kahneman and Tversky first reported the existence of representativeness and the tendency to
ignore base rates. Since then, research in both the psychological and accounting fields has
investigated the phenomenon. The evidence is inconclusive in that it shows base-rate information
is sometimes neglected and sometimes used appropriately in assessing the probability of an
event. The use of base-rate information seems to be highly sensitive to a variety of tasks and
contexts, and this has led to the hypothesis that probabilistic reasoning involves contingent
processing.
Joyce and Biddle used an accounting adaptation of the employee theft/lie detector example
previously given in this chapter to illustrate the application of Bayes’s theorem. In this example
related to management fraud, again a very low base rate was used. It was expected that, as in the
previous example, subjects would pay insufficient attention to the low base rate and, therefore,
have too high an expectation of fraudulent activity. Practising auditors were the subjects, and
results indicated that, although the auditors did not ignore the base rate given, they also did not
consider it sufficiently. Since audit judgements often involve low-base-rate, high-impact events
(e.g. fraud), the authors were concerned about the implications of this result. However, Holt cast
doubt on the findings of Joyce and Biddle, arguing that it was the wording of the problems,
leading to a framing effect, which had driven their results. Framing effects are defined as a
cognitive perspective elicited by the task characteristics. They are often illustrated by use of the
half-empty/half-full glass analogy where a pessimist regards a glass as half empty and an
optimist regards the same glass as half full.

Availability: The Evidence

The basis of this rule of thumb is that likelihood judgements are based on retrieval from memory
of relevant instances or construction of plausible scenarios. The more instances recalled, or the
greater the ease with which one can recall instances or generate plausible explanations for an
event, the higher will be the judged probability of occurrence of the event. However, this
requires a large sample of probabilities to improve prediction accuracy.
Moser examined the availability rule of thumb in connection with investors’ predictive
judgements. He asked half of his 58 subjects to list reasons the target company’s profits would
increase and then reasons the company’s profits might decrease. The remaining subjects listed
reasons in the opposite order. Moser found that the first group of subjects made predictions of
higher probability regarding an increase in profits for the company, despite there being no
objective basis for their optimism. The outcome for which the subjects were able to generate the
most supporting reasons was judged more probable. He concluded that some environmental
event regarding a particular company that caused disproportionate news coverage might
systematically influence predictive judgements. Individual investors might either overprice or
underprice shares because of a tendency for all to think about the company from the same
perspective - optimistic or pessimistic.

Anchoring and Adjustment: The Evidence

Joyce and Biddle again used practising auditors as subjects in examining the effect of changes in
internal control systems on the extensiveness of substantive tests (audit tests designed to search
for the existence of dollar errors in the accounts). It was expected that the subjects would adjust
for changes in internal controls by adjusting audit scope but that the adjustment would be
insufficient as anchoring on the initial internal control would occur. No evidence of anchoring
and adjustment was found. However, Kinney and Uecker did find evidence of anchoring and
adjustment in analytical review (ratio analysis) and compliance test (audit test of internal
controls) tasks.

Expert Judgement and Rules of Thumb

The research involving expert judgement is concerned with examining the thought processes of
experts and the determinants of expertise. Newell and Simon provided the analytical framework
with their theory of bounded rationality. They suggested that humans possess a short-term
memory with very limited capacity (4-7 chunks) and a virtually unlimited long-term memory.
The structure of these memories and the characteristics of tasks combine to determine the way
different types of problems are represented in memory (cognitive representation) which in turn
determines the way problems are solved.
Much of the early work on expert memory involved medical practitioners and chess masters. The
ability of experts to effectively expand their memory capacity in situations related to their
expertise has been a consistent finding in the literature. It seems that a small set of cue values
brings to mind a larger set of cues associated with a prototype. The explanation lies in the use of
representativeness to recognise a familiar pattern and bring a prototype into short-term memory
out of long-term memory.
Bouwman found evidence of representativeness in an accounting task when he asked 15 students
(novices) and 3 accountants (experts) to analyse four cases containing extensive financial
information to determine any underlying problem areas. He found that the experts followed a
directed strategy based on standard checklists, complex trends and stereotypes, and developed an
overall picture of the company. The experts, when encountering a violation of the stereotype,
attempted to uncover the cause, searching for both confirming and non-confirming evidence. The
students, on the other hand, followed a simple undirected sequential strategy, evaluating
information in the order presented, searching for a significant fact which could explain the
situation. They searched only for confirming evidence and did not attempt to find causal
explanations.
Numerous studies in auditing have confirmed that audit experts have better recall, integrative
abilities and error frequency learning ability than do novices. Audit experts exhibit evidence of
all three rules of thumb and it is not clear that this necessarily results in inferior quality in
decision making. The double-entry nature of bookkeeping means that audit tests often overlap
and mechanisms exist (e.g. partner review) which try to assure quality. Process tracing methods
may be a good way of learning more about the differences between the decision processes of
experts and novices. This knowledge would be valuable for training purposes.