BEHAVIORAL RESEARCH IN ACCOUNTING Vol. 24, No. 1 2012 pp.

7390

American Accounting Association DOI: 10.2308/bria-10074

Internal Control Assessment and Interference Effects

Janet B. Morrill Cameron K. J. Morrill University of Manitoba Lori S. Kopp University of Lethbridge
ABSTRACT: Both U.S. Generally Accepted Auditing Standards and International Standards on Auditing require risk-based audits, where audit effort is concentrated on accounts and financial statement assertions where the risk of material misstatement is high. Assessing risk requires the auditor to evaluate the auditees internal control systems; however, current standards and practice vary regarding the point at which risks are to be identified. Using output interference theory, we hypothesize that risk assessment performed by the auditor before evaluating the clients internal control systems will lead to a more complete identification of sources of internal control deficiencies as compared to assessing risk after evaluating internal control systems. In our experiment, auditors who identified risks first identified more, and more important, internal control deficiencies than did auditors identifying controls first, although the number of risks identified was not significantly different between the two groups. Overall, our results suggest that audit efficiency and effectiveness depend on the sequence in which internal control evaluation subtasks are performed. Keywords: auditor judgment; internal control evaluation; interference. Data Availability: Data are available from the authors upon request.

INTRODUCTION he successful design and evaluation of internal control systems is playing an increasingly important role in corporate governance and nancial statement audits. Both U.S. Generally Accepted Auditing Standards and International Standards on Auditing require risk-based audits, where audit effort is concentrated on accounts and nancial statement assertions where the

Janet Morrill and Cameron Morrill are Chartered Accountants Research Fellows at the University of Manitoba. We express our gratitude to the Deloitte Touche Research Foundation and the University of Manitoba Centre for Accounting Research and Education, funded by the Institute of Chartered Accountants of Manitoba, for their nancial support; the Certied General Accountants of Manitoba and British Columbia that provided participants for this study; and Jeff Thomas and Erin Sharpe for their assistance. We also appreciate helpful comments from Theresa Libby, two anonymous reviewers, Ed ODonnell, participants at the University of Manitoba faculty workshop series, the Certied General Accountants/University of Manitoba Accounting Research Conference, and comments from participants and reviewers of annual meetings of the American Accounting Association (AAA), the Canadian Academic Accounting Association, the Administrative Sciences Association of Canada, and the AAA Auditing Section.

Published Online: January 2012

73

74

Morrill, Morrill, and Kopp

risk of material misstatement is high. To assess the risk of material misstatement, the standards require the auditor to acquire an understanding of the risks of an auditees business, including its internal control systems. The assessment of the internal control systems is integral to the process as controls may mitigate some of the risks that would otherwise lead to material misstatements. Identifying risks and evaluating and improving internal control systems are also of vital interest to public companies, given the increased scrutiny and reporting obligations in this area imposed by the Sarbanes-Oxley Act. At the same time, there is evidence to suggest that risk assessment and the evaluation of internal control systems are very difcult. Roybark (2006) analyzed the inspection reports issued in 2005 by the Public Company Accounting Oversight Board (PCAOB) and found that audit deciencies in internal control assessments were particularly prevalent and were an area of particular importance. Thus, techniques that increase the effectiveness of risk assessment and internal control evaluation may improve nancial statement audits while simultaneously yielding substantial benets to organizations seeking to improve or assess the quality of their internal controls. The determination of nancial statement accounts and assertions containing signicant risks and/or internal control deciencies forms the basis for the risk-based audit wherein the audit is subsequently designed to focus audit effort in these areas. To make this determination, the auditor must acquire knowledge of the auditees business, the business and entity-wide risks that entail a risk of material misstatement in the nancial statements, and how the entitys internal control systems address those risks. Risks are identied at increasing levels of specicity from business and entity-wide risks, to nancial statement risks, to assertion level risks, and nally to transaction level risks. A key outcome of this process is the auditors determination of whether the risks at a transaction level are appropriately reduced or eliminated by the organizations internal control systems, or if signicant internal control deciencies remain. In this paper, we focus on sub-components of this process: The identication of transaction level risks facing the organization, the assessment of controls, and the identication of internal control deciencies under different task sequences. While a risk-based audit is required by auditing standards, the standards do not mandate when risks need to be evaluated in the process. The purpose of this study is to determine whether consideration of risks rst affects the internal control evaluation. We hypothesize, based on output interference theory, that risk identication performed by the auditor before evaluating the clients internal control systems will lead to a more complete identication of internal control deciencies than risk identication performed after controls are evaluated. We theorize that the identication of controls before identifying risks interferes with the ability to generate risks that are not being controlled, resulting in fewer deciencies being identied. We also investigate whether auditors who identify controls before identifying risks will be less successful in identifying important deciencies. Consistent with our expectations, we found that participants who generated risks before identifying controls (hereafter risks-rst) identied signicantly more internal control deciencies than participants who performed those tasks in the reverse order (hereafter controls-rst). In addition, we found that the risks-rst participants identied more important deciencies than did the controls-rst participants, where importance was dened by the judgments of an expert panel. On this basis, we conclude that a risks-rst approach appears to be more effective. However, we also found that the risks-rst participants identied signicantly fewer controls, indicating that the increase in effectiveness may entail sacrices in the efciency of the audit. Given the critical importance of internal control design and evaluation to effective corporate governance, nancial statement audits, and internal control reporting and audits required under SOX 302 and 404, these insights could be of signicant interest to academics, standard setters, and practitioners.
Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

75

HYPOTHESIS DEVELOPMENT Identication of Risks in Internal Control Evaluation Risk assessment is a critical and early step in the design of an organizations internal control system. The Committee of Sponsoring Organizations (COSO) framework for internal control (COSO 1944) and the updated guidance for small companies (COSO 2006) recommend that companies perform a risk assessment process, which, in turn, informs the design of the internal control system and the implementation of control activities. Risk assessment consists of two major steps: risks must rst be identied, and then their signicance must be analyzed. The timing of risk identication and analysis in the evaluation of control systems for external nancial statement audit purposes is less clear. While U.S. and international GAAS require riskbased audits, it is not clear whether they require, or whether in practice this is equivalent to, risksrst audits. Auditing Standard 5 (AS5), produced by the PCAOB in 2007, recommends that auditors use a top-down approach to the audit of internal controls. This approach begins at the nancial statement level with the auditors understanding of the overall risks to internal control over nancial reporting (PCAOB 2007, AS5, paragraph 21). The auditor should determine the likely sources of potential misstatements in the nancial statements and might do so by asking himself or herself what could go wrong? within a given signicant account or disclosure (PCAOB 2007, AS5, paragraph 30). Therefore, the U.S. standard implies that the auditor should perform a risk generation or identication process, and that process should occur early in the internal control evaluation. The international standard ISA 315 suggests that the risk assessment procedure and acquiring an understanding of internal controls are iterative and simultaneous: ISA 315 states that all the risk assessment procedures are performed by the auditor in the course of obtaining the required understanding (IAASB 2007, ISA 315, paragraph 7; emphasis added) of the entity and its environment. To acquire some insights regarding when risk assessment is done in practice, we examined the Canadian Institute of Chartered Accountants Practice Engagement Manual (CICA 2007) and the Certied General Accountants Association Public Practice Manual (CGA 2007). Neither manual contains an explicit individual step wherein the auditor identies risks, but the internal control evaluation process in both manuals culminates in a list of internal controls cross-referenced to a list of transaction risks. Specically, the CICA Practice Engagement Manual (CICA 2007) contains a risk matrix listing transaction risks and their related assertions in columns and the internal controls identied by the auditor in the rows. The auditor then places a check at the intersection of the control and the risk to which it relates. Given that questionnaires documenting internal controls precede the nal questionnaire cross-referencing transaction risks to identied controls, it would appear that auditors likely perform a transaction risk identication either while they document the internal controls (consistent with ISA 315) or after they document internal controls. The Certied General Accountants Association Public Practice Manual (CGA 2007) contains internal control questionnaires listing nancial statement assertions, associated transaction risks, and possible controls. The auditor enters the description of the key control addressing each transaction risk, again implying that risk assessment is performed while internal controls are documented. Finally, we interviewed audit partners from each of the Big 4 audit rms to determine at what stage risks are identied in the internal control evaluation process. None of the rms had strict guidelines in this area, and they varied somewhat in their approaches. One rm reported that the risk identication stage was explicitly addressed by senior personnel as part of the audit planning preparation, and the areas of identied high risk were communicated to audit personnel, which would have some impact on the depth of understanding of internal control acquired. Another rm, on the other hand, expressed that by the end of the audit, all the evidence needs to have been gathered, consistent with a risk-based audit. All rms suggested that interviews with client personnel to
Behavioral Research In Accounting Volume 24, Number 1, 2012

76

Morrill, Morrill, and Kopp

acquire an understanding of internal control tended to be unscripted. In some cases, the client personnel would be invited to explain what controls were present in their system, whereas in other cases the interview might start by asking client personnel what risks they perceived to be important when the system was designed. In practice, it therefore appears that order is not consistently applied. In the next section of the paper, we use insights from output interference theory to explain why we believe order matters. In fact, output interference theory suggests that auditors should identify risks before they document internal controls. Output Interference Output interference is a phenomenon whereby the act of recalling, or the cuing of, some items inhibits the recall of other items (Slamecka 1968). Specically, output interference inhibits the retrieval of cues from memory, thereby affecting the amount of information recalled. In the auditing literature, output interference has been found to result from the use of information provided by the client or from audit decision aids such as questionnaires and checklists. For example, Frederick (1991) found that auditors were affected by output interference when attempting to recall a list of 33 internal controls given to them. After reviewing the internal controls using the taxonomic categorization of internal controls (e.g., authorization, validity of transaction, proper recording, etc.), auditors given a partial list of internal controls recalled fewer of the remaining controls than auditors that were not given a list of internal controls. In addition, Bierstaker (2003) found that providing high- and low-knowledge auditors with incomplete owcharts in addition to complete narratives interfered with the low-knowledge auditors ability to recall internal control deciencies and strengths. There was no interference effect for the high-knowledge auditors. Perhaps of greater relevance to our work are studies that document that interference also affects the generation of concepts from long-term memory, rather than simply recall of recently viewed items. That is, concepts that have received heightened activation are likely to be re-generated, while there is a decreasing likelihood of generating items that have not been activated. During analytical procedure judgments, Anderson et al. (1992) examined whether the generation of explanations from one category (errors or non-errors) inhibited auditors ability to generate explanations from the other category. Auditors who were asked rst to think about and list non-error explanations generated fewer error explanations than auditors asked to think about and list error explanations before non-error explanations. Church and Schneider (1993) found that providing auditors with a hypothesis from a particular transaction cycle inhibited retrieval of additional hypotheses from the same transaction cycle. The generation of transaction risks to which an organization is exposed is a difcult process. One reason is that there are many potential risks to consider. For example, paying the same invoice twice violates the occurrence assertion for expense accounts. The occurrence assertion for expenses accounts could also be violated if invoices are paid for legitimate goods or services that were never received, if cutoff errors occurred, or if there are ctitious expenses arising from employee fraud or management fraud. In addition, the risks may be specic to the rm or industry. Comparing the pre-specied transaction risks in the CICA and CGA manuals illustrates the complexity of the risk identication process. For example, for the completeness, existence, accuracy, and valuation assertions of purchases, payables, and payments, the CICA manual lists ten transaction risks. The CGA manual contains ve of those risks plus four that are not in the CICA manual. Both questionnaires provide space for additional risks. In current practice, auditors may embark on an identication of controls before specically identifying the risks to which the company is exposed. We hypothesize that performing an internal control evaluation in this order will inhibit the generation of risks because of interference effects, resulting in a less effective internal control assessment. In the risk generation phase, risks that are
Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

77

FIGURE 1 Hypothesized Cognitive Process with Controls-First Approach

Task Steps 1. Document system: What controls are in place? 2. What risks are addressed by the identied controls? 3. Generate risks Hypothesized Cognitive Effect Activates mental representations of controls Activates risks addressed by the clients internal controls Controlled risks activated in the previous step are likely to be re-generated and activated even further. Other uncontrolled risks are less likely to be activated due to interference from controlled risks Likely to be yes, since few uncontrolled risks would have been generated in prior step Likely to be no, since all risks identied (which are controlled) have been addressed (because they are controlled)

4. Determine if all risks are addressed 5. Determine if there are signicant deciencies

addressed by controls compete for generation with risks that ultimately are not addressed by controls. The risks that are addressed by controls are more likely to be generated as they would have been activated during the previous step, when the auditors matched the controls they identied to the risks addressed by those controls. Therefore, the process of matching controls to risks would inhibit the subsequent generation of risks that are not controlled, i.e., internal control deciencies. This process is summarized in Figure 1. If such interference occurs, then beginning the internal control evaluation with the documentation of the existing internal control structure will reduce the likelihood that internal control deciencies are identied by the auditor. We hypothesize that internal control deciency identication is enhanced by asking the auditor to generate risks before identifying the controls in place, as the links to controlled risks will not have been previously activated.1 This leads to the following hypothesis, stated in alternative form: H1: Auditors who identify risks rst will identify more internal control deciencies in the system than auditors who identify controls rst. H1 proposes that the quantity of risks and internal control deciencies identied will be affected by the order in which tasks are performed. We also explore whether task order will affect the importance of internal control deciencies identied, where the importance of an internal control deciency is a function of the likelihood of occurrence of the underlying risk in the absence of the control, and the signicance of the repercussions of that occurrence (COSO 1994). Auditors in the risks-rst task order group must generate risks before investigating the controls in place. Prior research has shown that an important component of auditor knowledge is the
1

This hypothesis is also somewhat related to framing effects. Emby (1995) found that auditors rely less on internal controls when words such as risk are embedded in the task materials. It is suggested that the use of these words activates a different problem representation, or frame, that changes the auditors judgments. There is at least one substantial difference between the framing studies and this study: Framing studies investigate the impact of changing the wording of the experimental materials whereas, in our study, we vary the way (specically the order) in which the auditor performs the task. We did not expect framing to have a signicant effect, but we further consider this issue in a later section of the paper.

Behavioral Research In Accounting Volume 24, Number 1, 2012

78

Morrill, Morrill, and Kopp

knowledge of error occurrence rates and that experienced auditors will generate more frequently occurring errors as an explanation for audit ndings (Libby and Frederick 1990; Tubbs 1992). In similar fashion, we expect that auditors in the risks-rst condition are likely to generate more frequently occurring risks. If any of these risks are not addressed by the system of internal controls, it is likely that the resulting internal control deciencies would be very signicant. Auditors in the controls-rst condition, on the other hand, are less likely to be able to identify signicant internal control deciencies as they are hampered by the interference effects resulting from the identication of risks that are already controlled by the auditee. Companies implement internal controls subject to a cost-benet criterion (COSO 1994). The controls adopted are thus likely to be related to typical risks, in the sense that these risks are likely to be signicant to many organizations. Libby and Frederick (1990) and Heiman-Hoffman et al. (1995) found that an inherited typical (i.e., frequently occurring) error inhibited the retrieval of additional errors or hypotheses more than an inherited atypical error. In an internal control evaluation setting, auditors in a controls-rst condition are more likely to inherit a typical (frequently occurring) risk as these are the risks that are most likely to be controlled by the company. Therefore, we expect that this will signicantly inhibit the generation of additional risks, which could include typical high frequency risks, resulting in less effective internal control evaluations.2 In the worst case, one of those high frequency risks could be uncontrolled by the company. Again, if the important risk is not identied, then an important deciency would not be identied. In other words, auditors in the risks-rst group are more likely to generate important risks, which will lead to identifying important deciencies, as auditors likely know what risks are important. Auditors in the controls-rst group are less likely to generate important risks and are, hence, less likely to generate important deciencies, as the typical controls that they likely encountered in the system generate the most interference. To summarize, we argue that important deciencies and risks are less likely to be identied by auditors in the controls-rst condition, as the controls they identify are likely to be typical and will thus interfere with the generation of other typical risks. We are, therefore, making a link between typical and important. The two terms are associated in that, as noted above, controls become typical as many companies adopt them. The reason companies adopt them is presumably because they are important. However, we acknowledge that these terms may not be equivalent, particularly as companies may face important risks that are specic to their situation that would not be considered typical. Similarly, we argue that auditors in the risks-rst condition are more likely to uncover important deciencies as they are likely to generate frequently occurring risks. As importance is a function of both frequency and the magnitude of the consequences, importance and frequently occurring are related terms but are not necessarily equivalent. Therefore, we phrase our investigation of task order effect and the importance of deciencies as a research question rather than a hypothesis: RQ1: Do auditors who identify risks rst identify more important internal control deciencies in the system than auditors who identify controls rst?

The arguments presented here relate to internal control evaluations conducted in real life, whereas our participants are working with a hypothetical case. The controls described in our experimental case address typical risks only insofar as the internal control system described in our case is representative of typical systems. Our experimental case was pilot-tested for realism by ten managers from Big 4 audit rms. If interference theory is applicable in an internal control evaluation setting, as we suggest, we are condent that the controls in our case should induce interference effects in our experimental setting in the same way that they would in real life.

Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

79

METHOD We conducted our study using accountants completing their auditing courses required to receive the Certied General Accountant (CGA) professional designation.3 The experiment was administered to a total of 78 participants on three separate occasions, to accountants attending sessions at the end of CGA Auditing 1 or Auditing 2 courses in two different cities.4 The responses of three participants were eliminated as their responses were incomplete and impossible to encode. Demographic information on the participants is provided in Table 1. The mean age of participants was 33.4 years, and their mean work experience was 42.9 months, reecting the fact that the auditing course is typically taken within one year of receiving the CGA professional designation. There were no statistically signicant differences in demographic variables detected between the two experimental groups. All experimental participants appeared to be motivated and were compensated for their time. Additionally, a $100.00 prize was offered to the participant with the highest score, so we have no reason to believe that the participants were not fully engaged in the task. Participants were given as much time as they needed to complete the task. In the session where we monitored time, the average time to complete the task was 56 minutes, with the minimum and maximum time being 40 and 90 minutes, respectively. The task, adapted from Kopp and Bierstaker (2006), involved the evaluation of the internal controls of the purchases/payables/payments cycle of a medical supply company. The system description is included in Appendix A. The participants were randomly assigned to one of two conditions, which varied in the order of the sub-tasks to be completed. Participants in the rst condition (the risks-rst condition) identied the risks that should be addressed by an internal control system. They then received the detailed narrative description of the internal control system, after which they were given a second opportunity to identify risks in the system. They then identied the controls in the system and identied any deciencies. These steps are outlined in Figure 2. The second task order condition group (the controls-rst group) received the narrative description rst and were instructed to identify the controls in the system, after which they were asked to identify all the risks and nally any internal control deciencies in the system. In both conditions, once the rst step was completed, participants were permitted to go back to previous steps at any time, but were not able to skip forward. Participants in both groups were provided a training task before completing the experimental task. The training task was a complete evaluation of a sales/receivables/receipts system. The training task demonstrated the order of subtasks to be completed, which varied between the two groups, but the content of the task (the system description, and the risks, controls, and deciencies identied) was identical for both groups. In order to ensure that the groups adhered to the proper order of subtasks to be completed, we implemented two procedures. First, participants had to complete their case on three-part, carbonless
3

On average, participants reported being in level 4.4 of the 5-level CGA program, which requires an undergraduate university degree. We performed t-tests of differences in the dependent variables used in the study (number of risks, controls, and deciencies identied) to ensure that there were no signicant differences between City and Course groups. The groups (City 1 versus City 2 and Audit 1 versus Audit 2 students) had no signicant differences between the number of risks identied or the number of controls identied. On average, however, the City 1 participants generated signicantly more deciencies than did the City 2 participants (mean 3.26 versus 2.22, p , 0.05), and the Audit 1 participants, somewhat surprisingly, generated signicantly more deciencies than did the Audit 2 participants (3.53 versus 2.43, p , 0.04). Such differences could be due to different work environments or instructor effects, for example. We argue that these differences do not affect our conclusions as participants in the different city/course combinations were assigned randomly to the two experimental conditions, and the overall results reported here hold within each City and Course group. Nevertheless, we controlled for City, Course, and Months of Experience in our reported analyses.

Behavioral Research In Accounting Volume 24, Number 1, 2012

80

Morrill, Morrill, and Kopp

TABLE 1 Participant Information

Mean Mean (Standard Deviation) of (Standard Deviation) of Risk First Condition Control First Condition n 36 n 38 Months of accounting work experience Age in years Difculty of task (1 not, 11 very) Realism of task (1 not, 11 very) Difculty of ordering of task steps (1 not, 11 very) Difference between performance of the task and how they usually perform it (1 not, 11 very) 49.7 (55.5) 34.1 (7.7) 5.5 (2.4) 8.1 (2.1) 5.2 (2.6) 6.6 (4.0) 36.0 (29.4) 32.6 (7.6) 5.9 (1.9) 7.8 (2.5) 5.5 (1.9) 7.1 (3.5)

t 1.23 0.80 0.79 0.52 0.45 0.59

Mann-Whitney U 0.67 1.10 0.46 0.48 0.38 0.79

None of the differences between conditions are statistically signicant at conventional levels.

answer sheets. The answer sheets consisted of a three-column table, with columns for risks, controls, and deciencies, with the order varying according to the condition. Each row of the table was to contain a risk and a control that addressed that risk, or a deciency if no control existed. On the top sheet, only one column of the three-column table was available (the other two columns were shaded in). That column was entitled Risks for the risk-rst condition, and was entitled Controls for the control-rst condition. After completing that rst step, the participants turned to the second page of the answer sheet, which now had two columns free and the third column shaded out. The rst column contained a carbon copy of the answers they had written on the top page, leaving the second column to be lled in. The second column provided space for controls, for the risk-rst condition, and risks, for the control-rst condition. After lling in the second column on the second page, participants proceeded to the third page where they lled in their deciency assessment in the last column. These three-part sheets assured us that participants did not skip steps. In particular, by examining the second page, we could ensure that the participant had lled out the rst step on the rst page, as only a carbon copy appeared on the rst column of the second page, while the second column of the second page contained original ink. As an additional control, we had participants in the risks-rst condition request the detailed narrative description part way through the task, and we checked that they had attempted to identify risks if they were in the risk-rst condition before distributing the narrative. Hence, we were assured that all participants completed the tasks in the order prescribed. Participants also were asked how difcult and realistic they found the task; how difcult they found the ordering of the task to be; and how different they found the performance of the task to be from the way they usually performed it.
Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

81

FIGURE 2 Order of Steps in Experimental Task

Participants in both conditions were told not to skip steps, although they were permitted to return to previous steps at any time.

RESULTS Information about the participants in our study and results of manipulation checks are presented in Table 1.5 There were no signicant differences between the means of the two groups, indicating that (1) the perception of key features of the task was consistent across groups; and (2) the random assignment of participants to groups was effective in controlling for potentially confounding variables. The possible risks, deciencies, and controls that could have been identied from the case are listed in Table 2. The risks, deciencies, and controls identied by each participant were coded by two of the co-authors independently.6 The third sheet was used for this analysis, which, therefore, contained all the risks, deciencies, and controls that were ultimately identied by each participant, either in the initial step or in later steps. The variables RISK, CONTROL, and DEFICIENCY represent the total number of different valid risks, controls, and deciencies, respectively, identied by each participant. Descriptive statistics of these variables are presented in Table 3. On average,
5

Results of independent samples t-tests are presented in Table 4. We also performed Mann-Whitney nonparametric U-tests. Results were qualitatively similar. The kappa coefcient of inter-coder reliability was 89.6 percent, and all disagreements were reconciled.

Behavioral Research In Accounting Volume 24, Number 1, 2012

82

Morrill, Morrill, and Kopp

TABLE 2 Descriptions of Controls, Risks, and Deciencies Panel A: Controls

Description of Control C1. Purchases are initiated by supplies manager (segregation of duties) C2. Inventory re-order point exists C3. Large purchases are approved by the production manager C4. Goods are counted when received C5. Invoices are checked for accuracy when received C6. Not used C7. A disbursement voucher is prepared as A/P is entered C8. A/P manager reviews disbursement vouchers C9. A/P manager approves payments C10. Batch summary is checked by cashier to checks C11. Aborted payments are marked C12. There is adequate segregation of duties C13. Invoices are matched to supporting documentation C14. The cashier is authorized by the Board of Directors C15. Goods are counted to verify that quantities are correct C16. The packing slip is matched to receiving report C17. All purchase orders are authorized C18. Purchase order is matched to receiving report C19. Invoice is matched to receiving report C20. Invoice is matched to purchase order n 20 16 53 52 45 5 18 30 23 18 21 0 10 0 0 0 44 40 46 Importance 6.0 4.0 5.7 5.0 5.0 1.3 5.0 6.0 4.7 2.7 6.7 4.7 1.3 6.0 5.0 5.7 4.7 6.7 4.7

n refers to the number of participants, out of 74, who identied the control. Importance is the mean ranking of importance by three judges, on a scale of 1 (not important at all) to 7 (very important).

Panel B: Risks
Description of Risk R1. All purchases relate to the entity R2. All purchases are made from valid approved suppliers R3. Purchases are made if and only if required R4. All goods ordered are received R5. The quantity received equals the quantity on receiving report R6. All goods received are updated to inventory records R7. All goods received are posted to A/P R8. Payables are recorded if and only if goods are received R9. Inventory records are adjusted if and only if goods are received R10. The quantity entered in inventory is the quantity received R11. Pricing and extensions of invoices are correct R12. Amounts are correctly posted to inventory and A/P R13. Payables and inventory are not recorded twice R14. Payables are posted to correct suppliers R15. Payments are not paid twice R16. Payments are only made for legitimate A/P R17. Aborted payments are not sent out R18. All checks are made out in proper amount R19. All payment amounts are correctly recorded n 61 34 0 37 48 10 14 0 0 7 43 16 1 5 13 51 13 6 8 Importance 6.7 3.7 4.0 3.0 4.3 6.0 6.3 6.7 6.0 6.0 5.0 6.0 5.3 5.0 4.7 6.0 2.7 5.3 5.7 (continued on next page)

Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

83

TABLE 2 (continued)
Description of Risk R20. R21. R22. R23. R24. R25. R26. R27. R28. R29. R30. R31. R32. R33. R34. R35. R36. R37. R38. R39. R40. Payments are posted to correct suppliers All payments are recorded All payments are paid promptly Payables are not paid too early There is an adequate segregation of duties Logical access is controlled Physical access is controlled Purchase orders are not used twice Purchase orders are not lost or out of sequence Prices paid are not too high The quality of goods ordered is sufcient Goods are not received that were not ordered Items received are not damaged Goods returned are properly authorized Goods returned are accounted for properly Purchases are made when required Purchases are made only when needed Payables are recorded when goods are received No payables are recorded unless goods are received Inventory records are updated as soon as goods are received Inventory records are updated only when goods are received n 5 6 8 1 22 4 25 2 5 13 14 11 2 1 2 20 63 12 38 4 4 Importance 5.0 6.0 4.3 3.0 6.7 6.7 6.3 2.3 2.3 3.7 4.0 3.0 4.0 5.0 5.3 4.7 3.7 6.3 5.7 5.7 6.0

n refers to the number of participants, out of 74, who identied the risk. Importance is the mean ranking of importance by three judges, on a scale of 1 (not important at all) to 7 (very important).

Panel C: Deciencies
Description of Deciency D1. Should have an approved supplier list D2. Should perform periodic inventory counts D3. Should compare batch totals of inventory record adjustments to receiving reports D4. Invoices should be stamped when recorded D5. There should be follow-up of purchase order numbers D6. Invoices should be cancelled once paid D7. Should perform a periodic reconciliation of supplier statements D8. Bank reconciliations should be done D9. A batch summary of checks paid should be compared to A/P posting D10. Receiving reports should be pre-numbered with follow-up D11. There should be periodic follow-up of open order le D12. Disbursements should be made on due date less processing time D13. Software programs should be password protected D14. Warehouse access should be restricted D15. There should be examination/follow-up of quality of goods D16. Should review signed checks before sending out or checks should have a 2nd signature n 26 8 7 0 16 11 4 8 1 5 12 1 3 10 5 27 Importance 3.7 6.7 6.0 5.0 4.0 4.3 5.3 7.0 4.7 4.7 4.3 4.0 6.3 6.3 4.3 6.0 (continued on next page)

Behavioral Research In Accounting Volume 24, Number 1, 2012

84

Morrill, Morrill, and Kopp

TABLE 2 (continued)
Description of Deciency D17. Should review minimum re-order levels D18. Should compare prices/get bids D19. Should restrict access to check-signing machine D20. Should follow up incorrect orders D21. Should ensure prompt recording/proper cutoff at year end D22. Should follow up disbursement vouchers D23. Should have control over goods returned D24. Should have control over invoices reaching A/P, such as having a list of invoices that have been forwarded D25. Should have segregation of purchase authorization and receiving for ALL purchases D26. Should review all aborted payments n 4 7 3 0 4 1 1 4 17 2 Importance 3.3 3.7 6.3 4.0 5.3 5.0 4.0 4.3 4.0 2.7

n refers to the number of participants, out of 74, who identied the deciency. Importance is the mean ranking of importance by three judges, on a scale of 1 (not important at all) to 7 (very important).

individual participants identied 5.50 different controls, 8.50 different risks, and 2.53 different deciencies. It is important to note that there is not a one-to-one mapping of risks, controls, and deciencies, that is, the number of deciencies is not necessarily equal to the number of risks less the number of controls. This is because one control may address more than one risk. For example, management review of documentation before payment could prevent unauthorized purchases, incorrect payment amounts, and/or unsupported payments. Similarly, risks may be addressed by more than one control. Unauthorized purchases may be prevented by both a purchase requisition system and by management review of documentation before payment. H1 deals with the task order effect on the number of deciencies identied. ANCOVA tests for this hypothesis, controlling for city, course, and months of experience, are presented in Table 4. Participants in the risks-rst condition identied 2.92 deciencies on average, compared to 2.11 in the controls-rst condition. This difference is signicant at p , 0.01, strongly supporting H1. To assess the importance of deciencies identied for RQ1, we had three judges rate the importance of the risks, controls, and deciencies in our experimental case. One judge was a partner of risk advisory services for a Big 4 public accounting rm; the second judge was an internal auditor of a large publicly traded rm who had extensive experience in the design and evaluation of internal control systems; and the third judge had taught courses in internal control, which included the evaluation of internal control eld projects performed by students registered in the course. Each judge assessed importance on a scale of 1 (not important at all) to 7 (very important).7 The importance of each risk, control, and deciency was then measured by the mean assessment of the three judges. As can be seen from Table 2, the importance scores of all the strengths, deciencies, and risks varied widely. An important justication for our research question is that it gives us a better indication of the effectiveness of the internal control evaluations. While there were several deciencies in the system, some were signicantly more important than others. Therefore, an effective control evaluation would identify both a signicant number of deciencies, and important
7

Importance was not explicitly dened for the judges. They were told to assess the importance based on whatever they would be most concerned with in the situation of being an advisor to the company.

Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

85

TABLE 3 Descriptive Statistics

Variable CONTROL RISK DEFICIENCY CONTROLIMPORT RISKIMPORT DEFICIENCYIMPORT n 74 74 74 74 74 74 Minimum 0 0 0 0 0 0 Maximum 11 17 6 56.33 85.33 29.33 Mean 5.50 8.50 2.53 30.29 42.30 12.28 Standard Deviation 3.19 2.99 1.72 15.82 15.46 8.24

CONTROL, RISK, and DEFICIENCY are the number of distinct controls, risks, and deciencies, respectively, identied by each of the participants. For the CONTROLIMPORT, RISKIMPORT, and DEFICIENCYIMPORT variables, the importance of each control, risk, and deciency was rated on a scale of 1 (not important) to 7 (very important) by three judges, and a mean rating was computed. CONTROLIMPORT, RISKIMPORT, and DEFICIENCYIMPORT are the sums of those importance ratings of the controls, risks, and deciencies, respectively, identied by each of the participants.

ones. For this reason, our variable adds the deciencies identied, effectively weighted by their importance. We constructed deciency importance variables (DEFICIENCYIMPORT ) for each participant equal to the sum of the mean importance scores of the deciencies identied by that participant.8 The mean importance scores, presented in Table 3, were 30.29, 42.30, and 12.28 for controls, risks, and deciencies, respectively. To investigate RQ1, the internal control deciency importance scores (DEFICIENCYIMPORT ) of the risks-rst group were compared with those of the controls-rst group. ANCOVA results, controlling for city, course, and months of experience, are presented in Table 4. The mean importance score of the risks-rst group was 13.99, compared with 10.47 for the controls-rst group, which is statistically signicant at p , 0.01. Our importance scores are based on both the importance of the item identied and the number of items identied. Therefore, a participant could theoretically achieve a high importance score by identifying a large number of less-important controls. As a check on the validity of our results, presented in Table 2, we identied and analyzed only those controls and deciencies that were judged as having above-median importance (greater than or equal to 5 on a seven-point scale) by all three judges. This resulted in a possible set of seven deciencies and ve controls. Similar to our reported analysis, the risks-rst group identied marginally, but signicantly, more of these deciencies but signicantly fewer of these controls. It is also interesting to note from Table 4 that the risks-rst group identied signicantly fewer controls than the controls-rst group, a result that we had not anticipated. Specically, the risks-rst group identied 3.47 controls versus 7.64 controls identied on average for the controls-rst group. The importance of the controls identied was also signicantly different. It may be that the task ordering effectively induced a framing effect. Emby (1995) found that auditors recommended more (less) extensive substantive testing, indicating less (more) reliance on internal controls, when instructed to evaluate the risk (versus strength) of the internal control system. We did not expect a framing effect in our experiment, as the wording we used was identical in both task conditions.
8

The list of identied risks, controls, and deciencies, with mean importance scores, are presented in Table 2. A participant who identied only risks R1, R2, and R3 (see Panel B of Table 2), for example, would be assigned a RISKIMPORT score of 6.7 3.7 4.0 14.4.

Behavioral Research In Accounting Volume 24, Number 1, 2012

86

Morrill, Morrill, and Kopp

TABLE 4 Cell Means by Task Order Condition

Mean (Standard Deviation) of Risk First Condition (n 38) DEFICIENCY RISK CONTROL DEFICIENCYIMPORT RISKIMPORT CONTROLIMPORT 2.92 (1.63) 8.63 (2.94) 3.47 (2.15) 13.99 (7.67) 42.92 (15.42) 22.40 (13.67) Mean (Standard Deviation) of Control First Condition (n 36) 2.11 (1.74) 8.36 (3.07) 7.64 (2.67) 10.47 (8.54) 41.64 (15.69) 38.61 (13.62)

F (df 1,59) 11.33*** 1.72 39.76*** 8.16*** 0.90 16.75***

*, **, *** Indicate signicance at p , 0.10, p , 0.05, and p , 0.01 levels, respectively. CONTROL, RISK, and DEFICIENCY are the number of distinct controls, risks, and deciencies, respectively, identied by each of the participants. For the CONTROLIMPORT, RISKIMPORT, and DEFICIENCYIMPORT variables, the importance of each control, risk, and deciency was rated on a scale of 1 (not important) to 7 (very important) by three judges, and a mean rating was computed. CONTROLIMPORT, RISKIMPORT, and DEFICIENCYIMPORT are the sums of those importance ratings of the controls, risks, and deciencies, respectively, identied by each of the participants. F is the result of ANCOVA tests of the risk-rst versus control-rst effect, controlling for City and Course (xed effects), and months of experience as a covariate.

Even so, our results were consistent with such a framing effect. In Embys (1995) case, losses were averted by increasing substantive testing. In our case, our losses may have been averted by concentrating on internal control deciencies more than concentrating on internal control strengths. Our nding that participants in the risks-rst condition identied fewer controls also has important implications. While our hypothesis that more deciencies and more important deciencies would be reported by the risks-rst group was supported, it is perhaps surprising to note that the number of risks generated by the two groups was not signicantly different; the risks-rst group generated 8.63 risks while the controls-rst group generated 8.36 risks, but this may be an artifact of our experimental design. Since they identied more controls, they also naturally identied more risks. However, it is an illustration of our theory that the risks generated did not lead to a more effective identication of deciencies, as these risks were already controlled, and they were unable to generate more risks (that were not controlled) in the subsequent step. Therefore, the controls-rst group did identify almost as many risks as the risks-rst group did, but ultimately did worse in identifying deciencies. Supplemental Analysis We were surprised that overall, our participants identied only 5.5 out of 20 controls, 8.5 out of 40 risks, and 2.53 out of 26 deciencies, and the minimum number of controls, risks, and deciencies identied across all participants was zero. While this does suggest that the process of deciency identication is difcult, it also calls into question whether our participants were
Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

87

sufciently engaged or competent to perform this task. Participants appeared to be engaged, and in the session in which we measured time, all participants spent at least 40 minutes on the task. None of the participants had a zero score on more than one variable, so we concluded that none needed to be eliminated because of non-engagement. With respect to performance, our participants did not rate the task as being especially difcultparticipants in the two conditions ranked the task as 5.1 and 5.5 on an 11 point scale, where 0 was not difcult and 11 was very difcult. Furthermore, it is important to note that the total number of deciencies (the denominator of our ratio) is somewhat overstated, as the list includes any valid deciency that any of our participants generated. For example, several participants identied that two signatures were not required for checks under $200. Our judges, recognizing that this control was implemented due to cost-benet considerations, did not rate this as an important deciency. This is the reason that we performed our second set of analyses where each participants responses were weighted by the importance of the risks, controls, and deciencies that they identied. Therefore, the participant would be considered to have generated a deciency for the purposes of testing H1, but the participant would receive a lower score, all other things being equal, on the DEFICIENCYIMPORT variable used to test RQ1. However, this coding scheme meant that we did have a very large number of possible deciencies, and it would not be surprising that participants did not identify many of them, as many were not important. Nevertheless, we did perform some tests to determine if our participants were sufciently competent for our study to be valid. We rst investigated the performance and results of our subgroups. As previously noted, the experiment was administered to a total of 78 participants on three separate occasions, to accountants attending sessions at the end of CGA Auditing 1 or Auditing 2 courses in two different cities. The number of risks identied and the number of controls identied did not differ signicantly between the groups (City 1 versus City 2 and Audit 1 versus Audit 2 students). On average, however, the City 1 participants generated signicantly more deciencies than did the City 2 participants (mean 3.26 versus 2.22, p , 0.05), and the Audit 1 participants, somewhat surprisingly, generated signicantly more deciencies than did the Audit 2 participants (3.53 versus 2.43, p , 0.04). However, even though some groups appeared to perform better than others, the overall results of our study hold within each City and Course group. LIMITATIONS, DISCUSSION, AND CONCLUSIONS This study examined the effect of the order of internal control evaluation sub-tasks on the quality of that evaluation. Interference theory suggests that auditors who generate risks before identifying system controls will identify more risks. We argue that identication of more risks, in turn, helps the auditor to identify more deciencies in the control system. Conversely, prior identication of controls will interfere with the auditors ability to generate risks and, therefore, to identify deciencies. Our experimental results are consistent with this argument. Participants that identied risks rst identied more deciencies. However, these risks-rst participants also identied fewer controls than participants that identied the controls rst. In other words, the risks-rst participants tended to under-identify controls that were present. These results indicate that there could be a trade-off between audit efciency and audit effectiveness. If auditors fail to identify deciencies in the clients system, this undermines the effectiveness of the audit. The auditors failure to identify deciencies may imply that errors are present in the nal audited nancial statements. Secondarily, the auditor cannot identify potential improvements to be made to the clients system, which are often a signicant component of the value-added services provided by the audit. Our study is subject to the usual limitations regarding experimental conditions, although, as noted, economic incentives were put in place and participants appeared to be engaged in the task. A
Behavioral Research In Accounting Volume 24, Number 1, 2012

88

Morrill, Morrill, and Kopp

particular limitation of this study, however, is that the nature of the task materials may have imposed some structure on the task. Since the participants had to ll out risks, controls, and deciencies in a tabular format, they were perhaps more organized in their approach than they would otherwise be. Additional research into the effects of such a structured approach would be a useful extension of this research. Another limitation is that our importance scores relied on assessments made by three judges. To ensure that our results were as valid as possible, we rst compared the judges assessments to ensure that there was a reasonable level of consensus. The pair-wise correlations of the three judges scores were all statistically signicant (at the p , 0.10 level between judges 1 and 2; at the p , 0.05 level between judges 1 and 3; and at the p , 0.01 level between judges 2 and 3). We also performed all analyses using the assessments of the individual judges rather than the average assessments and found qualitatively similar results. Our results suggest that audit effectiveness can be improved by identifying the risks to be addressed by the system before analyzing the controls. However, participants following this sub-task order tended to under-identify controls. This hinders audit efciency, as the auditor will then ultimately place less reliance on the internal control system in place than could be otherwise justied. A possible solution to this problem would be to employ both strategies: One member of the audit team could follow a risks-rst task order, while another member follows a controls-rst task order. Alternatively, the reviewer could follow a different task ordering than the subordinate. Research into the effects of a combined strategy of this sort could be of value to practitioners. Our results also have implications for standard setters. Current auditing standards do not specify the order in which auditors perform the sub-tasks comprising the internal control evaluation, or even clearly state that auditors should generate for themselves a list of the risks related to nancial reporting. This study suggests that these are alternative specications of the internal control evaluation process that have potentially important implications for audit efciency and effectiveness.

REFERENCES
Anderson, J. C., S. E. Kaplan, and P. M. J. Reckers. 1992. The effects of output interference on analytical procedures judgments. Auditing: A Journal of Practice & Theory 11 (2): 113. Bierstaker, J. L. 2003. Auditor recall and evaluation of internal control information: Does task-specic knowledge mitigate part-list interference? Managerial Auditing Journal 18 (1/2): 90100. Canadian Institute of Chartered Accountants (CICA). 2007. Practice Engagement Manual. Toronto, Canada: CICA. Certied General Accountants Association of Canada (CGA). 2007. Public Practice Manual. Vancouver, Canada: CGA Association of Canada. Church, B., and A. Schneider. 1993. Auditors generation of diagnostic hypotheses in response to a superiors suggestion: Interference effects. Contemporary Accounting Research 10: 333350. Committee of Sponsoring Organizations of the Treadway Commission (COSO). 1994. Report of the National Commission on Fraudulent Financial Reporting. New York, NY: COSO. Committee of Sponsoring Organizations of the Treadway Commission (COSO). 2006. Internal Control over Financial Reporting-Guidance for Smaller Public Companies. New York, NY: COSO. Emby, C. 1995. Framing and presentation mode effects in professional judgment: Auditors internal control judgments and substantive testing decisions. Auditing: A Journal of Practice & Theory 13 (Supplement): 102115. Frederick, D. M. 1991. Auditors representation and retrieval of internal control knowledge. The Accounting Review 66 (April): 240258. Behavioral Research In Accounting Volume 24, Number 1, 2012

Internal Control Assessment and Interference Effects

89

Heiman-Hoffman, V., D. Moser, and J. Joseph. 1995. The impact of an auditors initial hypothesis on subsequent performance at identifying actual errors. Contemporary Accounting Research 11 (2): 763779. International Auditing and Assurance Standards Board (IAASB). 2007. Handbook of International Auditing Assurance and Ethics Pronouncements. New York, NY: International Federation of Accountants. Kopp, L. S., and J. L. Bierstaker. 2006. Auditors memory of internal control information: The effect of documentation preparation versus review. Advances in Accounting Behavioral Research 9: 2750. Libby, R., and D. M. Frederick. 1990. Experience and the ability to explain audit ndings. Journal of Accounting Research 28 (2): 348367. Public Company Accounting Oversight Board (PCAOB). 2007. Auditing Standard No 5. Washington, D.C.: PCAOB. Roybark, H. M. 2006. An Analysis of Audit Deciencies Based on PCAOB Inspection Reports Issued During 2005. Journal of Accounting, Ethics & Public Policy 6 (2): 125154. Slamecka, N. J. 1968. An examination of trace storage in free recall. Journal of Experimental Psychology 76: 504513. Tubbs, R. M. 1992. The effect of experience on the auditors organization and amount of knowledge. The Accounting Review 67 (October): 783801.

APPENDIX A SYSTEM DESCRIPTION USED FOR EXPERIMENTAL TASKS You are the senior in charge of an important client for your rm, Wittim Medical Supplies, Inc. Wittim manufactures a variety of medical supplies, including test tubes, thermometers, and disposable surgical garments. Your task today is the internal control evaluation of the purchasing cycle. The supplies manager initiates the purchase and maintains the inventory records. The records include reorder points for all regularly used items. The supplies manager prepares a purchase requisition on a two-part, pre-numbered form. After signing the requisition, he les one copy by requisition number and sends the other copy to the purchasing department. Requisitions for items that will cost over $100 must be approved by the production manager before being sent to the purchasing department. PURCHASING DEPARTMENT The purchasing department checks a purchase requisition for proper approval and selects a vendor. A ve-part, pre-numbered purchase order is prepared. Copies are sent to the vendor, receiving department, accounts payable, and the supplies manager. The purchasing department records the current purchase and les its purchase order and requisition copies by purchase order number in the open order le. The receiving department les its copy in a le by purchase order number. The supplies manager les his copy with its corresponding purchase requisition. RECEIVING DEPARTMENT An authorized receiving department employee counts the goods to verify that they were received, compares the count to the packing slip, and prepares a four-part receiving report. Copies of the receiving report are sent to the supplies manager, the purchasing department, and accounts payable. The receiving department les its copy of the receiving report and the packing slip with its copy of the purchase order. The supplies manager updates the inventory records when he receives the receiving report and then les the purchase order, purchase requisition, and receiving report by
Behavioral Research In Accounting Volume 24, Number 1, 2012

90

Morrill, Morrill, and Kopp

purchase order number. The purchasing department les its copy with the order in the open order le. The purchasing department receives two-part invoices from the vendors. One copy of the approved invoice is sent to accounts payable. The purchase order, purchase requisition, invoice, and receiving report are then led in the closed order le by purchase order number. ACCOUNTS PAYABLE DEPARTMENT The accounts payable department receives and matches purchase orders and approved invoices from the purchasing department with receiving reports from the receiving department. The invoices are checked for prices, quantities, and mathematical accuracy. A clerk initials them if they are accurate. When all the documents are received, a clerk posts the payable amount to the particular vendors payable account, prepares a pre-numbered disbursement voucher, and attaches it to the purchase order, receiving report, and invoice. The package is then given to the accounts payable manager for review and approval for payment. The manager gives the approved disbursement vouchers to a second clerk. The clerk batches and totals the approved vouchers and prepares a batch summary. The batch summary is sent to the accounting department. A third clerk completes a two-part, pre-numbered check for each disbursement voucher. The check and the disbursement vouchers are sent to the cashier. CASH DISBURSEMENTS DEPARTMENT The cashier totals the checks and compares the total to the batch summary. She then signs the checks with the treasurers signature using a check-signing machine. She is authorized by the board of directors to disburse company funds. She then places the rst copies of the check/remittances in envelopes and sends them to the vendors. The second copy is sent to the accounting department. The checks for aborted payments are marked to prevent their payment.

Behavioral Research In Accounting Volume 24, Number 1, 2012

Copyright of Behavioral Research in Accounting is the property of American Accounting Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.