Journal Online

Evaluating and Selecting Sarbanes-Oxley Software

Roberta ann Barra, ph.d., cpa, is an assistant professor of accounting at the University of Hawaii at Hilo (USA). She can be reached at roberta. barra@hawaii.edu. arline Savage, ph.d., ca, is a professor of accounting at Cal Poly San Luis (California, USA). She can be reached at savage@calpoly.edu. mark G. Simkin, ph.d., is a professor of information systems at the University of Nevada (USA). He can be reached at markgsimkin@ yahoo.com.

It has been six years since the US Sarbanes-Oxley Act was signed into law, yet many of the software products designed to handle the laws complex demands are still maturing as accountants and auditors continue to require more sophisticated functions. This article reviews a selection of software products in this market and assesses how well they work. Complying with the Sarbanes-Oxley Act of 2002 costs the average company US $7.8 million and 70,000 hours of employee time.1 In response to the complex record keeping and auditing requirements of Sarbanes-Oxley, a number of companies have developed software products to automate these tasks. These products can be classified into four major categories:2 1. Data manipulation software 2. Document and workflow management software 3. Risk analysis and risk management software 4. Control self-assessment and continuous monitoring software Of these four types of software, the last two categoriesproducts that help accountants respond most directly to Sarbanes-Oxley requirementswill be the focus of this article. This article reports the results of a study the authors conducted to examine select SarbanesOxley software products in detail. The study included the compilation of a set of SarbanesOxley products that use a COSO-driven, topdown, risk-based approach; the acquisition and hands-on testing of eight major software packages; and a detailed comparison of the many features available in them. This evaluation is meant to enable SarbanesOxley software users to compare their products with others, enable Sarbanes-Oxley software buyers to purchase software products intelligently and prompt vendors to further develop their products.

Sample Selection and evaluation method The first task was to select software for evaluation.3 Only those software products that were designed to perform a Sarbanes-Oxley compliance audit using a COSO-driven, topdown, risk-based approach were selected. Products that were no longer being marketed were eliminated from the study. Each of the software vendors on the final short list was asked to participate in the study and provide an evaluation copy of their software. To ensure a meaningful list of products, vendors were also asked to identify their major competitor(s). In most cases, their responses confirmed the importance of the major players already on the short list. As a result, a final set of eight vendors was createdeach having agreed to participate in the study. Some of these products are fairly new to the market and were developed expressly to accommodate Sarbanes-Oxley auditing and reporting requirements.4 Others have been around for some time, were originally designed for risk management and have been modified to handle Sarbanes-Oxley requirements. The software products included in the evaluation are: ControlCase Compliance Manager Methodwares Enterprise Risk Assessor OpenPages FCM Paisley Enterprise GRC Protiviti Governance Portal SarbOxPro Sirius Solutions Risk and Controls Management Software SOX Automations SOX-DISC The next step in the investigation was to identify criteria by which to evaluate the software. In this endeavor, the functionality of each product was used to help define both core capabilities and unique features. This enabled the identification of not only the commonalities among these

ISACA JOURNAL VOLUME 1, 2009

products, but also those unique features that differentiated a particular package from its peers. In cases where all software products had the same feature, that criterion was eliminated from further study. For instance, all of the products use a COSO-driven, top-down, risk-based approach, so this criterion is not included in the table. Finally, in each case, the software was tested without training in order to best judge ease of use. (In general, the authors do not recommend such an approach to end users, who would obviously benefit from such education.) ReSultS All of the software products described here can perform the basic evaluation tasks consistent with Sarbanes-Oxley compliance requirements. Similarly, all of them run on PC microcomputers using standard Windows or Vista operating systems. Comparatively speaking, none of them is particularly expensive, especially for mid-sized or large companies. Other common software features include: Public Company Accounting Oversight Board (PCAOB) assertions that can be specified by the general ledger account Processes that can be identified with the general ledger account(s) A prepopulated library of risks for major business processes5 The ability to identify key controls The ability to track the history of tests on controls Automatic recording of tests of controls The ability to resuse a single control in many different places, with the software allowing the user to test the control only once Support for multiple entities/locations with either the same controls or different controls The ability to export data to Excel The ability to prevent any user from freely accessing control weaknesses What distinguishes these products from one another? Figure 1 (included at the end of the article) provides the results of the inquiry. The software vendors were allowed to check this report for accuracy. However, the opinions contained in the table evaluations remain those of the authors.6 Figure 1 lists products alphabetically in the columns, with no ranking implied by the authors in this summary. Within the body of the table, the term yes means that the software

quickly and easily performs the task or function indicated, while a no means the opposite. (Note: in most cases, the software vendors are willing to customize their products to overcome such deficiencies, but at additional cost to the user.) In a few instances, it was found that the software could perform a task only with some additional effort from the user. In such instances, a rating of OK was assigned for this feature. Thus, an OK rating is relative to the other products in the lineup. In figure 1, the criterion Does it show work papers or provide link to the same? under Tests of Controls requires further explanation. There is a difference between work papers that are actually embedded in the Sarbanes-Oxley software and work papers that are stored elsewhere and accessed by links. A yes indicates that one or the other occurs, and that a user can obtain work papers without concern for how this is done. A both indicates that users can choose which manner of storage they prefer. For example, linked work papers might work better for auditors preferring to store such documents elsewhere. For the Reporting criterion in the table, a rating of yes for e-mail notification means that specific users and/or other users will be notified when reports are generated by the software. These may, for example, be forms these users need to complete for their control evaluation tasksa feature not commonly found in financial accounting software. Integration is a feature that works differently in this software than in financial accounting software. In financial accounting software, one looks for integration between the ledgers and the journals. In contrast, Sarbanes-Oxley legislation requires an evaluation of controls over financial reporting, and, eventually, a company must assess its overall control process. Consequently, for Sarbanes-Oxley software, integration is needed between the evaluation of the controls and the overall evaluation of the process that the controls are designed to protect. All of these packages use a topdown, risk-based approach and finish by evaluating the controls. Therefore, the authors wanted to see whether the software used those evaluations to help the user evaluate the effectiveness of the overall process or, indeed, whether the overall process was even evaluated. If users had to evaluate the overall process themselves without assistance from the software, integration was rated lower. If the software aided the user in this evaluation, the integration was rated higher.

ISACA JOURNAL VOLUME 1, 2009

If the software did not accommodate rating the overall process, this is noted in the evaluation. While the integration criterion was primarily aimed at the controls/process evaluations, other integration was also looked for within each software product. Therefore, a package that does not rate the overall process could achieve an acceptable level of integration based on other aspects of its software. The evaluation table (figure 1) only provides detailed information about the capabilities of the various software products that differed. All of these products have important, useful features in common that, for the sake of brevity, were not included in this table. diScuSSion of ReSultS By pRoduct Figure 1 lists the comparative findings. The following sections discuss each of these products in turn and provide some additional insights that were gathered from the study. (Again, the sample products are presented in alphabetical sequence.) ControlCase ControlCase is free for those using the PC-based version. Although all the packages assess process, risk and controls, the softwares remediation module also allows one to document what is going to be done about a problem it has identified. Other packages have similar capabilities, but are not as complete or as well specified as this software. The software also time- and date-stamps documents to provide evidence of the chain of custody for documents that are part of a particular processa feature particularly important for legal evidence. This product is suitable for evaluating individual controls. One possible concern to the users of this product may be ease of use. Methodware Methodware has been available since 1998, and it is a mature, sophisticated and full-featured package. The user interface is different, interesting and (to the authors) visually pleasant. Developed by former employees of Ernst & Young, New Zealand, this software allows users to perform all of their own risk assessmentsnot just those for SarbanesOxley compliance. The software also performs Monte Carlo simulations, thereby eliminating a need for such assessments by the auditors or the managers. The software also includes a tool that that allows users to build their own business rules and customize the software to suit their firms requirements.

A unique feature of Methodware is that not all employees need be registered users to participate in Sarbanes-Oxley compliance. E-mail notification with drop-down menus that match the user interface can be sent to nonregistered users for control assessment. These can be completed and sent to registered users for immediate integration into the software. This earns top honors for Methodware when it comes to innovative thinking with respect to product integration and pricing, as fewer registered users reduce costs. This software is suitable for evaluation of controls and the overall process. OpenPages OpenPages is one of the leaders in this market. The firms president has an accounting degree and this background shows in the software. The company advertises itself as enterprise software and this places it at the high end of the spectrum. But, whereas one would expect an enterprise version of software to be difficult to use, this software has an intuitive user interface and is easy to use. The authors particularly like OpenPagess incident and issues reporting method. Incidents happen in a firm, apart from those related to Sarbanes-Oxley compliance, and this software allows users to document those incidents as they occur, so they can be considered during Sarbanes-Oxley testing. This is important and can be critical to Sarbanes-Oxley tests. The software also comes with an application programming interface (API) that allows OpenPages to synchronize with general ledger software or link with other software applications such as ACL or IDEA. Thus, this software is suitable for evaluation of controls and the overall process. Paisley Paisley is a software product that was initially offered as a general risk-based product in 1995. Consequently, this is a more mature product than some of the others in this sample. The president and founder was formerly an internal audit manager and this focus is evident in the software, which has the ability to add new risks and controls. It also has an excellent built-in library of controls. This software is available in six different languages and has two methods of deployment: the users desktop or Paisley hosting of the application. This is another enterprise-level software package, yet as with all products, the authors believe there is always room for improvement. The help menus were not particularly helpful and could be improved, although the authors recognize that
ISACA JOURNAL VOLUME 1, 2009

effective training may eliminate the need for excellent help menus. Also, out of the box scoring of the materiality of the process would be a nice feature. This can be done with some customization or user-designed fields, but standardization of such tools would be beneficial. Protiviti Protiviti is known for its consulting skills, but the company has also developed its own Sarbanes-Oxley software. The software does an excellent job of identifying risks, and users will be hard pressed to come up with new risks to add to those already built into the software (although it is relatively easy to add others, if necessary). Areas to consider for subsequent versions include user ability to more easily load a trial balance (currently, each balance has to be input by keying in the data), assessment of inherent risk for the overall process and making the reporting function easier to use. Finally, the products Sarbanes-Oxley compliance integration could be strengthened. During the demo, it was observed that test results can be marked as ineffective when the underlying tests are marked as effective, because the software does not automate the evaluations conducted at various levels. SarbOxPro SarbOxPro is a relatively new product that was developed specifically for evaluating Sarbanes-Oxley compliance for small to mid-sized companies. It is free to users, who can download it from the Internet. It also has a very easy-to-use interface that runs using Microsoft Access. For a relatively trivial price, users can also get training and support for the software, which the authors would highly recommend. The package the authors evaluated lacked some of the bells and whistles of the enterprise-class software packages, but the functionality is sufficient to do a basic Sarbanes-Oxley compliance evaluation. Users will not be able to do workflow analysis, for example, but will be able to do workflow in another application and link to the relevant file within SarbOxPro. Of all of these packages, in the authors opinion, this software does the best job of measuring inherent risk for a process. This is extraordinarily good software for the lower end of the market, and the price cannot be beat. One suggestion for improvement is to better develop the capability for users to develop custom reports.

Sirius Solutions Sirius Risk and Controls Manager (Sirius RCM) is another product that is built on an Access database. The initial user interface was somewhat simplistic, but users should understand that the actual controls repository and controlstesting portions of the software are more sophisticated. What it does, it does just as well as any of the other lower-end products, and it is relatively easy to use. This software does a better job of documenting the controls than documenting processes. But, users who understand Microsoft Access can modify the software as they see fitfor example, to evaluate controls compliance or perform process evaluations. SOX Automation SOX-DISC is a well-designed, competent software product. It is intuitive to use, clearly designed for accountants, and even includes guidelines in each major module to identify the relevant Generally Accepted Accounting Principles (GAAP), standards or guidelines for that module. This is handy for training and staying on top of compliance issues. The risk/ control matrix includes an extensive amount of information with even more reporting behind the matrixan exceptional design element. This software is very mature, well conceived and has everything one could ask for in a Sarbanes-Oxley product with one exception: it lacks customization. That is, it has only prespecified screens and reports, which are, nevertheless, very good. But, the vendor appears willing to work with users and should be highly responsive to user needs. A web-based product is under development and should be available soon. makinG a choice In one sense, all of the products essentially do the same thingautomate many of the tasks required to comply with the Sarbanes-Oxley Act. For example, they all have the ability to document a process, identify the risks and controls associated with that process, test those controls, and generate reports. Sarbanes-Oxley legislation is relatively new and, for this reason, Sarbanes-Oxley software is relatively young. Accounting professionals who are used to mature accounting software may be disappointed by some of the features, or lack thereof, in this emerging market. Products that show the best advantages are usually retrofitted productsi.e., products that started as risk management software and

ISACA JOURNAL VOLUME 1, 2009

then expanded with Sarbanes-Oxley capabilities when the legislation presented the opportunity. Unquestionably, these more mature products have much to offer to the consumer, especially to larger firms looking for a total solution to their risk management problems. But, not every firm requires such a solution, and many firms may want a Sarbanes-Oxleyonly package that has some of the same features they have come to expect from their financial accounting software packagesthat is, good integration, reporting ease and a user-friendly interface. As with all young software markets, this is a rapidly evolving market and new software versions come out every few months. In the future, the authors expect to see more and better integration, enhanced reporting capabilities, greater flexibility in user abilities to design custom reports, and better and more user-friendly interfaces. Thus, there should develop a complex synergy between the Sarbanes-Oxley vendors and the auditors and accountants involved in Sarbanes-Oxley compliance. This should also, over time, help the Sarbanes-Oxley software market grow and mature. For now, those involved in Sarbanes-Oxley compliance efforts should find the software discussed here well worth the price. For example, such efforts require evaluating and documenting internal control processestasks that these software packages make relatively painless. Also, the relative youthfulness of this market should not detract from the fact that the software can assist users today with a task that most would prefer not to do manually or with software designed for other tasks. The purpose of this evaluation was to provide information that can assist firms in making their own best choices. It also provides criteria and information that should be useful

in evaluating competing software that was not included in this evaluation. Additionally, every organization has unique requirements, and the authors do not discount the importance of customer service and trainingattributes that were not included in this evaluation. As with other types of software, picking the best product usually means finding the product with the features required for the particular organization (e.g., a particular language option), rather than picking a product with some kind of overall best scorea rating the authors deliberately chose to avoid making here. Nonetheless, there is a software package available that is suitable for every firm. endnoteS Lacy, Sarah; The Sarbanes-Oxley Software Race, Business Week Online, 12 July 2005 2 Bagranoff, Nancy A.; Laurie Henry; Choosing and Using Sarbanes-Oxley Software, Information Systems Control Journal, vol. 2, 2005, www.isaca.org/archives 3 Ibid. and Brooks, D.; M. Goldman; R. Lanza; 2006 Buyers Guide to Audit, Anti-Fraud, and Assurance Software, Ekaros, 2006 4 Shein, Esther; Thinking Inside the Sarbox, CFO, 22(5), 2006, p. 32 5 Every vendor in the sample reported that most public entities want to use their own controls and risks, rather than the controls and risks available in these libraries. 6 The authors also note that, because software is frequently updated, a feature that was identified as missing in figure 1 may now be available.
1

ISACA JOURNAL VOLUME 1, 2009

figure 1Sarbanes-oxley Software evaluation table (cont.)

vendor Software with an excellent rating would receive the following responses: controlcase
controlcase compliance manager

methodware

openpages

paisley

protiviti
protiviti Governance portal

Sarboxpro

Sirius Solutions
Risk and controls management Software (Rcm)
infor@sirsol.com www.sirsol.com

SoX automation

Software E-mail Address Web Site

contact@controlcase.com www.controlcase.com

advisor@methodware.com www.methodware.com

enterprise Risk assessor (eRa)

openpages fcm
info@openpages.com www.openpages.com

paisley enterprise GRc

support@paisley.com www.paisley.com

contactus@protiviti.com www.protiviti.com

sales@sarboxpro.com www.sarboxpro.com

Sarboxpro

gladys@soxautomation.com www.soxautomation.com

SoX-diSc

Sarbanes-oxley compliance Software criteria documenting processes Can one or more trial balance be imported? Can materiality levels be specified by G/L account? Audit trail of changes to processes? identifying Risks Can new risks be added by user? Can they be added easily? Can risks be deleted by user? Can they be deleted easily? Can inherent risk be specified for a process? Can inherent risk be scored with materiality for a process? identifying controls Are controls identified for major processes? Can new controls be added by user? Can they be added easily? Can controls be deleted by user? Can they be deleted easily? Is there a prepopulated library of controls? For multiple regulations or Sarbanes-Oxley only? For CobiT? tests of controls Does it show work papers or provide link to same? Are separation of duties highlighted? Can control deficiency be assessed? Can control deficiency materiality be assessed? Does it link to ACL/Idea for continuous control monitoring? Reporting Are the controls effective? Is there a risk and control matrix? Is there deficiency reporting? Is there drill-down capability? Is there whistle-blower capability? Is there e-mail notification? Is there automatic follow-up on notifications? Is there custom reporting? ease of use Was the software easy to use? Are the screens intuitive for an accountant? Can screens be customized? Can views be changed by task/ user? Yes Yes Yes Yes OK OK Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes OK OK OK No Yes, except for reporting Yes, except for reporting Yes Yes Yes Yes No No Yes Yes Yes Yes Yes Yes No No Yes Yes Yes Yes Yes Yes Yes Yes Yes OK Yes OK Yes Yes Yes Yes Yes Yes Yes OK No Yes Yes OK Yes Yes Custom report Yes No Yes Yes Yes Yes Yes Yes OK No OK Yes Yes Yes Yes Yes Yes No Yes Yes Yes Custom report Yes Yes No No No N/A OK Yes Yes Yes Yes No No No Yes Yes Yes Yes Limited No Yes Yes No Yes Yes Yes Yes Yes Yes No Yes No No Yes Yes Yes Yes Links can be built Both Can be built OK OK With API Yes No Yes Yes Yes Both Yes Yes OK Via API Yes Yes Yes Yes No Yes Yes Yes Yes No Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes Multiple Yes Limited Yes OK Yes Yes OK Multiple Yes Yes Yes Yes Yes Yes Yes Multiple Yes Yes Yes Yes Yes Yes Yes Multiple Yes Yes Yes Yes Yes Yes Yes Multiple Yes Yes Yes Yes Yes Yes Yes Multiple Yes Yes Yes Yes Yes Yes Yes SOX only Yes Yes Yes Yes Yes Yes Yes Multiple Yes Yes Yes Yes Yes Yes OK SOX only No Yes Yes Yes Yes Yes Yes Yes OK Yes OK OK No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes OK Yes Yes Yes Yes Yes OK Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Can be built in Access by user Can be built in Access by user Yes Yes Yes Yes Yes Yes Yes Yes Yes No No Yes Yes Yes Yes Yes OK Yes Yes Yes Yes OK OK Yes Yes Yes No Can be built in Access by user Can be built in Access by user Yes Yes Yes Yes

ISACA JOURNAL VOLUME 1, 2009

figure 1Sarbanes-oxley Software evaluation table (cont.)

vendor Software with an excellent rating would receive the following responses: controlcase
controlcase compliance manager

methodware

openpages

paisley

protiviti
protiviti Governance portal

Sarboxpro

Sirius Solutions
Risk and controls management Software (Rcm)
infor@sirsol.com www.sirsol.com

SoX automation

Software E-mail Address Web Site

contact@controlcase.com www.controlcase.com

advisor@methodware.com www.methodware.com

enterprise Risk assessor (eRa)

openpages fcm
info@openpages.com www.openpages.com

paisley enterprise GRc

support@paisley.com www.paisley.com

contactus@protiviti.com www.protiviti.com

sales@sarboxpro.com www.sarboxpro.com

Sarboxpro

gladys@soxautomation.com www.soxautomation.com

SoX-diSc

Sarbanes-oxley compliance Software criteria Does the software ensure predefined and uniform data capture? Is there search capability? Is built-in help function adequate? General considerations To what size firm is it marketed? Number of concurrent users? Are external documents linked or embedded? Is there a multilanguage user interface? N/A N/A Link Yes Low to midlevel 500+ Both No Low to high-end (three versions) 500+ Both Italian and Spanish Mid to highend 4,000+ Both Chinese, French, German, Japanese, Spanish Mid to highend 1,000+ Both French, German, Japanese, Portuguese, Spanish More than US $75,000 OK Web-based Yes Mid to highend 200 + Both Japanese, Portuguese, Spanish. Chinese and French pending. From US $40,000 upwards OK Web-based Yes Low to midlevel 5 Link No Low-end 15+ Link No Low to mid-level 200+ Both Chinese and English Yes Yes Yes Yes Yes OK Yes Yes OK Yes Yes Yes Yes OK No Yes Yes OK Yes Yes Yes OK Yes Yes Yes In Reports No

Pricing

N/A

Free

US $25,000US $75,000 Very high Both No

More than US $75,000 OK Web-based Yes

Free to less than US $5,000 High PC-based N/A

US $5,000US $25,000 OK PC-based N/A

Up to US $75,000

Overall level of integration Is it web- or PC-based? If embedded, can changes be tracked to (in) embedded documents? Can data be imported from Excel? Does software provide use survey capabilities? Security Does it include role-based security? What works really well?

High User preference Yes

OK Both Yes

High Both Yes

Yes Yes

Yes Yes

Yes Yes

Yes Yes

OK Yes

Yes Yes

Yes No

Yes Add-on tool

Yes, by vendor at no extra cost No

Yes

Yes Remediation module and support

Yes Risk assessment, Monte Carlo simulation, e-mail notification; integration; not all users need to be registered

Yes Incident and issue reporting, user interface

Yes Six languages, good library of controls, methods of deployment

Yes Excellent identification of risks, library of risks

No Extremely easy user interface, risk identification, control details, price, standard reports

Yes Documentation of tests of controls, help menus, easy to use

Yes Risk control matrix, user interface that is consistent and easy to use

A yes means the product does the task right out of the box without configuration or customization. A no means users must ask the vendor to configure this for them and this will cost extra. An ok means the product will do the task, but will not perform it as well or as easily as some of the other products. Thus, an ok is relative to the other products in the review.

The ISACA Journal is published by ISACA. Membership in the association, a voluntary organization serving IT governance professionals, entitles one to receive an annual subscription to the Information Systems Control Journal. Opinions expressed in the ISACA Journal represent the views of the authors and advertisers. They may differ from policies and official statements of ISACA and/or the IT Governance Institute and their committees, and from opinions endorsed by authors employers, or the editors of this Journal. ISACA Journal does not attest to the originality of authors content. 2009 ISACA. All rights reserved. Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, Mass. 01970, to photocopy articles owned by ISACA, for a flat fee of US $2.50 per article plus 25 per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the association or the copyright owner is expressly prohibited. www.isaca.org
ISACA JOURNAL VOLUME 1, 2009