Blackwell Publishing Ltd.Oxford, UK and Malden, USAIJAUInternational Journal of Auditing1090-67382005 Blackwell Publishing Ltd.2005917990Original ArticlesSystem Controls Reliability and Assessment EffortR. J. Daigle, T. Kizirian and L. D. Sneathen, Jr.
International Journal of Auditing Int. J. Audit. 9: 79–90 (2005)
System Controls Reliability and Assessment Effort Ronald J. Daigle1, Tim Kizirian2 and L. Dwight Sneathen, Jr.3 1
Louisiana State University California State University, Chico 3 Georgia Southern University 2
This study investigates whether internal control reliability affects information system audit hours and fees. Based on archived workpaper data from 60 clients of an international auditing firm we provide quantifiable evidence that welldesigned internal controls that are in place and operating effectively lead to decreased information system control assessment effort and lower assessment fees. The information system audit fee savings we document may be considered a significant benefit by companies determining the merit of an investment in information system controls. Key words: System controls, information system auditing, audit fees, audit effort, Sarbanes-Oxley Act.
SUMMARY This study uses information system audit workpaper data of a Big 4 auditing firm to investigate whether well-designed and documented system controls influence both information system audit effort (hours) and fees. We posit that when an information system auditor encounters a firm with a well-documented and designed system, fewer assessment hours are required to complete the controls reliability evaluation, thereby reducing information system audit effort and fees. The results of our study indicate that both the general controls strength assessment and application controls strength
Correspondence to: Tim Kizirian, Associate Professor, California State University, Chico, COB, AMIS (011), Chico, CA 95929, USA. Email:
[email protected] Data availability: A confidentiality agreement with the datagranting firm precludes revealing the firm’s identity or disseminating the data.
assessment (proxies for reliability) are both inversely related to information system audit hours and fees. While prior literature and professional guidance promote effective system controls to achieve information reliability, prior studies report conflicting results as to whether internal control reliance influences either financial statement audit effort or fees. We are unaware of any study that focuses on these relationships in an information system audit setting. The continued proliferation of information technology in systems, as well as conflicting results from prior research with respect to the influence of control assessments, provide substantial motivation for this study.
1. INTRODUCTION This study investigates whether the assessment of system controls strength impacts both the information system (IS) auditor’s reliability assessment effort (hours) and fees. The
ISSN 1090-6738 © Blackwell Publishing Ltd 2005. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
80
proliferation of information technology (IT) necessitates the need for assessing the effectiveness of systems that rely heavily on IT for recording, storing, and processing data and reporting information. Utilizing IS audit workpaper evidence from 60 clients of a Big 4 firm, we find an inverse relationship between general and application control strength assessments (a proxy for reliability) and IS audit hours and fees. Our results show that reliable system controls result in related IS audit cost savings, which can be interpreted as a tangible return on effective investment in IT. The importance of effective system controls has long been recognized, specifically with the issuance in 1992 of the Committee of Sponsoring Organizations (COSO) Internal Control framework, which serves as a basis for IS auditing. The accelerated integration of IT within financial systems has continued since the issuance of the COSO framework, and standard setting bodies have responded with new compliance requirements found in both the Sarbanes-Oxley Act of 2002 (SOX) in the United States and ISA 315 from the International Auditing and Assurances Standards Board (IAASB). Section 404 of SOX requires management of SEC registered companies to make an annual assertion concerning the effectiveness of internal control over financial reporting procedures, and for the company’s external auditor to provide assurance on these assertions. Upon passage of SOX, affected companies immediately recognized the substantial effort and costs, including that for IT, needed for complying with Section 404. While ISA 315 does not require documentation of the effectiveness of internal control to the extent SOX does, it does discuss the importance of understanding internal control in a similar fashion. Similar to SOX, ISA 315 utilizes the COSO framework as a foundation for control evaluation (ISA, 2004). The IAASB also recognizes the need for IT specialists within the audit process to obtain a sufficient understanding of internal controls when clients have significant automation of their systems. Our results indicate that the assessed strength of system controls is inversely related to both IS audit effort and fees. We reason that welldocumented system controls that are in place and operating effectively are relied upon to reduce both IS audit procedures and fees. While our results may appear intuitive, previous studies investigating relationships between financial statement level risk © Blackwell Publishing Ltd 2005
R. J. Daigle, T. Kizirian and L. D. Sneathen, Jr.
assessments (including internal control) and the amount of financial statement auditing hours and fees provide mixed results. Specifically, O’Keefe et al. (1994) and Felix et al. (2001) both fail to find an association between internal control assessments and actual audit hours and/or billed fees, while Bedard and Johnstone (2004) find a significant association between internal control assessments and planned audit hours and fees. To our knowledge no study has focused on the influence of a client’s use of IT within their systems on audit decisions. The IS audit fee savings we identify may also be considered in a larger context as a return on the effective investment in IT, thereby adding to the research literature studying benefits obtained from IT spending. Dehning and Richardson (2002) provide an extensive literature review on studies that investigate whether IT investments generate tangible returns. Early archival studies (e.g., Dos Santos et al., 1993) typically find no relation between IT spending and company performance. Despite more recent findings documenting tangible relationships between IT spending and firm performance (i.e., ROA, ROE, ROS, and inventory turnover) (e.g., Sircar et al., 2000; Chatterjee et al., 2001; Hayes et al., 2001; Dehning & Stratopoulos, 2002; Brynjolfsson et al., 2003), whether IT investments generate sufficient tangible returns continues to be debated by both researchers and professionals (Martinsons & Martinsons, 2002). To our knowledge no study has focused on tangible benefits obtained from system controls. IS audit fee savings from properly investing in strong system controls is a tangible benefit that should be considered when determining the economic feasibility of implementing IT. Accounting professionals may also leverage this client savings as a promotion for auditing and assurance services. This paper proceeds as follows. The next section outlines the literature review. Section 3 provides hypotheses development and methodology. Section 4 provides empirical results and Section 5 summarizes and discusses implications of findings.
2. PRIOR LITERATURE System controls strength and IS audit effort At most assurance firms, including the Big 4 firm that supplied our data, the primary function of Int. J. Audit. 9: 79–90 (2005)
System Controls Reliability and Assessment Effort
the IS auditor is typically to support financial statement auditors by providing assessments of the reliability of system controls (Sayana, 2002). Specifically, financial statement auditors utilize the specialized skills of IS auditors to understand, document, and assess the reliability of controls that address IT-related risks. In this supportive setting, the purpose and goal of the IS audit is to obtain a ‘sufficient understanding’ of how an entity’s use of IT may affect controls relevant to management’s assertions (SAS 94). System controls need to be understood as they relate to management’s assertions, and, regardless of their strength, their operational effectiveness needs to be evaluated (COSO, 1999; COBIT, 2002; ISACA, 2002).1 Experimental studies that focus on an IS audit environment show that auditors consider system controls when planning audit procedures (Murthy, 1995; Viator & Curtis, 1998). Murthy (1995) illustrates that a system’s stability (i.e., how often a system is modified) has an impact on the IS auditor’s control assessment and audit procedures. Viator and Curtis (1998) illustrate that an IS auditor’s prior work experience influences the control assessment and audit procedures selected. While we are unaware of any archival study focusing on the relationship between control reliability and both effort and fees in an IS audit setting, studies focusing on financial statement audits (not IS audits) obtain mixed results. A number of archival studies find little relationship between internal control assessments and financial statement audit decisions (Bedard, 1989; Mock & Wright, 1993; O’Keefe et al., 1994; Stein et al., 1994; Hackenbrack & Knechel, 1997; Felix et al., 2001). Other archival studies find a relationship (Willingham & Wright, 1985; Kreutzfeldt & Wallace, 1986; Messier & Plumlee, 1987; Roberts & Wedemeyer, 1988; Kreutzfeldt & Wallace, 1990; Bedard & Johnstone, 2004). Adding to these conflicting results, experimental method studies also report a relationship between internal control assessments and financial statement audit decisions (Gaumnitz et al., 1982; Kaplan, 1985; Libby et al., 1985; Emby, 1993). Conflicting results at the financial statement level do not provide us with a clearly documented expectation about whether control assessments influence the audit process. Our study investigates whether system control assessments influence the IS audit process. IS auditors evaluate information reliability by assessing whether appropriate system controls are in place and operating effectively. To © Blackwell Publishing Ltd 2005
81
complete such assessments, IS auditors first gain an understanding of how the IS processes information. To gain an understanding of the system, IS auditors rely heavily upon clientprepared documentation that outlines the internal control structure, including IS processes and IT controls that mitigate system risks. After obtaining existing documentation of the system and its controls, IS auditors can begin to assess whether documented controls are actually in place and operating effectively. By means of re-performance, inquiry, observation, and review, IS auditors verify the accuracy of flowcharts, narratives, and other documents that provide evidence on the placement and effectiveness of controls. Controls that are well-documented, appropriately placed, and operating effectively contribute to information reliability. If the client provides accurate and updated control documentation, the IS auditor can more easily verify the existence of controls asserted to be in place. For clients without adequate or updated documentation, IS auditors must conduct additional management and employee interviews to understand and accurately document the IS processes and controls in place. Thus, the effort expended on controls documentation is primarily verification for clients that provide updated controls documentation, but typically involves preparation and verification for clients who do not provide adequate records. It is possible that erroneous controls documentation can lead to audit inefficiency as the IS auditor attempts to verify documented controls. If documented controls are not in place, the IS auditor must revise documentation and reevaluate the control environment. Similarly, when documented controls are in place but are not effective, the IS auditor must review the internal controls in light of the deficiency. These controls, as well as their documentation, are part of the firm’s IT investment. Greater investment in the design and documentation of system controls should garner returns by reducing IS audit effort, and therefore IS audit fees.
System controls reliability assessments The COSO framework provides IS auditors with specific guidance for assessing and evaluating ITrelated activities (COSO, 1999; POB Report, 2000; COBIT, 2002; ISA, 2004). This guidance promotes IS design, storage and security procedures to Int. J. Audit. 9: 79–90 (2005)
82
prevent and detect errors and frauds and to promote control consciousness in operations. System controls are classified as either general or application controls. General controls are designed to manage and monitor the systems environment, thereby affecting all system-related activities. They include control aspects such as segregation of duties, assignment of authority, management integrity and corporate governance, software acquisition and maintenance, physical and online security, access to data, and contingency planning – anything that involves entity-wide system concerns (Guldentops, 2001; COBIT, 2002; Greene, 2002). The primary objective of application controls is to ensure the accuracy and integrity of specific applications such as sales order processing or accounts payable. They include input, processing, and output controls within an application. Input controls relate to data that is entered into the system; processing controls relate to data transformation; and output controls work to ensure that processed data is distributed and utilized appropriately by authorized users for authorized purposes (Hall, 2000). Taken together, general and application controls serve to ensure validity, accuracy, and completeness of financial information produced by the IS. Section 404 of SOX requires management of SEC registrants to provide (1) an annual statement of their responsibility for establishing and maintaining an adequate internal control structure and procedures for financial reporting and (2) an assessment of the effectiveness of the internal control structure and procedures for financial reporting. Section 404(b) requires that the firm issuing the company’s financial statement audit report also attest to management’s internal control assessment in accordance with standards established by the Public Company Accounting Oversight Board (PCAOB, 2004), which was created by SOX. These standards explicitly require management and auditors to base their assessment of the effectiveness of internal control over financial reporting on the COSO framework. Companies are spending substantial amounts on IT to ensure their internal control is sufficient to comply with Section 404. Similarities between a Section 404 audit and an IS audit have led audit professionals to recognize the need for IS audit expertise in the Section 404 audit process. Similarly the IAASB recognizes the © Blackwell Publishing Ltd 2005
R. J. Daigle, T. Kizirian and L. D. Sneathen, Jr.
need for systems specialists on audits when the client’s financial information system utilizes IT heavily. IS auditors are therefore valuable members of audit engagement teams for the purpose of assessing the overall IS internal control structure.
3. HYPOTHESES AND METHODOLOGY Hypotheses We argue that well-documented system controls that are in place and operating effectively may potentially lead to decreased IS assessment effort for the following reasons: • If system controls are well-documented, then the IS auditor may be able to substitute the client’s accurate and updated control documentation that the IS auditor would otherwise need to create. • If documented controls are in place, then the client documentation is accurate. Controls erroneously alleged by the client to be in place (e.g., invalid documentation) will necessitate greater inquiry, observation, and documentation effort by the IS auditor. • If documented controls are not operating effectively, then additional investigation is required to determine the magnitude and pervasiveness of the controls deficiency, in addition to greater documentation effort. Accordingly, we expect that the strength of system controls will have an inverse effect on the total audit effort involved in documenting and assessing system controls (i.e., stronger controls can be relied upon to reduce IS audit effort). Both Davis et al. (1993) and O’Keefe et al. (1994) document a strong positive relation between financial statement audit hours and fees. Similarly, we expect that the reduced IS audit effort resulting from stronger system controls is associated with reduced IS audit fees. Because effort and fees are highly correlated, the effect of controls strength assessments on each should be similar. We therefore do not distinguish our hypotheses on the basis of these audit outcomes; rather, we test both for the sake of robustness. The assessment of the two categories of system controls, general controls and application controls, is mutually exclusive, and therefore we present separate hypotheses for each of these assessments, leading to the following hypotheses: H1a: The strength of general controls is inversely related to IS audit effort. Int. J. Audit. 9: 79–90 (2005)
System Controls Reliability and Assessment Effort
H1b: The strength of general controls is inversely related to IS audit fees. H2a: The strength of application controls is inversely related to IS audit effort. H2b: The strength of application controls is inversely related to IS audit fees.
Proprietary data A Big 4 firm provides the data used in this study. The firm granted access to its archived audit working paper records for one practice office. Using a random number generator, we selected sample IS audits from the list of archived engagements containing audit files from 1996 to 1999. The firm provided IS audit data for 60 engagements from 60 different clients deemed to be system-dependent. Each engagement in the sample has a calendar year-end. The consistency in year-ends eliminates potential variation caused by staffing resource constraints faced by the auditing firm. Accounting firm policy requires an IS audit be conducted to support the financial statement auditors in obtaining system controls understanding and strength assessments for firms that process a significant amount of transactions electronically. The firm uses in-house IS auditors to obtain system controls understanding and assessments. Ultimately, these system-related assessments assist the financial statement auditors in arriving at a control risk assessment. The system dependency of these clients provides a unique opportunity to investigate the potential effect of assessed IT controls strength on both IS audit effort and fees. While these IS audits are conducted to support the IS audit, they are billed separately. The audit firm has been auditing these clients for an average of seven years. Within the sample, 54 clients are publicly traded. The data set does not contain any first-year audits.2 The firm assisted in the coding of variables used in this study, and provided a subsequent multi-level review to facilitate consistent coding of the data.
Variable measurement and model specification To test H1a and H1b, we estimate Equations 1 and 2 using ordinary least squares: ISHRS i = b0 i + b1GEN i + b 2 TA i + b 3 TENUREi (1) + b 4 PUBi + b 5IND i + b6 YR i + b7 ROA i + e i © Blackwell Publishing Ltd 2005
83
ISFEES i = b0 i + b1GEN i + b 2 TA i + b 3 TENUREi (2) + b 4 PUBi + b 5IND i + b6 YR i + b7 ROA i + e i The dependent variable in Equation 1 is the natural log of IS audit hours (ISHRS), and the dependent variable in Equation 2 is the natural log of the dollar amount of IS audit fees charged to the client (ISFEES). Theoretically the two dependent variables are measures of auditor effort. However, the inclusion of IS audit fees portrays a real economic impact of strong controls. Consistent with prior literature the natural log is used to address the nonlinearity of audit effort and fee distributions (recent example, Bedard & Johnstone, 2004). The test variable is the IS auditor-assessed strength of general controls (GEN) which is assessed and documented as ‘strong,’ ‘moderate,’ or ‘weak.’ This assessment is coded 3 for strong, 2 for moderate, and 1 for weak. The assessment of GEN is conducted in a manner consistent with generally accepted standards (e.g., COSO, 1999). Consistent for both H1a and H1b, we expect to obtain negative coefficients on GEN. We include several control variables to mitigate the potential for a correlated omitted variable problem. Since no prior archival study has focused on factors that may influence IS audit decisions, we include factors that have been shown to impact financial statement audit decisions. Prior literature has noted that the length of the auditor-client relationship may affect risk assessments and audit effort due to learning over time (Ashton, 1991; O’Keefe et al., 1994). We control for this by including the number of years the auditor has been auditing the client (TENURE). As a result of the audit process, the client should have attained an understanding of control deficiencies to be mitigated. Over time, these refinements should result in a decrease in IS audit hours resulting in a negative coefficient for TENURE. To control for client size, we include the book value of client total assets (TA) for the year under audit. Prior literature has shown that the relationship between auditor effort and client size is nonlinear (O’Keefe et al., 1994). To address this issue, we utilize the natural log of total assets. While larger firms may have more resources leading to potentially stronger system controls, they may have more complex control structures and greater decentralization, potentially increasing system risks. It is unclear how these effects will aggregate and affect the relationship between Int. J. Audit. 9: 79–90 (2005)
84
effort and assets. Accordingly, we do not have an expectation of the sign on TA. Prior research suggests the auditor is more likely to be sued if the client is publicly held (e.g., St Pierre & Anderson, 1984). Additionally, incentives to override controls to overstate financial standing and results of operations are suggested to be greater for managers of public firms due to market driven compensation structures (O’Keefe et al., 1994). In order to compensate for the related increase in auditor business risk, public client’s system controls are likely to bear greater scrutiny. We control for this by including an indicator variable (PUB) (public = 1, private = 0), which we expect to exhibit a positive association with ISHRS and ISFEES. Consistent with prior financial statement audit fee research, we include return on assets (ROA) as a control for risk (e.g., Craswell & Francis, 1999). We do not have an expectation of the sign on ROA. To control for potential systematic differences in the manner in which IS audits were conducted between years and between industry groups as identified by the data-granting firm, we include IND and YR. IND is an indicator variable representing the two industry subcategories in our sample (biotech = 1, high-tech = 0), and YR is a vector of dummy variables with n - 1 elements, where n equals the number of years from which observations are drawn. Because we do not have any evidence concerning major changes in IS audit approach by the data-granting firm between industry groups, or over time, we hold no expectation for the coefficient on IND or YR. To test H2a and H2b we estimate Equations 3 and 4 using ordinary least squares: ISHRSi = b0 i + b1APPi + b 2 TA i + b 3 TENUREi (3) +b 4 PUBi + b 5IND i + b6 YR i + b7 ROA i + e i ISFEESi = b0 i + b1APPi + b 2 TA i + b 3 TENUREi (4) +b 4 PUBi + b 5IND i + b6 YR i + b7 ROA i + e i The test variable in Equations 3 and 4 is the IS auditor-assessed strength of application controls (APP) which is assessed and documented as ‘strong,’ ‘moderate,’ or ‘weak.’ This assessment is coded 3 for strong, 2 for moderate, and 1 for weak. Similar to GEN, the assessment of APP is conducted in a manner consistent with generally accepted standards (e.g., COSO, 1999). Consistent for both H2a and H2b, we expect to obtain negative coefficients on APP. © Blackwell Publishing Ltd 2005
R. J. Daigle, T. Kizirian and L. D. Sneathen, Jr.
GEN and APP address different risks and are assessed by the IS auditors separately. However, the comprehensive prevention and detection of risks to the system is based on the operation of both GEN and APP. General controls without application controls will compromise a system, and vice versa, suggesting an incremental contribution of each reliability assessment over the other. In order to accommodate the possibility that GEN and APP together will affect ISHRS and ISFEES, we estimate Equations 5 and 6 using ordinary least squares: ISHRS i = b0 i + b1GEN i + b 2 APPi + b 3 TA i + b 4 TENUREi + b 5PUBi + b6 IND i + b7 YR i + b8 ROA i + e i
(5)
ISFEES i = b0 i + b1GEN i + b 2 APPi + b 3 TA i + b 4 TENUREi + b 5PUBi + b6 IND i + b7 YR i + b8 ROA i + e i
(6)
4. RESULTS Table 1 presents descriptive statistics for the variables (none logged) used in the regression analysis (Panel A) and Spearman correlations (Panel B) for the dependent and test variables used to test H1 and H2. The high degree of correlation between IS audit hours and IS audit fees (0.98, p < 0.0001) is consistent with both Davis et al. (1993) and O’Keefe et al. (1994). Table 2 presents OLS regression results for tests of H1a and H1b. GEN obtains a significant, negative coefficient in both Equation 1 (-0.9633, p < 0.0001) and Equation 2 (-0.9189, p < 0.0001). These results indicate that as the assessed strength of general controls increases, general controls are relied upon to reduce both the number of IS audit hours required and IS audit fees decline. IS audit effort and IS audit fee regressions are provided for robustness. The conclusions drawn from these regressions are similar. Consistent with prior literature, both TA and TENURE are significant with the predicted signs in Equations 1 and 2, respectively. Further, the high R2 (47.26% and 45.92% for Equations 1 and 2) suggest that effectiveness of system design, as captured by the general controls assessment, significantly explains subsequent IS audit effort and fees. Results for TA suggest that the larger the firm, the greater the IS audit effort and fees. Results for TENURE indicate that the longer the auditor-client relationship, the Int. J. Audit. 9: 79–90 (2005)
System Controls Reliability and Assessment Effort
85
Table 1: Descriptive statistics and Spearman correlations (N = 60) Panel A: Descriptive statistics
TOTISHRS TOTISFEES GEN APP TA TENURE PUB IND ROA
STD DEV
MEAN
QUARTILE #1
MEDIAN
QUARTILE #3
231.61 $51,044 0.91 0.78 1.80 4.30 0.30 0.49 0.40
130 $30,175 2.20 2.17 18.24 6.85 0.9 0.63 -0.06
23 $5,000 1 2 17.10 4 1 0 -0.29
37.5 $8,000 3 2 18.27 6 1 1 -0.02
164 $40,000 3 3 19.27 8 1 1 0.11
Panel B: Spearman correlations TOTISHRS
TOTISFEES
GEN
APP
1.00
0.98
