Information Systems Research

Vol. 16, No. 3, September 2005, pp. 307326

issn1047-7047 eissn1526-5536 05 1603 0307
informs

doi 10.1287/isre.1050.0063
2005 INFORMS
On Data Reliability Assessment in
Accounting Information Systems
Ramayya Krishnan
The Heinz School, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213-3890, rk2x@cmu.edu
James Peters
The R. H. Smith School of Business, University of Maryland, College Park, Maryland 20742-7215,
jmpeters@umd.edu
Rema Padman, David Kaplan
The Heinz School, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213-3890
{rpadman@cmu.edu, djk@andrew.cmu.edu}
T
he need to ensure reliability of data in information systems has long been recognized. However, recent
accounting scandals and the subsequent requirements enacted in the Sarbanes-Oxley Act have made data
reliability assessment of critical importance to organizations, particularly for accounting data. Using the account-
ing functions of management information systems as a context, this paper develops an interdisciplinary
approach to data reliability assessment. Our work builds on the literature in accounting and auditing, where
reliability assessment has been a topic of study for a number of years. While formal probabilistic approaches
have been developed in this literature, they are rarely used in practice. The research reported in this paper
attempts to strike a balance between the informal, heuristic-based approaches used by auditors and formal,
probabilistic reliability assessment methods. We develop a formal, process-oriented ontology of an accounting
information system that denes its components and semantic constraints. We use the ontology to specify data
reliability assessment requirements and develop mathematical-model-based decision support methods to imple-
ment these requirements. We provide preliminary empirical evidence that the use of our approach improves
the efciency and effectiveness of reliability assessments. Finally, given the recent trend toward specifying
information systems using executable business process models (e.g., business process execution language), we
discuss opportunities for integrating our process-oriented data reliability assessment approachdeveloped in
the accounting contextin other IS application contexts.
Key words: workow and process management; accounting information systems; mathematical modeling;
internal control
History: Salvatore March, Senior Editor; Kalle Lyytinen, Associate Editor. This paper was received on June 4,
2003, and was with the authors 14.75 months for 3 revisions.
1. Introduction
The reliability of data produced by organizational
information systems used to plan, diagnose, and con-
trol business operations has long been considered
important. Despite extensive study of this problem in
the context of accounting information systems, few
rigorous yet practical tools have emerged in the liter-
ature (Felix and Niles 1988, Waller 1993). The present
guidance consists mainly of frameworks that do not
provide rigorous, systematic ways to assess data reli-
ability. These frameworks provide checklists of issues
that affect data reliability but do not provide formal
denitions of the key concepts in the frameworks
nor decision rules or algorithms to insure that the
reliability assessment is both efcient and effec-
tive
1
(cf. Committee of Sponsoring Organizations of
the Treadway Commission (COSO) 1994, Informa-
tion Systems Audit and Control Foundation (COBIT)
2000). The lack of formal concept denitions and deci-
sion rules makes it difcult to develop practical data
reliability assessment systems.
However, the relevance of data reliability assess-
mentparticularly in the context of accounting
1
We provide specic denitions for efcient and effective
in 2.2.2.
307
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
308 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
information systemshas increased considerably
with the recent passage by the U.S. Congress of
the Sarbanes-Oxley (SOX) Act (H.R. 3763Sarbanes-
Oxley, Title IV, 404 2002; Securities and Exchange
Commission 2003). The act requires a rms CEO and
CFO to certify the reliability of the data reported
in the nancial statements as well as the reliability
and documentation of the information system that
produced those data. These mandates may well be
overdue because reports of signicant reliability prob-
lems in current accounting information systems have
begun to appear in practitioner outlets. For example,
CFO magazine states Experts estimate that anywhere
from 10 percent to 30 percent of the data ow-
ing through corporate systems is bad. . . (Goff 2003,
pp. 9798). In addition, recent surveys indicate that
improving the reliability of a rms AIS to meet SOX
requirements will require signicant effort. For exam-
ple, Business Week (2003) reports that a recent survey
found SOX will . . . prompt 85% of Americas largest
100 companies to overhaul many components of their
nancial-reporting systems.
In this paper, we develop a formal approach to data
reliability assessment motivated by a eld study of a
major international accounting rm. The eld study
focused on data reliability assessment of account-
ing functions of management information systems.
Accounting functions are all information-capturing
and processing activities that lead to the mainte-
nance of general ledger account balances in a manage-
ment information system (MIS), regardless of whether
those functions are embedded in an integrated MIS
or in a freestanding accounting information system
(AIS).
2
General ledger account balances are the core
nancial data used by organizations to make manage-
rial decisions and to report the nancial status of an
organization to external stakeholders (Hollander et al.
2000). The eld study identied important decision
support requirements and a data reliability assess-
ment task (key control selection, discussed in detail
in 2) considered to be important to auditorsthe
professionals tasked with conducting data reliabil-
ity assessments of organizational information sys-
tems. The approach we have developed consists of
2
For simplicity, we refer to these accounting functions as an AIS
for the balance of this paper.
(a) an ontological metamodel that permits the repre-
sentation of key concepts needed to assess data reli-
ability, and (b) a set of algorithms that can process
instances of the ontological model to support deci-
sion making by auditors. We provide preliminary evi-
dence that the use of this approach improves the
effectiveness of auditors engaged in data reliability
assessment.
The rest of the paper is organized as follows. Sec-
tion 2 denes the key-control-selection problem and
describes how it ts into the process of evaluating AIS
data reliability using an illustrative example of a por-
tion of an AIS. Section 2 also reviews relevant prior
work on reliability assessment, including work in the
accounting and auditing literature, the software lit-
erature, the data quality literature, and the literature
on CASE tools and emerging work on declarative,
yet executable, business process models. Section 3
presents our process-oriented ontological metamodel
of basic concepts and relationships that describe an
AIS as a directed, attributed, acyclic graph. Using
this ontology, 4 formulates key control selection as
a set-covering problem and presents two procedures
required to compute the parameters of the model
from instances of the ontological model. Section 5
presents preliminary results comparing our AIS data
reliability assessment approach with that of experi-
enced auditors. Finally, 6 discusses how the general
approach we develop for AISs is more broadly appli-
cable to other IS applications.
2. The Key-Control-Selection Problem
2.1. Description of the Field Study
We conducted a eld study in a large international
accounting rm to understand their data reliability
assessment practices. The eld study consisted of
extensive interviews with seven audit managers in
three different ofces of the audit rm. The rms
director of audit research selected the managers to
interview based on their extensive knowledge and
experience with AIS reliability assessment. All inter-
views with the audit managers were recorded and
transcribed for further analysis. The purpose of the
interviews, and the analysis of their content, was to
identify the process auditors used to evaluate data
reliability in an AIS. The analysis identied an impor-
tant taskkey control selectionwithin that process
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 309
that the auditors identied as a step that required
computer-based decision support. We reviewed the
results of our analysis of the transcripts with the
rms director of audit research to conrm our nd-
ings. In addition, we conducted a review of the
professional literature to conrm the importance of
the key-control-selection task, the recommended pro-
cedures for key control selection, and the deni-
tion of the concepts required to select key controls.
A description of the key-control-selection task and
how it relates to AIS data reliability assessment is pre-
sented in the next section.
2.2. Introduction to AIS Reliability Assessment
and Key Control Selection
3
2.2.1. AIS Reliability Assessment. An AIS is a
transaction-oriented information system, and errors
arise in the context of these transactions. Auditors
who assess AIS reliability rely on four important con-
cepts: (a) assertions, (b) error classes, (c) information
transformation processes (ITPs), and (d) control pro-
cedures (AICPA 1999). Each of these are discussed
in turn.
Assertions and error classes are closely related. An
assertion is a statement about the absence of a partic-
ular class of error in the general ledger account data.
Auditors begin by determining the assertions they
would like the data in the general ledger accounts
to support. Five classes of errors, namely complete-
ness, existence, valuation, rights and obligations, and
presentation and disclosure (AICPA 1990, 1999), are
considered. Completeness errors occur when a valid
transaction that should be in the system is missing
(i.e., it is either not recorded or has been deleted
incorrectly). Existence errors occur when an invalid
transaction is added to the system. Valuation errors
occur when the data in the system does not accurately
reect the economic results of the transaction that
created the data. These three error classes are mutu-
ally exclusive, and exhaustive in terms of errors that
affect the accuracy of the data in the AIS. Auditors
also consider rights and obligations, and presentation
and disclosure errors, but these errors are relevant to
3
This section, as well as 3, 4, and 5, contains a variety of technical
terms. We provide a glossary of these terms in an appendix to this
paper.
the production of external nancial statements from
the AIS database and not the accuracy of the AIS
database, per se. Therefore, we do not consider them
in our research. Professional standards do not spec-
ify a tolerable error from these sources, but leave this
determination to the auditor (AICPA 1990). The set
of error classes determined by an auditor to be rele-
vant to a reliability assessment study are referred to
as target error classes.
The three error classes we consider capture two
key elements of data reliability that have been regu-
larly used in IS data quality research: completeness
and accuracy (Ballou and Pazer 1985; Redman 2001,
ch. 14; Wang et al. 1995). Based on our eldwork and
review of the literature, we nd that accuracy is a
combination of existence and valuation. That is,
the concept of accuracy includes both the exclusion
of invalid transactions in the AIS as well as the accu-
rate valuation of valid transactions. Because different
activities may lead to completeness, existence, or val-
uation errors, the auditors practice of decomposing
accuracy into existence and valuation errors, as well
as considering completeness, contributes to a more
complete assessment of AIS reliability.
Information transformation points (ITP) are points
in the AIS where these different classes of errors can
be introduced. ITPs are arranged in connected struc-
tures, such that information ows from one ITP to
another. This structure begins at the boundary of the
organization when the AIS rst captures data about
a transaction. This data then ows through a series
of ITPs until it reaches the general ledger account,
which, for our purposes, is the terminal point of the
AIS. These intervening ITPs alter the data in a vari-
ety of ways and, therefore, are capable of introduc-
ing errors.
Controls are procedures designed to prevent or
detect one or more of these errors. The internal control
structure of an AIS species the set of control pro-
cedures included in the AIS; their capacity for error
prevention/detection; and the information ows from
which each control can prevent/detect errors. The
reliability of an AIS is evaluated with respect to the
absence of error classes in the general ledger accounts.
Given an AIS and its internal control structure, the
auditor assesses the risk that one or more of the target
error classes will be present in one or more general
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
310 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
ledger accounts (GLAs). In other words, the audi-
tor has to determine if the error elimination
4
capa-
bilities of the controls are adequate to prevent errors
from reaching the general ledger accounts (GLAs).
Key controls are the subset of controls in the AIS that
provide the auditor with the desired assurance about
the absence of these error classes in the GLA.
This framing of the data reliability assessment prob-
lem focuses on the evaluation of an AIS and its exist-
ing internal control structure with respect to a target
set of assertions/error classes. Our eld study con-
rmed that this was the framing of the data reliability
assessment problem employed by the auditors. The
set of controls in the system is taken as given and
treated as exogenous to the data reliability assessment
problem. We discuss in 6 opportunities for further
research into the related problem of how to design
the set of controls in an AIS to meet specied data
reliability objectives.
2.2.2. Key Control Selection. The auditor needs
to develop a plan to test a set of controls in the
AIS to ensure that the general ledger accounts are
free of the types of error in the target assertions. An
AIS normally contains redundant or overlapping con-
trols, and so the auditor need not test all the con-
trols to determine if the AIS is reliable (i.e., his/her
target assertions are being met). The subset of the
set of controls selected for testing are referred to as
key controls. Testing controls is costly. For example,
auditors use techniques such as reprocessing a sam-
ple set of transactions to verify that the population
of transactions has been executed accurately. There-
fore, the auditor would prefer to select the smallest set
of key controls to test that will provide the required
assurance that they are functioning as designed. An
effective set of key controls ensures that the selected
controls have the capacity to eliminate target error
classes (TECs) in the GLAs. An efcient set of key con-
trols includes the fewest (or cheapest to test) set of
controls, while still being effective. Auditors nd the
development of effective and efcient key control sets
to be difcult. In 5, we present preliminary evidence
4
While auditors distinguish between preventive and detective/
corrective controls, our approach does not, and (for simplicity) we
will use eliminate in the balance of the paper to refer to a con-
trols ability keep errors classes from reaching the general ledger.
to demonstrate how even experienced (e.g., an aver-
age of 57 months of audit experience) auditors do
poorly when asked to construct effective and efcient
testing plans. They choose either too many controls or
too few controls, leading to problems with data relia-
bility assessments. Thus, the fundamental importance
of selecting key controls for data reliability assessment
led to our focus on key control selection in this paper.
2.3. Illustrative Example of an AIS
Figure 1 documents the main portion of an orga-
nizations purchasing processes and depicts infor-
mation transformation processes (ITP), information
ows, control procedures, general ledger accounts
(GLA), and target error classes (TEC). It is based on
a real case developed by the rm that participated
in our eld study. Developing such documentation is
the rst step in the data reliability assessment process
(Ballou and Pazer 1985). There are a variety of nota-
tions available for documenting the components of an
AIS. We have used notation loosely based on business
process modeling notation (BPMN) (see BPMI.org)
that permits the representation of the key concepts
underlying our approach in a direct manner. We leave
the question of developing robust extensions of the
BPMN notation suited to the needs of data reliabil-
ity assessment to future work. The formal semantics
associated with this notation is presented in 3.
The notation describes the AIS in terms of eco-
nomic events, information transformation processes,
controls, accounts, and error classes. We discuss the
example starting at the left of Figure 1. At the left
are economic events that create the information ow
that triggers the different ITPs shown in the g-
ure (i.e., when merchandise is requested by a unit
within the rm, when merchandise is received from
a vendor, or when an invoice is received from a ven-
dor). The three documentspurchase order, receiving
report, and invoiceare merged to produce a pay-
ment voucher that is used to support a cash pay-
ment to the vendor, resulting in entries to the cash,
accounts payable, inventory, and expense general
ledger accounts. The rounded rectangles of Figure 1
are ITPs. Examples are processes labeled Purchase
order and Check register. These are points at
which errors can be introduced. For example, com-
pleteness, existence, and valuation errors can be intro-
duced by the Purchase order ITP. The rectangles are
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 311
Figure 1 Diagram of Purchasing Activities Example
f 1
Request
merchandise
Purchase order
(PO) C, E,V Check register
C, E,V
Cash
C, E
Review coding,
math, &
authority
E,V
Account for PO
numbers
C
Match to VP
C,E,V
Match to
requisition
E,V
f 4
f 5
f 6
f 3
f 8
f 10
f 11
f 12
f 13
Cancel original
documents
C, E
f 9
Approve PO
E,V
Accounts
payable
C,V
Compare batch
totals
C, E,V
f 2 f 7
Disbursement
C, E,V
Receive
merchandise
Voucher package
(VP) C, E,V
Receiving report
(RR) C, E,V
Account for RR
numbers
C
Inventory
C, E,V
f 14
f 15
Match PO, RR
and I
C, E,V
Voucher register
N
Expense
C,V
Purchase invoice
(I) C,E,V
Receive
invoice
Error codes:
Key:
fn
Control
Error classes
covered
GLA
TECs
C Completeness
ITP
Error classes
at risk
Assurance-
control's
span
Economic
event
Information
flow
E Existence
V Valuation
N none
controls and are associated with ITPs, as indicated by
the assurance arrows. For example, the Account for
PO numbers control eliminates completeness errors
on ow ] 4.
Most controls function by comparing information
owing out of an ITP with information owing into
that ITP. Preventative controls function by screening
information owing into an ITP and, thus, prevent
the information owing out of the ITP from contain-
ing errors. Irrespective of the type of the control, the
objective of the control is to eliminate errors from the
information owing from an ITP.
The set of errors the control is able to eliminate
is referred to as its coverage capability. For exam-
ple, the Compare batch totals control in Figure 1
can prevent completeness, existence, and valuation
errors. The information ows on which the control
can eliminate errors are referred to as the span of a
control. For example, Compare batch totals com-
pares information from the Disbursement ITP with
information from the Check register and Voucher
register ITPs to assure that completeness, valuation,
and existence errors are absent in all the ows out of
Disbursement, Check register, and Voucher reg-
ister transformation points.
As noted above, reliability assessment begins with
the auditor using his judgment to establish a set of
target error classes (TECs) or assertions in the general
ledger account (GLA). The circular nodes in Figure 1
represent the GLAs, and the TECs for each GLA are
shown in italics. The set of GLAs and their associated
TECs describe the goal of the reliability assessment
process. In the example, the auditor has set complete-
ness, existence, and valuation as target error classes
for the cash account, while only testing completeness
and valuation for the accounts payable account. Audi-
tors use their judgment to select target error classes.
For example, in Figure 1 the auditor has elected not
to consider existence violations for accounts payable
and expenses due to the low probability that the sys-
tem would record expenses or liabilities if they did
not exist.
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
312 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
Given the documentation of the AIS and its internal
control structure as depicted in Figure 1, the auditor
has to construct effective and efcient control-testing
plans. This raises four important research questions:
Question 1: What should be the formal repre-
sentation of the AIS for purposes of data reliabil-
ity assessment? We develop a formal approach to
support AIS data reliability assessment that permits
algorithmic processing but is intuitive enough that
auditors could apply it in practice. The foundation
of the approach is a semantically precise notation
required to model an AIS. While many diagramming
notations have been developed by rms for inter-
nal use by trained auditors, they do not have for-
mal semantics and therefore do not lend themselves
to computer-based decision support. We develop an
ontological metamodel of an AIS suited to the needs
of data reliability assessment.
Given the characteristics of the problem described
above, there are supporting research questions that
need to be answered to build a formal approach:
Question 2: Given the internal control structure,
on which information ows can a given control pre-
vent or detect errors? That is, what is a controls span?
The span of the control depends on the structure of
information ows in the AIS and the set of ITPs from
which the control has access to information. Comput-
ing the span of the control given the documentation
of the AIS is difcult for auditors. We present our
approach to this problem in 4.3.
Question 3: Given the TECs associated with
each GLA, what TECs should be eliminated from
each ow that contributes, either directly or indirectly,
to the GLA? This is important because controls are
designed to provide assurances for ows coming from
each ITP, and different ITPs may generate different
sets of error classes. If the correct set of errors is elim-
inated at each ow, this will ensure that TECs will
not be present in the GLAs. We detail our approach
to this problem in 4.2.
Question 4: Given the TECs for each ITP and
the span of the controls, how should an effective and
efcient set of key controls be selected? Even expe-
rienced auditors do poorly at constructing effective
and efcient key control sets as we discovered during
our eldwork and our preliminary evaluation study
reported in 5. We discuss our approach to this prob-
lem in 4.1.
2.4. Prior Work and Related Literature
2.4.1. Auditing Literature. The most directly
related literature is the work on auditing in the
accounting literature. While the auditing literature
has not studied the key-control-selection problem,
it contains work on probabilistic and deterministic
models for AIS reliability assessment and decision
support. Probabilistic modeling research has focused
on nding ways to combine the necessary probability
estimates to provide an overall, quantitative assess-
ment of AIS reliability (e.g., Ahituv et al. 1985; Ballou
and Pazer 1985; Bodnar 1975; Cooley and Cooley
1982; Hamlen 1980; Haskins and Nanni 1987; Knechel
1983, 1985; Lea et al. 1992). Probabilistic research in
AIS evaluation has had little, if any, inuence on prac-
tice because the models tend to make too many sim-
plifying assumptions to achieve tractability (Felix and
Niles 1988).
Research on deterministic models for AIS reliability
assessment has focused more on decision support by
modeling the evaluators cognitive process and build-
ing expert systems (e.g., Kelly 1985, Meservy et al.
1986). However, because the approach is not based on
formal methods, it cannot guarantee the completeness
or accuracy of the resulting evaluation. One determin-
istic approach that evolved into an applied system
was the TICOM project (Bailey et al. 1985). Price-
waterhouseCoopers built on the modeling approach
developed in the TICOM project to implement a deci-
sion support system, called COMET, to help auditors
evaluate AIS systems reliability (Nado et al. 1996).
However, COMET does not help auditors link weak-
nesses in AIS reliability directly to TECs and, there-
fore, provides little support to assist AIS designers in
reengineering the internal control structure of the AIS
to eliminate future errors.
Our work strikes a balance between the informal
heuristic methods used by practitioners and extant
formal decision support approaches. We extend the
state of the art by developing a formal graph-based
notation suited to the needs of reliability assessment.
The notation has syntactic similarities to the informal
notation used by auditors (cf. Grant Thornton 1996),
has formal semantics, and can be processed algorith-
mically (using a mathematical modeling framework)
to provide decision support in the creation of both
effective and efcient testing plans.
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 313
2.4.2. IS Data Quality Literature. Complement-
ing the work in the accounting literature is the work
on data quality (Kaplan et al. 1998, Pierce 2004, Pipino
et al. 2002, Wang et al. 1995). This literature draws
on the analogy between physical manufacturing pro-
cesses and information manufacturing and argues
that data quality should be concerned about accuracy,
timeliness, consistency, and accessibility (Wang et al.
1995). In the AIS context, we decomposed accuracy
into existence and valuation to match how practi-
tioners view data reliability. The data quality research
offers a starting point towards dening error classes
for other IS applications that are analogous to the
error classes used to assess data reliability in AIS.
Distinctions are made in this literature between
quality problems that arise because of aws in IS
design versus quality problems that arise due to oper-
ational aws. Recent work by Pierce (2004) relates
this perspective on data as an information product
to ideas from auditing by dening a structure called
a control matrix. However, in contrast to our work,
Pierce (2004) neither models the information ows
in the information system and the location of con-
trols, nor provides a method for reasoning with the
control matrices to arrive at overall measures of data
quality. In addition, many papers in this literature
assume that the systems represent worlds with high
rates of change (e.g., stock market data) and offer pre-
scriptions (Orr 1998) with an emphasis on timeliness
and accessibility. While the emphasis is different in
our application context, the core ideas underlying our
work rely on an abstractionconsisting of controls
and error classesof an information system that has
applicability in contexts other than AIS (Pierce 2004).
2.4.3. Business Process Modeling and CASE
Tools. Because reliability assessment works with a
model of information ows and information trans-
formation processes, process-modeling methodolo-
gies and related systems analysis and design tools
are relevant to our work. In general, the literature
on these topics focuses on supporting analysis and
design. As noted in 2.2, data reliability assessment
of AIS presently focuses on the evaluation of the
internal control structure of an AIS rather than on
its design. However, modeling representations devel-
oped in process modeling and CASE literatures are
relevant to our work because documenting the AIS
and its internal control structure is essentially an exer-
cise in representation. We briey discuss the unied
modeling language (UML) and business process mod-
eling notation (BPMN) and the reasons why we have
not used these approaches in our work.
UML (see www.uml.org) is an industry standard
for software design. It provides a number of diagrams
(e.g., state transition diagram, activity diagram) for
visualizing and documenting the artifacts that make
up a software system. Tools are available to help
designers create these diagrams. While the advantage
of using UML as a starting point in our work is the
ability to have an impact on reliability assessment and
reliability design in domains other than AIS, there are
signicant differences in both the conceptualization
and the semantics underlying our approach and that
of UML. Specically, UML does not support a pro-
cesscentric view of the world that is central to our
conceptualization. Further, UML diagramming nota-
tions lack a formal semantics (Mcumber and Cheng
2001). Even the recently released UML 2.0 speci-
cations from the Object Management Group (OMG)
offer only a natural language semantics for UML dia-
gramming notation, and developing a formal seman-
tics is still an active area of research.
In contrast to UML, the recent initiatives in the
area of business process modeling and management
share the processcentric view underlying our work.
Examples include the work on creating declarative,
yet executable, models of business processes such
as business process execution language (BPEL) and
the business process modeling language/notation
(BPML/BPMN) (see BPMI.org 2003). The advantage
of using a broadly adopted and popular notation is
the opportunity to integrate our reliability assessment
approach in tools created to support business pro-
cess modeling and execution. However, extensions to
BPMN notation and semantics are required to adapt
it for use in data reliability assessment.
Because BPMN does not provide specic object
types to represent fundamental data reliability con-
cepts such as ITPs, controls, and their attributes, we
need specialized notational extensions to BPMN that
Distinguish Between ITPs and ControlsBoth
could be represented as BPMN atomic tasks, however,
BPMN cannot capture semantic differences between
these objects (e.g., the fact that ITPs generate errors
and controls eliminate them).
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
314 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
Distinguish Between Message Flows from ITPs to
Controls and from Controls to ITPsAs with ITPs and
controls, the different types of information ows in an
AIS have different semantics. However, information
ows are represented as message ows in BPMN, and
message ows do not have the expressive power to
distinguish between the different types of information
ows in an AIS (e.g., information ows from controls
to ITPs provide assurance, while ow between ITPs
may contain errors). In addition, the BPMN syntax
only allows messages to link to tasks and not to other
information ows, as our assurance ows do.
Specify Error Classes for ITPs and ControlsThe
only structure in BPMN that allows tagging of ITPs
and controls with error classes is text annotation,
which is not executable and lacks formal semantics.
While these notational extensions are relatively easy
to make, there is a fundamental difference between
the semantics underlying BPMN and the seman-
tics required for data reliability assessment. BPMN
semantics are based on a variant of the pi calculus
(Milner 1992) and are designed to represent and rea-
son about distributed processes. This is in contrast to
the semantics required to reason about the structure
of information ows and the error prevention capabil-
ities of internal controls. Because of the basic nature
of these changes, we defer further discussion of nota-
tional issues to future research and discuss this topic
briey in 6.
3. An Ontological Metamodel of
an AIS
The key-control-selection problem assumes a cer-
tain conceptualization of the AIS (e.g., in terms of
accounts, ows, controls, error classes, and their rela-
tionships). Both auditors and systems designers use
graph-based structural models of AIS (such as in
Figure 1) that are an informal realization of this
conceptualization to support both design and testing.
As discussed in 2.3, question Q1 is concerned with
the formalization of the graphical model required to
support the computer-based specication of AIS by
auditors as part of a data reliability study. Impor-
tantly, such formalization should be open to analysis
by inferential procedures. For example, these proce-
dures are required to derive the span of a control,
a parameter of the model used to determine effective
and efcient key control sets. Formalized models of
the structure of a system are referred to as ontologi-
cal metamodels (Gruber 1991, Noy and Hafner 2000,
Wand and Weber 1990) in the literature.
An AIS can be conceptualized at two levels of gran-
ularity at least: an action level and a process level.
Action-level ontologies (e.g., Hamscher 1992) and the
TICOM model (Bailey et al. 1985) are ne grained
and describe an AIS in terms of actions, like waiting
for a document, and objects on which they operate,
such as repositories of les where documents are kept.
Other ontologies of AIS have focused on the artifacts
developed within the system, but not on the processes
that produce them (Wand and Weber 1990). Action-
level ontologies are well suited to simulate processing
and document ow through an AIS. However, they
are too ne grained for the purposes of data relia-
bility assessment. In contrast, a process-level ontol-
ogy is coarse grained and species an AIS in terms
of ITPs and information ows. ITPs are at a higher
level of abstraction than actions and can be thought
of as collections of actions that are used to capture
or transform information. This level of abstraction is
well suited to the needs of data reliability assessment
because the objective is to not to simulate actions, but
to identify key controls.
We therefore develop a process-level ontological
metamodel using data gathered from eld interviews
with practicing auditors and a detailed review of the
existing literature (e.g., Arens and Loebbecke 1997,
Grant Thornton 1996). The model uses set theoretic
notation and provides a language to state the key
components of an AIS and their interrelationships in
a precise manner. We provide a brief overview of
the ontology using set-theoretic notation developed
in the model management literature (Bhargava and
Kimbrough 1993).
3.1. Denition of AIS Components
The metamodel of an AIS consists of the following
major components and their interrelationships.
Economic Event (EE)Event that generates the ini-
tial need for the AIS to capture information.
General Ledger Accounts (GLA)Maintain total of
nancial activity (i.e., the output of the AIS from
which the nancial statements are produced).
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 315
Information Transformation Process (ITP)Processes
that transform information from one form to another.
ITPs can introduce errors. These processes also are
where control procedures designed to eliminate errors
introduced by processes are located.
Control Procedures (CONTROL)Procedures de-
signed to eliminate specic error classes. A control is
associated with one or more ITPs in that it has access
to information present in the ows emanating from
them. Controls have the capacity to eliminate error
classes that are introduced by ITPs.
Information Flows (FLOW)Abstractions of infor-
mation that ow through the AIS, i.e., the output of
ITPs.
Target Error Classes (TEC)Error classes the evalua-
tor wants to determine are not present in an account.
ECs are associated with accounts and ITPs as well as
with controls. In the case of accounts, their association
implies the need to check an account for the presence
of the given error class. In the case of ITPs, the asso-
ciation is used to identify the classes of errors the ITP
can generate. In the case of a control, the association
is used to assert the capability of the control to elimi-
nate the EC.
3.2. Formal Specication of Relationships
Between AIS Components
An AIS is conceptualized as a directed, acyclic, at-
tributed graph (see Figure 2). The nodes of the graph
are GLAs, EEs, ITPs, and controls. The arcs are the
FLOWs. TECs are attributes of GLAs. These objects
are modeled as sets, and each set is required to be
nonempty. In our description of the ontological meta-
model, we use special fonts as follows: sets, functions,
relations, and VARIABLES. Additionally, we refer to the
power set of a term using square brackets (e.g., [EC] is
the power set of EC). Each node and arc in the notation
is required to be an object in this conceptualization.
Each AIS component also has both an ID and a
description attribute. The ID is a unique identier.
Figure 2 The Six Components of an AIS
AIS_COMPONENT
CONTROL
NODE
GLA ITP EE FLOW
Subset of
Attribute of
TEC
Figure 3 AIS Components
GLA AIS_COMPONENT
ITP AIS_COMPONENT
EE AIS_COMPONENT
TEC AIS_COMPONENT
CONTROL AIS_COMPONENT
FLOW AIS_COMPONENT
ID: AIS_COMPONENT Names
description: AIS_COMPONENT Text
I AIS_COMPONENT,I (An AIS component cannot be an empty set)
I,J AIS_COMPONENT,I J = (Components are mutually exclusive)
The description allows an auditor to make any special
notes about each component. Each node and arc has
an identier and a description (see Figure 3).
Next, we specify the FLOWs in the AIS (see Fig-
ure 4). Each FLOW has an origin and a destina-
tion where information ows from the origin to the
destination. Origin and destination are modeled as
functions. Their domain and range for each class of
ow is declared in Figure 4. Because the output of
an ITP is a ow and because ITPs contain processes
that can create errors, we can only draw conclu-
sions about the degree to which FLOWs whose ori-
gins are ITPs are free of errors. We refer to these
FLOWs as Checkable_Flows. We distinguish between
two classes of checkable ows, Middle_Flows and
End_Flows. End_Flows are FLOWs whose destination
is a GLA, the output of the AIS. Middle_Flows are
FLOWs between ITPs. In contrast, Beginning_Flows
are those whose origin is an economic event and rep-
resent information-capturing activity at the boundary
of the organization where the auditor does not have
the ability to assess data reliability. That is, the audi-
tor cannot verify the accuracy of the information con-
Figure 4 Flows and Their Properties
Beginning_Flow Flow
Checkable_Flow Flow
Middle_Flow Checkable_Flow
End_Flow Checkable_Flow
I, J Flow, I J =
origin: Beginning_Flow EE
destination: Beginning_Flow ITP
origin: Middle_Flow ITP
destination: Middle_Flow ITP
origin: End_Flow ITP
destination: End_Flow GLA
I ITP,( K ITP and I K and J Middle_Flow) or
( J Beginning_Flow and K EE), origin(J) = K
destination(J) = I
I GLA, K ITP and J End_Flow, origin(J) = K
destination(J) = I
I Beginning_Flow J Middle_Flow destination(I)= origin(J)
I Middle_Flow J End_Flow destination(I)= origin(J)
FLOW
Beginning_Flow Checkable_Flow
Middle_Flow End_Flow
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
316 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
Figure 5 Directionality
node-components = GLA EE ITP
upstream: node-components node-components
X,Y node-components, upstream(X,Y) is interpreted as X is
upstream of Y
I Flow, upstream(origin(I), destination(I))
X,Y,Z node-components, upstream(X,Y) upstream(Y,Z) upstream(X,Z)
tained in the economic event, only that the AIS has
captured that information accurately. In Figure 1, ] 1
is a Beginning_Flow, ] 5 is a Middle_Flow, and ] 11 is
an End_Flow.
Because the AIS is a connected set of FLOWs,
the destination of a Beginning_Flow is the origin of
a Middle_Flow. In turn, the destination of a Mid-
dle_Flow is the origin of an End_Flow. This require-
ment leads to the specication of an AIS as a
connected graph whose nodes are EEs, ITPs, and
GLAs and whose arcs are FLOWs (refer to Figure 4).
Because ows are directed from EEs to GLAs,
we introduce a transitive relation upstream that is
dened on the set of ITPs, EEs, and GLAs (i.e., the
nodes in the graph structure). Upstream captures the
directionality of ows. Node A is upstream of Node B
if information ows from A to B, possibly through
other nodes (see Figure 5).
By construction, cycles are not possible in an AIS
because the origin of a node is required to be
upstream of the destination node. We now turn to
error classes. Recall that the auditor associates a set of
error classes, referred to as target error classes, with a
GLA. We model this association as a set-valued func-
tion, TEC (target error class)see Figure 6. In Fig-
ure 1, the TEC of the cash account is {C, |] indicating
that completeness and existence are the target error
classes for the cash account. We model the TEC map-
ping as a function to facilitate inference.
ITPs create errors in information ows, and controls
are designed to eliminate those errors. The essence
of data reliability assessment is to ensure that the
controls are operating as designed. The error classes
that a control is designed to eliminate are modeled
as a relation called covers. In Figure 1, the control
Figure 6 Target Error Classes
TEC: GLA [EC]
I GLA, E [EC] such that TEC(I) = E
Compare batch totals can eliminate completeness,
existence, and valuation errors. A control has access to
information from the ows that emanate from at least
one ITP in an AIS. We model this as a relation located-
at that relates a control to the ITPs from which it can
obtain information (see Figure 7). This relation is not
explicitly represented in Figure 1. The control Com-
pare batch totals is located at the Check register
and Voucher register ITPs. In addition, a control
may have access to other ITPs that are upstream of
the ITPs at which it is located. For example, the con-
trol Compare batch totals has access to information
from the Disbursement ITP as well, as indicated by
its span (dened below). Given the set of ITPs from
which the control has access to information, an eval-
uator needs to determine the set of owsreferred
to as the span of a controlon which the control can
eliminate errors. This requires an analysis of the struc-
ture of information ows in the AIS and is a dif-
cult and error-prone step for human auditors (cf. Q2
in 2.3). However, span is an important parameter of
the key-control-selection model, and 4.3 presents an
algorithm that uses the ontological model discussed
in this section to compute the span of each control in
the AIS.
The set of objects and the relations presented dene
the structure and the semantic constraints on the
AIS. The expressiveness of these constructs was val-
idated using the set of real-world cases made avail-
able by the rm that participated in the eld study
(see 5). This specication can be implemented using
a logic programming language such as Prolog or
implemented using a database with the constraints
stated using an imperative language such as C (the
option we chose in our implementation). Irrespective
of the option chosen, the ontological metamodel pro-
vides both a language and formal semantics for rep-
resenting and reasoning about the information ows
and structure of an AIS. We are not aware of other
Figure 7 Controls and Their Properties
covers: Control EC
C C E EC, covers(C,E) (A control covers at least one error class)
located-at: Control ITP
C C, T ITP, located-at(C,T) (A control is located at an ITP)
spans-to: Control ITP
C C, T
1
,T
2
ITP, spans-to (C,T
1
) AND located-at(C,T
2
) upstream(T
1
,T
2
)
(A control can only span to an upstream ITP)
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 317
ontological models that have been developed for the
purpose of AIS reliability assessment and, therefore,
there may be other ways to structure an ontologi-
cal metamodel to support AIS reliability assessment.
Thus, as the rst metamodel for this task presented
in the literature, we focus on demonstrating that it
will achieve the goals we have set for it. We now
present the mathematical model and associated algo-
rithms used for reliability assessment and discuss its
relationship to the ontological model.
4. Mathematical Model Development
The Key-Control-Selection Problem: Given an instance
of the metamodel of an AIS (as dened in the pre-
vious section) and a set of target error classes, deter-
mine the smallest set of controls that will need to be
tested to assess the absence or presence of the target
error classes in the accounts of the AIS.
As noted in the previous section, a control is
designed to eliminate a specic set of error classes in
the set of ows in its span. Referring to the graph-
ical model of the AIS (Figure 1), if the accounts in
the AIS are to be free of the target error classes, this
requires that each ow that directly or indirectly con-
tributes to the information in the accounts should also
be free of these errors. Auditors can establish if this
is the case by testing a set of controls whose coverage
capability on the ows in their span is a superset of
the target error classes on these ows. We formulate
this as a set-covering problem (Cormen et al. 1992)
under the following assumptions. These assumptions
were derived from our eld interviews with experi-
enced auditors and from their rms technical guid-
ance (Grant Thornton 1996) and were not adopted
simply to make our model more tractable. Auditing
research also nds that auditors use simple determin-
istic models to assess AIS reliability (Felix and Niles
1988, Waller 1993), rather than probabilistic models.
Assumption 1. Controls are designed to operate deter-
ministically. For each error class in its coverage set, a con-
trol is designed to eliminate the error with probability 1.
Assumption 2. The capability of a control to elimi-
nate an error is independent of the capability of any other
control.
Assumption 3. The cost of testing a control is xed
and is the same for every control.
Assumption 4. All ITPs generate all possible error
classes.
Although the assumptions on which the model is
based may seem to be simplistic and restrictive, the
most restrictive assumptions offset each other because
they have the opposite impact on reliability. Assump-
tion 4 (i.e., that all ITPs generate all error classes) off-
sets Assumption 1 (i.e., that controls eliminate errors
with probability 1). Further, the preliminary evalu-
ation we conducted (described in 5) supports the
validity of our approach. The mathematical formu-
lation of the key-control-selection problem is shown
below in Figure 8.
The objective function of the model minimizes the
number of key controls selected for testing and fol-
lows directly from Assumption 3. However, it is quite
straightforward to take differences in control testing
costs into account by modifying the objective function
into a weighted, additive cost function. The constraint
set insures that at least one control that can cover each
target error class on each ow will be chosen. The
constraint set follows from Assumptions 1, 2, and 4.
The inputs required to use the model for key con-
trol selection are the set of information ows ( ), the
error classes on these ows that need to be detected
or prevented (|(] )), the set of available controls (C),
and their span (D
ce.f
). These inputs are derived (as
noted in Figure 8) from an instance of the ontological
model developed by the auditor as a rst step in the
data reliability assessment process. In other words,
no additional effort is invested by the auditor to use
Figure 8 The Set-Covering Model Formulation
Min K
c
(Select the Minimum Number of Controls)
cC
s.t.
K
c
= 0 or 1
c
C
Where:
E(f) is the set of error classes that need to be tested in a flow f (this is algorithmically
derived from an instance of the ontological model)
F is a target set of Flows (this is specified exogenously by the auditor)
C is the set of Controls (this is declared in the instance of the ontological model)
K
c
0/1 Variable with the following interpretation:
= 1, when the cth Control is selected as a Key Control;
= 0, when the cth Control is not selected as a Key Control.
D
ce.f
0/1 matrix of constants such that: (this is derived algorithmically from an
instance of the ontological metamodel)
= 1, where the cth Control covers the eth TEC for the fth flow;
= 0, where the cth Control does not cover the eth TEC for the f th flow

D
ce.f
* K
c
1
f
F and
c
E(f) (Cover the set of TECs E(f) for each flow f)
cC

Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems

318 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
the model. The output is the set of key controls. Two
metrics used to evaluate the quality of a set of key
controls are effectiveness and efciency, respectively.
A key control set is effective if each of the required
information ow/error class combinations is covered
by at least one key control. The smallest effective
key control set is efcient. Any feasible solution to the
model in Figure 8 is effective and the optimal solution
is efcient. This follows directly from an inspection
of the structure of the set-covering model (Cormen
et al. 1992). If the set of available controls is insuf-
cient to meet the target reliability goals set by the
auditor, the constraints will not be satised, thereby
resulting in the model not having a feasible solution.
In this case, an auditor could use judgment to mod-
ify the set of available controls and use the model to
design a feasible internal control structure. We remark
on this issue and the problem of optimally design-
ing a control structure to meet desired data reliability
objectives in 6.
Because the set-covering model is NP-hard
(Cormen et al. 1992), large instances of the model
are intractable. We have developed heuristic meth-
ods to both select key control setswhat we refer
to as key control recommendationas well to evalu-
ate auditor-proposed key control setswhat we refer
to as key control evaluation. Because of space con-
straints and because the procedures are variants of
well-understood heuristics for the set-covering prob-
lem (Feo and Resende 1989, Cormen et al. 1992,
Fisher 1990), we do not discuss these procedures in
detail. Before we proceed to the next section, we
reiterate that the model and the associated proce-
dures can be used in two modes. In the recommenda-
tion mode, the model would recommend to auditors
the set of key controls to test. The decision support
offered in this context is primarily sensitivity anal-
ysis where the auditor may make changes to the
AIS model and rerun the recommendation algorithm.
Alternatively, the model could be used to evaluate
the effectiveness of, and to suggest improvements to,
auditor-generated key control sets. Computationally,
evaluation is fast because it only checks to see if
the constraints of the mathematical model are sat-
ised. If the constraints are not satised given a
set of controls (as discussed in the previous para-
graph), the auditor can be guided in the selection of
controls to add to regain feasibility. Recent work by
Chinneck (2001) discusses tools for automating infea-
sibility analysis. Through these features, the system
supports the decision-making capabilities of auditors.
We next discuss methods used to compute the param-
eters of the model.
4.1. Computing Mathematical Model Parameters
from the Ontological Model
Of the four inputs required to use the model, |(] )
and D
ce..f
are not directly specied in the ontologi-
cal model of the AIS. For example, in Figure 1, tar-
get error classes are associated with GLAs. However,
|(] ) is the set of target error classes that need to be
eliminated in every information ow that can have an
effect on the account.
4.2. Computing Target Error Class
Propagation|(] )
Because errors are introduced into the GLAs by
ITPs, the information owing out of every ITP ows
directly or indirectly into a GLA and needs to be
tested for TECs. For example, the information ow
] 7 created by process Voucher package in Figure
1 can affect all four GLAs. Therefore, to ensure data
reliability, information owing from it must not con-
tain TECs associated with all these GLAs (i.e., |(] 7) =
{C, |, V]). Because each ITP may produce informa-
tion ows that affect the balances of numerous GLAs
in a real AIS, the determination of the error classes
that need to be tested for in each ow can be tedious
and error prone. Exploiting the graphical structure
of the AIS, we formulate a procedure (referred to as
propagate) that conducts a depth-rst search of the
graphical representation specied in the instance of
the ontological model. Given a table consisting of the
set of GLAs and their target error classes and the
set of checkable ows, propagate returns a table of
error classes that need to be tested for each check-
able ow in the AIS. The sample input and output of
the propagate function used to compute |(] ) for the
ows in Figure 1 are shown below (Figure 9).
4.3. Computing the Span of a ControlD
ce.f
The span of a control denes the set of ows for
which a control can prevent or detect its error classes.
By denition, a control needs to have access to all
the information that ows into an ITP for it to be
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 319
Figure 9 Sample Output
SAMPLE INPUT
set-of-flows = {{f1, f2,..., f15}} Flow Error classes
TEC(Cash) = {C, E, V} f15 C, V
TEC(Accounts payable) = {C, V)} f14 C, E, V
TEC(Inventory) = {C, E, V} f13, f12 C, V
TEC(Expense) = {C, V} f11 C, E
f1 f10 C, E, V
SAMPLE OUTPUT
able to eliminate errors in the ows emanating from
the ITP. Consider the case of the Compare batch
totals control. It has the capability to eliminate com-
pleteness, existence, and valuation errorshowever,
on which ows? It is located at the Check register
and Voucher register ITPs, and the documentation
of the AIS asserts that it spans to the Disburse-
ment ITP. Here the term spans is interpreted to mean
that the control has access to information in ow ] 7
that ows into the Disbursement ITP, which in turn
implies that the control is able to eliminate errors in
ows ] 9 and ] 10, as shown using the dotted arrows
linked to this control in Figure 1. Because the control
has access to all the information (ow ] 9) owing into
the Check register ITP, it is able to eliminate errors
in ows ] 11 and ] 12 that emanate from the Check
register ITP. However, it does not have access to
all the information owing into the Voucher regis-
ter ITP. It has access to ow ] 10, but does not have
access to ow ] 8, which is one of the outputs of the
Voucher package ITP to which this control does not
span. Therefore, the Compare batch totals control
cannot provide assurance about the absence of the
error classes on ows ] 13, ] 14, and ] 15 that emanate
from the Voucher register ITP. Through this sort of
reasoning about the structure of ows in the AIS, an
auditor can determine that the span of the Compare
batch totals is the set of ows {] 9, ] 10, ] 11, and ] 12}.
In this section, we exploit the fact that the AIS is rep-
resented as a directed, acyclic graph to formulate an
algorithm to compute the span of a control, thereby
enabling the parameter D
cc.]
to be computed from the
instance of the ontological metamodel.
The informal reasoning illustrated in the example
demonstrated the need to reason with ows in the
AIS. We generalize this intuition by rst considering
paths in the AIS. A path is a sequence of informa-
tion ows leading from an economic event (EE) or
ITP through one or more ITPs to a GLA or ITP. In the
above example, consider the candidate set of infor-
mation ows for which the Compare batch totals
control can provide assurance. Because it is located
at the ITPs Check register and the Voucher regis-
ter, the ows {] 11, ] 12, ] 13, ] 14, ] 15] that ow out
of these ITPs are candidates for inclusion in the span
of the control. Because the control spans to Disburse-
ment (i.e., has access to information owing into this
ITP), ows {] 9, ] 10] are added to the set of candidate
ows. Now consider paths from the ITP Voucher
package to a GLA such as Inventory. Why choose
Voucher package? We do so because it is the high-
est ITP upstream of ITP Disbursement to which
the control Compare batch totals spans, which has
a path to the GLAs that share ows with the paths
to the GLAs from Disbursement. There are two
paths from Voucher package to Inventory. The
rst, (] 7, ] 10, ] 14), is a strict superset of the path
(] 10, ] 14) from Disbursements to Inventory. The
second path, (] 8, ] 14), shares a ow ] 14 with the
path (] 10, ] 14) from Disbursement to Inventory.
However, the ow ] 8 is a ow to which the con-
trol does not have access. As errors in the ow ] 8
may propagate through to ow ] 14, and because the
control Compare batch totals cannot eliminate these
errors, the ow ] 14 that is shared between these paths
cannot be in the span of the control. For the same rea-
son, ows ] 13 and ] 15 cannot be included in the span
of this control. This leads to the span of this control
consisting of ows ] 9, ] 10, ] 11, ] 12.
Span computation requires reasoning with the
structure of the ows and paths in the AIS. The pro-
cedure presented below formalizes the intuition and
reasoning illustrated in the example.
Procedure: Span-computation.
Input: Declarations in the ontological metamodel
instance encoded as a directed, acyclic, attributed
graph, and C, the control whose span is to be
computed.
Output: The set of ows in the span of the control C.
1. Let G::=AIS-graph.
2. Let SG:=topsort(G). Topsort is the topological
sort (Aho et al. 1983) of a directed, acyclic graph. It
orders the nodes in linear sort order. In our setting,
the topological sort orders the nodes (the economic
events, information transformation points, and GLAs)
in a linear order. Using this ordering, the procedure
can determine node precedence relationships.
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
320 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
3. Let SpanITP be the ITP that control C partic-
ipates in a spans-to relation. In Figure 1, Disburse-
ment is the only ITP that Compare batch totals
participates in a spans-to relationship. If the control
does not participate in a spans-to relation, let Span-
ITP be the ITP at which the control is located. In our
example, SpanITP is Disbursements.
4. Let candITPset := precede(SpanITP, SG). Pre-
cede is a function that computes the set of ITPs that
precede SpanITP from the topologically sorted AIS
graph SG. In our example, candITPset is {Purchase
Order, Receiving Report, Purchase Invoice, Voucher
Package}.
5. Let Spancandows be candidate ows that
should be included in the span of a control. Spancand-
ows is the union of the set of ows whose origin
nodes are either the ITPs that the control C is located
at or the ITP with which it participates in a span-to
relationship. In our example, the Spancandows is
{] 9, ] 10, ] 11, ] 12, ] 13, ] 14, ] 15].
6. Let Deleteset be the set of ows in the Span-
candows set that should be removed to determine
the span. For each element E of the candITP set and
each GLA, let path(E, GLA) be the set of paths
sequence of owsfrom E to the GLA. In our exam-
ple, the path from Voucher package to Inventory
consists of two paths (] 7, ] 10, ] 14) and (] 8, ] 14).
Eliminate all paths that are strict supersets of paths
from SpanITP to the GLAs. In our example, SpanITP
is Disbursements. The path from Disbursements to
Inventory is (] 10, ] 14). Thus, all paths that contain
(] 10, ] 14) are eliminated. Of the paths that remain,
compute the set of ows that are both elements of the
paths from candITP set to the GLA and the SpanITP
to the GLA. This is a simple set intersection operation.
In our example, the ows are (] 13, ] 14, ] 15).
7. Return Spancandows Deleteset; In our
example, this returns the set {] 9, ] 10, ] 11, ] 12) as the
ows in the span of the control.
4.4. Application of the Model
To apply our model, the auditor would need to pro-
vide four classes of information:
(1) A list of all the components of the AIS to include
such as economic events, ITPs, controls, GLAs, and
how they are related (i.e., the ows between these
items). This is the AIS graph shown in Figure 1 and
would be encoded using the notation introduced in
the ontological metamodel.
(2) The error classes that each ITP could generate.
(3) The error classes each control could eliminate,
the ITP where the control was located, and a list of
the ITPs to which the control spans.
(4) A set of TECs for each GLA.
As noted, this information is represented in the
instance of the ontological metamodel. Two impor-
tant parameters, |(] ), and each controls span, D
cc.]
,
are computed using the procedures described in
4.2 and 4.3. Given these parameters, an instance
of the set-covering model is created and solved
using the heuristic methods briey discussed in 4.1.
Using the model, auditors can select an effective and
efcient set of key controls or can evaluate the effec-
tiveness and efciency of a key control set that is judg-
mentally derived.
5. Model Validation
5.1. Validation Strategy
The goal of our validation strategy was to demon-
strate that our approach produces solutions to the
key-control-selection problem that approximate the
solutions that would be produced by expert auditors
using the rms methods (i.e., the right answer).
A complete test of our premise would lead to a large
set of more specic hypotheses that should be tested
to validate the model and its underlying approach. As
a rst step in justifying expending substantial auditor
time to test the features of the model and its underly-
ing approach, we developed an exploratory validation
study that was designed to provide prima facie evi-
dence that the decision support tool can produce a set
of key controls comparable to sets produced by expert
auditors. This exploratory study used an existing data
set from an experiment where experienced auditors
employed by the same rm that participated in our
eld study selected a set of key controls in three cases
developed by expert auditors (Davis 1996). We found
that our approach selected a set of key controls in
those three cases that were comparable to the sets
selected by the expert auditors (experts in the rm
who developed the materials used in the study) and
was better than the sets selected by the experienced
auditors (auditors with an average of 57 months of
experience; please see 5.3) in the Davis study.
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 321
Because an AIS can contain several controls that,
in different combinations, could produce the same
potential reliability, possibly at the same cost, merely
comparing the specic controls in two key control sets
may not reect the true degree of similarity between
those sets. Therefore, our validation study compared
the similarity in effectiveness and efciency between
the models key control sets and the auditors sets
in Daviss study to two types of professional bench-
mark sets.
All three cases used in Daviss study were based
on training materials developed by expert auditors.
Therefore, our rst professional benchmark was the
solution to each case provided by the case developers
(the expert auditors). In addition, because Daviss par-
ticipants selected control-testing plans individually,
while in practice these plans would be determined
by an audit team, we also established a consensus
set of key controls (i.e., those controls selected by
at least half the auditors in each case) that should
more closely represent key control sets that might be
produced by audit teams.
5.2. Test Case Representativeness
Davis gave his subjects case materials that closely
approximate the information an auditor would need
to apply our model. This included
(1) a diagram showing the economic events, ITPs,
controls, and ows in the portion of the AIS repre-
sented by the case;
(2) each controls span and coverage; and
(3) a set of target error classes.
Davis then asked his subjects to select key controls
sets for each case.
The cases were relatively diverse, were based on
rms in different industries, and contained a vari-
ety of types of controls and ITPs. Table 1 summa-
rizes the number of ITPs, controls, GLAs, and ows
in the cases as well as the types of controls and indus-
tries. To demonstrate the variety of controls included
in the cases, Table 1 classies the controls along
three dimensions: the type of action performed by
the control, whether the control was computerized or
manual, and whether the control was preventive or
detective.
These cases are a holdout sample in that they
were not used to develop our model. In addition, the
Table 1 Summary of Test Case Control Characteristics
Cases
A B C
Retail Clothing Beverage
Type of rm lumber yard wholesaler manufacturer Total
Size of portion of AIS described in each case
Number of ITPs 7 7 7
Number of controls 7 11 17
Number of GLAs 2 2 2
Number of ows 7 7 8
Types of controls included in the cases
Classied by type of action performed
Access controls 1 1 2
Analytical procedure 1 2 3
Approval and 1 1 1 2
signatures
Completeness 1 3 4
checks
Edit controls 1 1
General review 1 1
Matching and 5 4 2 11
re-performance
Reconciliation 1 2 7 9
Total 7 11 17 35
Classied by computer-based or manual
Computerized 0 3 7 10
Manual 7 8 10 25
Total 7 11 17 35
Classied by preventative or detective
Preventative 2 4 8 14
Detective 5 7 9 21
Total 7 11 17 35
test cases were drawn from training materials and
professional guidance that was designed to present
auditors with realistic cases on which to train. There-
fore, we believe the cases used in Daviss study are
representative of real rms and the way auditors
approach internal control evaluation in practice.
5.3. Subjects
The auditor subjects in Daviss study
5
were from a
single international public accounting rm and par-
ticipated in Daviss experiment as part of a training
session on internal control documentation and eval-
uation. The experienced auditors had, on average,
5
Although Davis (1996) studied both experienced and inexperi-
enced auditors, only the data from Daviss experienced auditor
group were used in the validation analysis.
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
322 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
57 months of experience, performed 47 audits, super-
vised 22 audits, documented the internal control
structure in 24 audits, and made preliminary control
risk assessments on 32 audits. These auditors also had
18 hours of rm training with the internal control doc-
umentation and evaluation procedures used by the
rm and in Daviss study. There were 5, 9, and 15
subjects in Cases A, B, and C, respectively.
5.4. Analysis Method
The documentation of the AIS in each of these
cases was used to obtain the inputs required to
create the instance of the mathematical model pre-
sented in Figure 8. The key control sets recom-
mended by the model were compared for similarity
to the sets developed by the subjects (i.e., the expe-
rienced auditors) and the professional benchmarks.
The comparison was based on two attributes of the
sets: effectiveness, as measured by the percentage of
required information ow/error class combinations
covered by the key control set, and efciency, as mea-
sured by the percentage of controls tested. Required
ow/error combinations were calculated based on the
target-error-class propagation method described 4.2.
Because the case descriptions given to the auditors
included each controls extended span, the span prop-
agation algorithm was not needed to calculate D
cc.]
.
Percentages were used to allow the results from the
three test cases to be combined.
5.5. Validation Results
Figure 10 presents a graph of the efciency versus
the effectiveness metrics for the consensus and profes-
sional benchmarks described above, our model, and
the individual auditor subjects who solved each case.
Any key control sets that are effective by our de-
nition would lie on the right-hand axis of the dia-
gram (i.e., 100% of required error class/information
ow combinations covered). The goal of key control
selection, however, is to achieve 100% effectiveness by
testing the minimum number of controls (in this case,
represented as a percentage of total controls in the
case and plotted on the Y-axis). Therefore, the lower
the point representing the key control set is on the
Y-axis, the more efcient the set is; and the farther
to the right it is on the X-axis, the more effective the
set is.
Figure 10 Cost/Effectiveness Trade-off
Panel A - Case A
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 90.00 100.00
Percent of flows covered
0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 90.00 100.00
Percent of flows covered
0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 90.00 100.00
Percent of flows covered
P
e
r
c
e
n
t

o
f

c
o
n
t
r
o
l
s

t
e
s
t
e
d
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
P
e
r
c
e
n
t

o
f

c
o
n
t
r
o
l
s

t
e
s
t
e
d
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
P
e
r
c
e
n
t

o
f

c
o
n
t
r
o
l
s

t
e
s
t
e
d
(2)
(1)
Panel B - Case B
(2)
(1)
Panel C - Case C
(1)
Consensus
Benchmark
Model
Auditors
Consensus
Benchmark
Model
Auditors
Consensus
Benchmark
Model
Auditors
(3)
Note. Numbers in parentheses indicate multiple observations with the same
values; auditor observations that are the same as the model, benchmark; or
the origin of the graph.
In two of the three cases, both the model and the
professional benchmark produced effective key con-
trol sets (i.e., sets that cover 100% of the ow/TEC
combination needed to achieve the target level of
assurance the subjects were asked to achieve). How-
ever, the model was able to achieve 100% by testing
fewer controls. The professional benchmark did not
develop an effective key control set in one of the three
cases. Only two of the auditors in Daviss study devel-
oped an effective key control set, and there were sig-
nicant differences in the auditors key control sets.
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 323
These results imply that the model comes closer to
the correct answer than the experienced auditors
in Daviss study. The fact that the model seems to
achieve more efcient and effective results than even
the benchmark could be because of subtle differences
in the goals of the expert auditors who built the cases,
or because the model is more internally consistent in
applying the same decision rules as the expert audi-
tors. Research in decision support systems for audi-
tors indicates that a major advantage of these systems
is their internal consistency compared to auditors
(Messier 1995).
We believe these data provide sufcient support for
our approach to justify further testing. The models
key control sets are similar to the professional bench-
mark, indicating that its results are approximating
what the developers of the cases believed were the
optimal solutions to each case. However, the individ-
ual auditors showed signicant variability in selecting
key control sets and rarely achieved results similar to
the benchmark. Therefore, providing evaluators with
the models information on a case should help them
develop key control sets that more closely approxi-
mate those of expert auditors.
The signicant variation among auditors should be
of some concern given that they all came from the
same rm, had similar amounts of experience, and
were asked to apply the same approach in which they
had just received 18 hours of training. The variation
does not appear to be a function of the different cases.
The models, professional benchmarks, and consen-
sus auditors results were very similar across cases,
indicating that the expert auditors who developed the
cases believe different AIS have broad similarities in
how their controls are designed. Use of the proposed
model-based tool might therefore be used in train-
ing to assist evaluators in identifying that underlying
similarity and developing more consistent key con-
trol sets.
6. Discussion and Future Work
This work contributes to the literature on data reli-
ability assessment. While the work was motivated
by and set in the context of AISs, the key concepts
that underlie our approach have the potential to be
more broadly applicable. Essentially, the approach
denes a structured methodology for data reliability
assessment based on the following core concepts:
A denition of data reliability in terms of pres-
ence/absence of error classes.
A commonly understood and shared denition
of error classes.
Formalized conceptualization of the information
system in terms of information transformation pro-
cesses that can introduce errors and controls that can
prevent/detect error classes.
Decision support methods that exploit the for-
malized conceptualization to assist in data reliability
assessment, permitting an auditor to combine appro-
priately human judgment and algorithmic processing.
There are at least two specic opportunities for
transferring these ideas to domains and contexts other
than the problem studied in the paper. The rst
opportunity relates to the distinction we made in 2
between evaluation of a given internal control struc-
ture versus design of an internal control structure
to meet desired reliability objectives. Recall that the
dominant practice in reliability assessment assumes
that the AIS and its internal control structure are
xed. The emphasis is on evaluating the reliability
of the given system. However, given a set of con-
trols and their coverage capability, is there an optimal
internal control structure for a given set of informa-
tion transformation points and ows? In other words,
what is the optimal location of each of the controls
that make up the internal control structure? At which
ITP should they be located? This can be viewed as an
optimization problem over the set of admissible inter-
nal control structures for an AIS. Because the meth-
ods developed in this paper evaluate any AIS and
its associated internal control structure, they can be
used as part of a solution to the design problem. The
design problem is not limited to AIS and can arise
in the context of other IS applications. In fact, one
could make the argument that IS designers would
benet from considering the data reliability perspec-
tive during system design. The principal condition for
transferring the ideas developed in the AIS context to
other domains is the conformance with the conceptual
(ontological) model developed in the paper. Speci-
cally, error classes, transformation processes, controls
and their interrelationships are fundamental to the
approach. As Pierce (2004) demonstrates in applying
a very similar conceptualization to the data quality
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
324 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
evaluation of an e-mail archive, the conceptualiza-
tion we have developed is not limited to AIS. Once
the data reliability assessment problem in the new
domain is mapped into the graph-theoretic, ontolog-
ical metamodel developed in this paper, the decision
support methods used in this paper become directly
applicable. Of course, extensions or customizations
to the ontological model to meet application-specic
needs will require extensions to the decision support
methods developed in this paper.
The second opportunity is complementary to the
rst and related to the use of a BPMN-based notation
for specifying AIS. The advantage of using BPMN
notation is twofold. The rst is the availability of
visual tools (see www.intalio.com) for modeling with
BPMN. The second advantage is tied to its growing
popularity for IS specication, permitting the work
developed in this paper to have a broader impact.
However, as noted in 2, the semantics underly-
ing BPMN are based on pi-logic (Milner 1992) and
designed to reason about distributed processes. Data
reliability assessment requires reasoning about the
structure of the information ows, as discussed in 4.
How can one gain the benets of using a popular
notation like BPMN while still taking account of the
semantic differences? In 2, we identied extensions
and customizations of BPMN that would be needed to
model information reliability for an AIS. Implementa-
tion of an interpreter for the extended BPMN notation
with access to the methods that implement the seman-
tics described in 3 and 4 permits the interoperability
sought for integration. As with the rst opportunity,
the impact of this integration of our ideas with BPMN
is not limited to AIS. Any information system that can
be specied using the extended BPMN notation can
benet from both the evaluation methods developed
in this paper as well as any methods developed to
support design.
Beyond these opportunities, there are a number of
interesting ideas for future research. Some of these
stem from the limitations of this work. While the
assumptions underlying the model were developed in
discussions with experienced auditors and represent
the way they approached key control selection, they
are simplistic and do not represent the way processes
and controls really function. For example, controls do
differ in the costs needed to test them. Some controls,
like authorization controls, can be tested by merely
verifying a signature, while others, like matching doc-
uments, require reconstruction of the matching pro-
cess to test them properly. In addition, controls differ
in their strength. Further, a control that matches all
aspects of two documents would be much stronger
than a control that merely compares general ledger
balances to budget numbers. All of these observa-
tions demand model renements. However, we left
these renements to future research because our goal
was to bring an increased level of formalism to exist-
ing practice, and these attributes of controls were
not considered by the experienced auditors we inter-
viewed in our eldwork.
With regard to transferability of the decision sup-
port methods, the model that was developed was
based on how auditors approach AIS evaluation.
While recent work (Pierce 2004) adopts a very similar
conceptualization of the problem, a thorough study
of how IS designers in other application contexts per-
ceive data reliability is required. For example, the
auditors main concern is that the GLAs are free of
errors, not that the output from all the ITPs in the
AIS are free from errors. Our denition of a controls
span assumes that errors can occur in ows within
the AIS as long as they are eliminated before the reach
the GLAs. A rms management may rely on the out-
put of ITPs as well as the information in a GLA for
making decisions and, therefore, would require that
the output of all ITPs be free of errors. What are
the requirements for data reliability in other IS con-
texts such as medical information systems or credit-
processing systems? Are they similar or different to
the AIS context? We believe application contexts merit
careful study (as we have using AIS) and the litera-
ture on data quality (Pipino et al. 2002) offers a start-
ing point for this type of work.
In conclusion, we believe the decision support sys-
tems approach to data reliability assessment that com-
bines human judgment and the appropriate use of
model-based algorithmic procedures has important
advantages. First, because it does not attempt to
mathematically model the entire reliability assessment
process, it is able to address the problem at a level
of granularity that renders previous mathematical
model-based approaches computationally intractable
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
Information Systems Research 16(3), pp. 307326, 2005 INFORMS 325
(Felix and Niles 1988). Second, its use of model-
based tools that recommend key control sets with
known error detection properties provides assessors
with assurance about the reliability of their assess-
ments. Finally, the ability to use the tool either to
recommend key control sets or to evaluate assessor-
proposed sets permits the human assessor to rely on
the model to the extent desired. However, will the
availability of this technology result in better data reli-
ability assessment outcomes? Our preliminary evalua-
tion study indicates that our approach holds promise.
A detailed evaluation of how auditors adopt this
technology, how they incorporate it into their work
processes, and the data reliability assessments that
result is an interesting topic for future study.
Acknowledgments
The authors would like to thank Jefferson Davis for pro-
viding the validation data and for assisting in coding those
Appendix. Glossary
Term Denition
AIS Accounting Information System.
Beginning ows An information ow that begins with an economic event and is not checkable.
Checkable ows Information ows that originate in an ITP.
Completeness All the required information about economic events has been captured by the AIS and
all economic events have been captured. None of this information has been lost in processing.
Control A process within an AIS that eliminates errors from information owing out of an ITP.
Controls coverage All the information ows from which a control can eliminate errors.
Controls span The set of information ows from which a control can eliminate errors.
End ows A checkable ow that ends in a GLA.
Error class (EC) As used in this paper, these are completeness, existence, and valuation.
Existence The information about economic events that has been captured by the AIS reects events that actually occurred and
affected the rm. No information about nonexistent events has been added during processing.
General ledger account (GLA) Maintain total of nancial activity (i.e., the output of the AIS from which the nancial statements are produced).
Information transformation A process within an AIS that captures or processes information in some way.
process (ITP)
Key control selection The process by which an AIS evaluator selects a key control set.
Key control set A set of controls that, if functioning as designed, will provide the evaluator with their target level of AIS reliability.
Middle ow A checkable ow that does not end in a GLA.
Paths A sequence of information ows leading from either an economic event or an ITP to either another ITP or GLA.
Pathsbad A path for which a control does not have access to all the information needed to insure that information leaving
the path is free of the error classes the control covers.
Pathsgood A path for which a control has access to all the information needed to insure that information leaving the path is
free of the error classes the control covers.
Target error class (TEC) Error classes (EC) the evaluator wants to determine are not present in an account.
Target reliability An AIS-evaluator-determined level of reliability for each error class and account in an AIS.
Valuation Given that information about an economic event that affects the rm has been captured by the AIS, the value
associated with that event has been properly recorded and has not been changed during processing.
data. This work was supported in part by grants from the
NSF CISE/IIS/KDI 9873005 and by NSF IRI-9312143.
References
Ahituv, N., J. Halpern, H. Will. 1985. Audit planning: An algorith-
mic approach. Contemporary Accounting Res. 2(1) 95110.
Aho, A., D. Hopcroft, J. Ullman. 1983. Data Structures and Algo-
rithms. Addison Wesley, Boston, MA.
AICPA. 1990. Consideration of the Internal Control Structure in a Finan-
cial Statement Audit. AICPA, New York.
AICPA. 1999. CPA SysTrust: Assuring Reliability of Systems. AICPA,
New York.
Arens, A. A., J. K. Loebbecke. 1997. Auditing: An Integrated Approach.
Prentice-Hall, Englewood Cliffs, NJ.
Bailey, A. D., Jr., G. L. Duke, J. Gerlach, G. Ko, R. D. Meservy,
A. B. Whinston. 1985. TICOM and the analysis of internal con-
trols. Accounting Rev. 60(2) 186211.
Ballou, D. P., H. L. Pazer. 1985. Modeling data and process qual-
ity in multi-output information systems. Management Sci. 31(2)
150162.
Bhargava, H. K., S. O. Kimbrough. 1993. Model management: An
embedded languages approach. Decision Support Systems 10(3)
277300.
Krishnan et al.: On Data Reliability Assessment in Accounting Information Systems
326 Information Systems Research 16(3), pp. 307326, 2005 INFORMS
Bodnar, G. 1975. Reliability modeling of internal control systems.
Accounting Rev. 50(3) 747757.
BPMI.org. 2003. Business Process Modeling Notation Working Draft
1.0. (August 25). http://www.bpmi.org/specications.esp.
Business Week. 2003. For CFOs a crash course in tech. (Septem-
ber 30) 3539.
Chinneck, J. W. 2001. Analyzing mathematical programs using
MProbe. Ann. Oper. Res. 104 3348.
Cooley, J. W., B. J. Cooley. 1982. Internal accounting control systems:
A simulation program for assessing their reliabilities. Simula-
tion Games 13(2) 211231.
Committee of Sponsoring Organizations of the Treadway Com-
mission. 1994. Internal ControlIntegrated Framework. AICPA,
New York.
Cormen, T. H., C. E. Leiserson, R. L. Rivest. 1992. Introduction to
Algorithms. The MIT Press, Boston, MA.
Davis, J. T. 1996. Experience and auditors selection of relevant
information for preliminary control risk assessments. Auditing:
J. Practice Theory 15(1) 1637.
Felix, W. L., Jr., M. S. Niles. 1988. Research in internal control eval-
uation. Auditing: J. Practice Theory 7(1) 4360.
Feo, T. A., M. Resende. 1989. A probabilistic heuristic for a com-
putationally difcult set covering problem. Oper. Res. Lett. 8
6771.
Fisher, M. 1990. Optimal solution of set covering/partitioning prob-
lems using dual heuristics. Management Sci. 36(6) 674688.
Goff, J. 2003. Drowning in data. CFO Magazine 19(11) 97102.
Grant Thornton. 1996. Accounting and auditing manual. Grant
Thornton, Chicago, IL.
Gruber, T. R. 1991. The role of common ontology in achieving
sharable, reusable knowledge bases. J. A. Allen, R. Fikes,
E. Sandewall, eds. Principles Knowledge Representation Reasoning:
Proc. 2nd Internat. Conf. Morgan Kaufmann, San Mateo, CA.
Hamlen, S. S. 1980. A chance-constrained mixed integer program-
ming model for internal control design. Accounting Rev. 55(3)
578593.
Hamscher, W. 1992. Analysis of accounting systems via abstraction
and simulation results into causal networks. Technical Report
25, Price Waterhouse, Menlo Park, CA.
Haskins, M. E., A. J. Nanni, Jr. 1987. Toward attribute models
of accounting control systems: Qualitative versus quantitative
approaches. J. Accounting Literature 6 111130.
Hollander, A. S., E. L. Denna, J. O. Cherrington. 2000. Accounting,
Information Technology, and Business Solutions. Irwin McGraw-
Hill, New York.
Information Systems Audit and Control Foundation. 2000. COBIT:
Control Objectives for Information and Related Technology. Infor-
mation Systems Audit and Control Foundation, Rolling
Meadows, IL.
Kaplan, D., R. Krishnan, R. Padman, J. Peters. 1998. Assessing data
quality in AIS. Comm. ACM 41(2) 7278.
Kelly, K. P. 1985. Expert problem solving system for the audit
planning process. Ph.D. dissertation, University of Pittsburgh,
Pittsburgh, PA.
Knechel, W. R. 1983. The use of quantitative models in the review
and evaluation of internal control: A survey and review.
J. Accounting Literature 2 205219.
Knechel, W. R. 1985. An analysis of alternative error assumptions
in modeling the reliability of accounting systems. J. Accounting
Res. 23(1) 194212.
Lea, R. B., S. J. Adams, R. F. Boykin. 1992. Modeling of the audit
risk assessment process at the assertion level within an account
balance. Auditing: J. Practice Theory 11(Supplement) 152179.
Mcumber, W., B. Cheng. 2001. A general framework for formalizing
UML with formal languages. Proc. 23rd Internat. Conf. Software
Engrg., Toronto, Ontario, Canada, 433442.
Meservy, R. D., A. D. Bailey, Jr., P. E. Johnson. 1986. Auditing inter-
nal controls: A computational model of the review process.
Auditing: J. Practice Theory 5(2) 4474.
Messier, W. F., Jr. 1995. Research in and development of audit-
decision aids. R. H. Ashton, A. H. Ashton, eds. Judgment
and Decision-Making Research in Accounting and Auditing. Cam-
bridge University Press, Cambridge, UK, 207230.
Milner, R. 1992. Polyadic Pi-Calculus: A tutorial. Proc. Internat. Sum-
mer School Logic Algebra Specication. Springer-Verlag.
Nado, R., M. Chams, J. Delisio, W. Hamscher. 1996. COMET: An
application of model-based reasoning to accounting systems.
Proc. 8th Innovative Appl. Articial Intelligence Conf., AAAI Press,
Menlo Park, CA, 14821490.
Noy, N. F., C. D. Hafner. 2000. Ontological foundations for experi-
mental science knowledge bases. Appl. Articial Intelligence 14
565618.
Orr, K. 1998. Data quality and systems theory. Comm. ACM 41(2)
6671.
Pierce, E. M. 2004. Assessing data quality with control matrices.
Comm. ACM 47(2) 8286.
Pipino, L. L., Y. W. Lee, R. Y. Wang. 2002. Data quality assessment.
Comm. ACM 45(4) 211218.
Redman, T. 2001. Data Quality: The Field Guide. Digital Press.
Sarbanes-Oxley Act. 2002. H.R. 3763.
Securities and Exchange Commission. 2003. SEC implements inter-
nal control provisions of Sarbanes-Oxley; adopts investment
company R&D safe harbor. Press release 2003-66. http://www.
sec.gov/news/press/2003-66.htm.
Ushakov, Igor. 1994. Handbook for Reliability Engineering. John Wiley
and Sons.
Waller, W. S. 1993. Auditors assessments of inherent and control
risk in eld settings. Accounting Rev. 68(4) 783802.
Wand, Y., R. Weber. 1990. An ontological model of an information
system. IEEE Trans. Software Engrg. 16(11) 12821292.
Wang, R. Y. 1998. A product perspective on total data quality man-
agement. Comm. ACM 41(2) 5865.
Wang, R. Y., V. C. Storey, C. P. Firth. 1995. A framework for analysis
of data quality research. IEEE Trans. Knowledge Data Engrg. 7(4)
623640.