Professional Documents
Culture Documents
Computer-Assisted Audit Tools for IS Auditing A Comparative Study
Computer-Assisted Audit Tools for IS Auditing A Comparative Study
Computer-Assisted Audit Tools for IS Auditing A Comparative Study
net/publication/340424194
CITATIONS READS
4 4,324
4 authors:
All content following this page was uploaded by Iman M. A. Helal on 22 April 2020.
A comparative study
Sara Kamal1, Iman M. A. Helal1[0000-0001-8434-7551], Sherif A. Mazen1[0000-0001-5079-218X],
and Sherif Elhennawy2
1
Faculty of Computers and Artificial Intelligence, Cairo University, Giza, Egypt
sara.kamal@nub.edu.eg, {i.helal, s.mazen}@fci-cu.edu.eg
2
Information Systems Auditing Consultant
selhenawy@gmail.com
1. Introduction
CAATs are commonly employed to audit application controls in order to reduce the
total audit hours [5]. They enable auditors to test 100% of the population rather than a
sample, thereby increasing the reliability of the audit test and conclusions. IS auditor
can use CAATs to write a script for automated periodic audits. This automation helps
to achieve continuous auditing and monitoring according to management objectives [2].
According to [6], CAATs have several analytic capabilities such as data analysis,
applied and managed analytics, and continuous auditing. IS auditors can repeat audit
work by executing automatic audits, thereby reducing audit time and costs.
The goal of auditing is to ensure the effectiveness of internal controls, which are
designed to facilitate the company’s management activities. IS auditors should decide
the scope for their projects and develop their CAATs processes based on the audit
policies. CAATs performance can be measured using indicators or metrics of team
project performance, such as efficiency, completeness, compliance or accommodation
with work progress, outcome quality, interaction, and communication [7-9].
Previous studies in [10-14], have found that complete CAATs establishment can
conserve auditors’ manpower resources, reduce audit costs, reduce the time spent
executing audit tasks, increase audit quality, and enable enterprises to improve
operating efficiency. CAATs can assist in implementing the Sarbanes-Oxley Act
(SOX) requirements, as well as facilitating monitoring activities and reducing their
time. Hence, eventually, the use of CAATs can increase enterprise efficiency and
overall performance.
In [16], a study introduced some recommendations to increase the efficiency and
effectiveness of the available software tools to the auditor. These recommendations
include: (a) determine the enterprise’s audit mission, objectives and priorities, (b)
determine the types and scope of audits, (c) consider the enterprise’s technology
environment, (d) ensure using the suitable tools, (e) identify the risks, (f) train audit
team on the tool usage, and (g) support of periodic review reports.
The need for CAATs emerged to support IS auditor reviews many areas in an
organization. There are some researches investigating the influential factors to use
CAATs during the auditing process [18,20]. In addition, there are other factors that
affect the auditor’s choice in selecting the most suitable tool to accomplish the required
tasks. CAATs have some basic characteristics and factors that affect the auditor
decision while searching for a suitable tool [5]:
- Ease of use; a measurement of how easy the tool is to use by its intended users.
- Ease of data extraction; which includes the ability to access a wide variety of data
files from different platforms, and to integrate data with different formats.
- Ability to define fields and select from standard formats.
- Menu-driven functionality for processing analysis commands.
- Simplified query building and adjustments.
- Suitable platform and/or operating system for the organization.
- Supporting documentation and periodic audit reports.
There are other items that the IS auditor considers as key items while selecting a
comprehensive tool as in [16]. There are some recommendations for increasing the
software tools usage efficiency and effectiveness to the auditor. Those
recommendations include the following:
- Determine the enterprise’s audit mission, objectives, priorities.
- Determine the types and scope of audits.
- Consider the enterprise’s technology environment.
- Ensure selecting the right tools.
- Training audit team on the use of the selected tool.
- Be aware of the risks.
- Review daily reports.
The following subsections address factors from different perspectives. The
discussion starts with the support of existing standards and follows with how the factors
are affecting internal auditors’ intentions to use and accept CAATs. Due to the huge
number of CAATs that assist IS auditors, a sample of tools was collected from websites;
e.g. Capterra1 and Software Advice2. These tools cover many IS auditing areas that
need further investigation to decide the key factors affecting the tool selection criteria.
1
https://www.capterra.com/audit-software/
2
https://www.softwareadvice.com/audit/
Table 1. Distribution of CAATs supporting Auditing areas (Tools’ references see Appendix)
Code Areas ISO standard Tools
ARBUTUS – TeamMate – Tackle – SmartSolve –
Symbiant Tracker – R-CAP – Ramce ERP –
ISO
a.1 Risk assessment MKinsight –MetricStream – Isolocity – Qwerks –
31000:2018
InfoZoom – DATEV – Debian – Analyzer–Ecomply –
TrustArc – Consenteye –BigID – ZenGRC
Onspring– ECAT- Assure– ZenGRC - ManageEngine
ISO/IEC
a.2 Security ADAudit Plus – Debian– Lynis – Janco- Xandria-
27000:2018
Onspring-ACL- Delphix – ECAT
ISO/IEC WinAudit – Aircrack-ng– cSploit – Open – AudIT –
a.3 Network security
27033-5:2013 AIDA64 – E-Z Audit- Fern Wifi Cracker
ISO/IEC TR
a.4 Governance ACL – Delphix – Collibra
38505-2:2018
ISO 25119- Informer – WinAudit – Belarc Advisor – E-Z Audit –
a.5 Hardware
3:2018 ManageEngine ADAudit Plus
ISO/IEC/IEEE
a.6 Software WinAudit – Belarc Advisor
24748-8:2019
Skeddly– CloudStack - Netskope Cloud Security
ISO/IEC TR
a.7 Cloud Computing Platform– MultCloud – RightScale – Ormuco Stack –
22678:2019
Cloud Management - Ramce ERP
ISO
a.8 e-Commerce DeepCrawl – SEMrush
10008:2013
ISO 17572- Onspring – Form.com – ACL – Active@ – IDEA –
a.9 Database
2:2018 Xandria– AuditBoard- Delphix
a.10 Sourcing code ISO 3166 Debian – Clang- Analyzer
Business ISO/TS
a.11 Janco
Continuity 22318:2015
Disaster recovery ISO/IEC
a.12 Onspring -Janco- Delphix
testing 24762:2008
ISO
a.13 Social media NetBase- Tailwind– Clean Cloud
26000:2010
General data
Catalystone – Iubenda – Delphix– Cookie Assistant –
protection ISO/IEC
a.14 Ecomply– PYXI – Termly– BigID – consentEye –
regulation 27000:2018
OneTrust – TrustArc – Quantcast–Consenteye – ACL
(GDPR)
Table 1 illustrates that the main areas supported by CAATs are risk assessment,
security, and general data protection regulation (GDPR) respectively. These tools are
mainly supporting ISO 31000:2018 and ISO/IEC 27000:2018 standards. Moreover, there
are tools that cover multiple areas but not all of them; such as Janco, Delphix, Debian,
Ramce ERP, and ACL. These tools can be very promising to IS auditors due to their
area coverage. They can minimize the number of required tools to cover all the areas
as well as minimize the learning curve and training for the personnel and employees.
Web Support
Open source
Easy install.
Friendly UI
Ease-of-use
Serial
Free Demo
Macintosh
Windows
Training
Training
Reports
CAATs
Online
offline
Linux
Areas
Config. Audit
Web Support
Open source
Easy install.
Friendly UI
Ease-of-use
Serial
Free Demo
Macintosh
Windows
Training
Training
Reports
CAATs
Online
offline
Linux
Areas
32 MultCloud a.7 √ √ √ √ x x x √ x √ √ x
33 RightScale a.7 x √ √ √ x √ √ √ x x x x
Cloud
34 a.7 x √ √ √ x x x √ x x x x
Management
35 Informer a.5 √ √ √ √ √ x x √ √ √ √ √
36 SmartSolve a.1 √ √ √ √ x √ √ √ x x x x
37 MetricStream a.1 x x x x x √ √ √ √ x x x
38 Assure a.2 √ √ √ √ x x √ √ √ x √ √
ManageEngine
39 a.2 √ √ √ √ √ √ √ √ √ √ √ √
ADAudit Plus
a.2 a.4 a.9
40 Delphix √ √ √ √ √ x √ √ √ √ √ √
a.12 a.14
41 Catalystone a.14 √ x x x √ √ √ √ √ x √ √
42 CleanCloud a.13 √ √ √ √ √ √ √ √ √ x √ √
43 Collibra a.4 √ √ √ √ √ √ √ √ x x √ √
44 Qwerks a.1 x √ √ √ √ √ √ √ x x √ √
45 MKinsight a.1 √ √ √ √ √ √ √ √ √ √ √ √
46 Ramce ERP a.1-a.7 √ √ √ x √ √ √ √ x x √ √
47 Taskle a.1 √ √ √ √ x √ √ √ √ x √ √
48 Symbiant Tracker a.1 √ √ √ √ x √ √ √ x √ x √
49 R-CAP a.1 √ √ √ √ x √ √ √ √ x √ √
50 Isolocity a.1 √ √ √ √ x √ √ √ √ x √ √
51 Tailwind a.13 √ √ √ x √ x x x x x x x
52 Active@ a.9 √ √ √ √ √ x x x √ x x √
53 InfoZoom a.1 √ √ √ √ √ x x x √ x √ √
54 AuditBoard a.9 √ √ √ √ √ x x √ x x x x
55 DeepCrawl a.8 √ √ √ √ x x x √ √ x x x
56 ECAT a.2 √ √ √ √ x x x √ x x x x
57 form.com a.9 √ √ √ √ √ x √ x x x √ x
58 Aircrack-ng a.3 √ x √ √ √ x x x √ √ √ x
59 Belarc Advisor a.5 a.6 √ √ √ √ √ x x x √ √ √ √
60 WinAudit a.3 a.5 a.6 √ √ √ √ √ x x x x √ √ √
61 ADAudit Plus a.2 √ √ √ √ √ x x x x √ √ x
62 ZenGRC a.1 a.2 √ √ √ √ √ x √ √ √ x x √
The use of audit software tools differs from an organization to another. Table 2
presents a comparative study, which investigates the factors affecting the success of
each tool to decide the influence factors. This study shows that many tools prefer web
interface support than supporting various types of operating systems. This can be due
to several reasons, one of them could be due to their ease of use without further
installation steps. Another reason could be due to the required time to install and costs
to provide support to various operating systems. In addition, not all CAATs provide
any type of training, which can be very difficult for the IS auditors. Moreover, several
tools do not provide a free demo for the end users testing; which can be an important
selling factor for the tool under assessment.
Figure 1 examines the support of influence factors and their coverage in the sample
of tools as presented in Table 2. Both configurable audit reports and web-support come
on the top of the list where several CAATs support. Moreover, training can be either
supported online or user documentation, not many CAATs support both. It is notable
that more than 85% of these tools support configurable audit reports, see Figure 2.
However, the applicability of CAATs over several operating systems is lacking.
Moreover, only ~63% of the tools provide training, this percentage needs more root-
cause investigation.
There are also tools capable of performing joint tasks between two or more areas.
For example, some tools are used in data analysis, task management, interactive audit
trail, pivot tables, and graphs, e.g. IDEA – ACL – Delphix, which service each of
governance, database, and general data protection regulation areas. We can take one of
the recent areas mentioned before such as general data protection regulation (GDPR)
and consider how CAATs can help achieve GDPR compliance.
The objective of the GDPR audit is to help management assess how effective it is
being governed, monitored, accurately managed. In order to help the assessment and
assurance processes, the researchers have categorized GDPR auditing controls. There
are basic controls, such as access controls, data mapping, risk management, consent
management, incident management, policy management as well as sensitive data
identification. These controls evaluate the effectiveness of GDPR. GDPR is area (a.14)
in Table 1. After studying CAATs, the researchers found that each tool serves a set of
features that achieve GDPR compliance, see Table 3. Considering the tools supporting
the GDPR areas, there is not a single tool that can fully support all the basic controls.
Table 3. Distributed controls which achieve GDPR compliance (a.14)
Controls
Sensitive
Tools Access Data Risk Consent Incident Policy
data
controls mapping mgmt. mgmt. mgmt. mgmt.
identification
Catalyst one √ √ √ √ √ √ √
Iubenda √ √
Delphix √ √ √ √
Cookie Assistant √ √ √
Ecomply √ √ √ √ √
PYXI √ √ √
Termly √ √
BigID √ √ √ √ √ √
consentEye √ √ √
OneTrust √ √ √ √ √
The GDPR area is sensitive to all other areas, e.g. business, cloud computing, social
networking, within the IT department and there are many common functions among
them. IS auditor need to get a comprehensive report, with the status of all the tasks in
IT department areas, to help to review every task efficiently. For example, it is
important to review security according to existing standards and guidelines. However,
there is not one tool that combines all these controls with various domains. Table 3
shows that Catalyst is the only tool supporting all the seven GDPR controls, while
Ecomply, consentEye, and OneTrust support five controls each.
There are some standards such as ISO 27001 and ISO 27002 which help
organizations to ensure that they have effective information security programs. ISO
27001 was originally created to help to secure both government services and citizen
data at the service provider's side. The use of ISO 27001 ensures the GDPR principles,
and the appropriate technological and organizational measures are all preserved to
protect information [21]. It helps organizations to define responsibilities, such as who
is responsible for information assets and who can authorize access to these data. Also,
ISO 27001 provides independent accreditation for information security management
systems, while ISO 27002 is a code of practice that is not accredited by external parties.
Either standard will help to ensure that an organization has strong aiding controls [22].
Although the factors presented in Table 2 can provide added competitive features,
there can be some challenges in software auditing tools, such as:
- Lack of compatibility of web application over different browsers.
- The user interfaces need adaptation over different environments.
- Reporting tool needs some improvements and tailored adjustments.
- Online support can be out-of-date and/or incompetent.
- Upgrading the system can be faulty and costly.
- Lack of support forums and community for new CAATs.
- The increased learning curve for advanced features.
- Reports with multimedia charts and tabular information are not easily provided.
- The need for multiple CAATs can be highly expensive.
These challenges can be the reason for most of the prominent defects that have
emerged while using CAATs. Yet, one of the most prominent flaws in all CAATs is
that there is no single tool capable of covering all the information systems auditing
tasks together.
As a result of this study, there are several challenges that emerged. The main
challenge is the difficulty to support all areas in one tool, which leads some
organizations to use multiple CAATs. This can be very expensive and lead to increased
complexity while integrating the resulting reports per each tool. Consequently, the
organization will waste more time and effort into training its personnel. Repeating the
implementation of the common controls can generate faulty results. All these obstacles
can generate several errors in the final reports and gaps in covering the audit tasks.
IS auditors use software auditing tools such as CAATs to help in performing all auditing
process tasks. Using these tools helps to measure the accuracy of audit tests, reduce
reviewing time, provide ad-hoc reports, and early detection of deviations. There are
many factors influencing the use of audit software. Recent research finds that the two
important factors that affect auditor’s decision to whether to use CAATs or not are
performance expectancy and facilitating conditions. However, there are other factors
that help in determining the appropriate tool to perform the tasks during the auditing
process. This paper aims to find these influential factors that help in choosing suitable
auditing tools to support the success of the required audit tasks.
In order to achieve this target, the researchers investigate several factors of selecting
these tools. Nevertheless, there are also many tools that serve the tasks of IS auditing
in all areas. As a result, we found that the most recent auditing tools comply with ISO
standards, which provide accurate guidelines to help the auditors achieve high-quality
audit results. Each of these CAATs cannot solely support all areas of IS auditing. This
can be very challenging to IS auditor to generate a comprehensive and accurate report
with the minimum cost and effort. As future work, the researchers aim to create a
framework for an integrated IS auditing tasks in one comprehensive tool.
References
1. Coderre, David, and Royal Canadian Mounted Police. "Global technology audit guide:
continuous auditing implications for assurance, monitoring, and risk assessment." The
Institute of Internal Auditors (2005): 1-34.
2. Sun, Chia Ming. "The Adaptation and Routinization Processes of A Continuous Auditing
System Implementation." (2012).
3. Braun, Robert L., and Harold E. Davis. "Computer-assisted audit tools and techniques:
analysis and perspectives." Managerial Auditing Journal 18.9 (2003): 725-731.
4. Al-hiyari A. & Hattab E. Factors that influence the use of computer assisted audit
techniques (CAATs) by Internal Auditors in Jordan. (2019). ISSN: 1096-3685.
5. Li Zhang, Amy R. Pawlicki, Dorothy McQuilken, and William R. Titera (2012) The
AICPA Assurance Services Executive Committee Emerging Assurance Technologies
Task Force: The Audit Data Standards (ADS) Initiative. Journal of Information Systems:
Spring (2012), Vol. 26, No. 1, pp. 199-205.
6. ACL, The ACL Audit Analytic Capability Model: Navigating the journey from basic data
analysis to continuous monitoring. (2011) - A White Paper.
7. Henderson, J. C., & Lee, S. Managing I/S design teams: a control theories perspective.
Management Science, (1992). Vol 38, No.6, pp.757-777.
8. Keil, M., Rai, A., & Liu, S. How user risk and requirements risk moderate the effects of
formal and informal control on the process performance of IT projects. European Journal
of Information Systems. (2012). Vol. 22, No.6, pp.650-672.
9. Lu, Y., Xiang, C., Wang, B., & Wang, X. What affects information systems development
team performance? An exploratory study from the perspective of combined socio-technical
theory and coordination theory. Computers in Human Behavior, (2011). Vol. 27, No. 2,
pp.811-822.
10. Vasarhelyi, M. A., Alles, M., Kuenkaikaew, S., & Littley. The acceptance and adoption of
continuous auditing by internal auditors: A micro analysis. International Journal of
Accounting Information Systems. (2012). Vol.13, pp.267-281.
11. Gonzalez, G. C., Sharma, P. N., & Galletta. The antecedents of the use of continuous
auditing in the internal auditing context. International Journal of Accounting Information
Systems, D. F. (2012). Vol. 13, No.3, pp.248-262.
12. Masli, A., Peters, G. F., Richardson, V. J., & Sanchez. Examining the potential benefits of
internal control monitoring technology. The Accounting Review, J. M. (2010). Vol. 85,
No.3, pp.1001-1034.
13. Janvrin, D., Bierstaker, J., & Lowe. An examination of audit information technology use
and perceived importance. Accounting Horizons, D. J. (2008). Vol. 22, No. 1, pp.1-21.
14. Rezaee, Z., Sharbatoghlie, A., Elam, R., & McMickle. Continuous auditing: Building
automated auditing capability. Auditing: A Journal of Practice & Theory, P. L. (2002).
Vol. 21, No. 1, pp.147-163.
15. Ramamoorthi, Windermere,The Pervasive Impact of Information Technology on Internal
Auditing, Institute of Internal Auditors Inc., Ch. 9. (2004).
16. Mahzan, N. & Verankutty, IT auditing activities of public sector auditors in Malaysia.
African Journal of Business Management, F. (2011).5 (5), 1551-1563.
17. Shukarova Savovska, K.& Sirois, B. A. Audit Data Analytics: Opportunities and Tips
(English). Centre for Financial Reporting Reform (CFRR). Washington, (2018). D.C.:
World Bank Group.
18. Bierstaker, J., Janvrin, D., & Lowe. What factors influence auditors' use of computer-
assisted audit techniques? Advances in Accounting, D. J. (2014). 30(1), 67-74.
19. Romney, M. B., Steinbart, P. J., & Cushing. Accounting information systems (Vol. 2). B.
E. (2000). Upper Saddle River, NJ: Prentice Hall.
20. Alcarraz, Gerardo D. and others. “Certified Information Systems Auditor – CISA - Review
Manual 2009”, (2009).
21. Calder. EU GDPR: a pocket guide. IT Governance Publishing Ltd., A. (2018).
22. Tzolov. One Model for Implementation GDPR Based on ISO Standards. In 2018
International Conference on Information Technologies (InfoTech) (pp. 1-3). IEEE., T.
(2018, September).
23. Mahzan, N., & Lymer. Examining the adoption of computer-assisted audit tools and
techniques: Cases of generalized audit software use by internal auditors. A. (2014).
Managerial Auditing Journal, 29(4), 327-349.
Appendix
Table 4. Tools with their URL references – last checked on 19th June 2019
Tool name Website URL reference
ACL https://www.acl.com/
Active@ http://www.lsoft.net/
AIDA64 https://www.aida64.com/products/aida64-network-audit
https://www.softpedia.com/get/Programming/Other-Programming-
Aircrack-ng
Files/Aircrack-ng-GUI.shtml
Analyzer https://clang-analyzer.llvm.org/
ARBUTUS https://www.arbutussoftware.com/products-solutions/audit-analytics
Assure https://www.asuresoftware.com/
AuditBoard https://www.auditboard.com/
Belarc-advisor https://download.cnet.com/Belarc-Advisor/3000-2094_4-10007277.html
BigID https://bigid.com/
Catalystone https://catalystone.com/gdpr-data-audit-tool/
Clang www.createaclang.com
CleanCloud https://cleancloudapp.com/
Cloud
https://www.virtustream.com/software/xstream/features
management
CloudStack https://reviews.financesonline.com/p/apache-cloudstack/
Collibra https://www.collibra.com/
Consenteye https://www.consenteye.com/
Cookie Assistant https://www.cookieassistant.com/
cSploit https://www.apksum.com/app/csploit/org.csploit.android
DATEV https://www.datev.com/
Debian https://www.debian.org/security/audit/tools
DeepCrawl https://www.deepcrawl.com/pain-point/regular-site-audit/
Delphix https://delphix.github.io/
https://ecat-group.com/audit-management-software/?utm_source=capterra-
ECAT
visit-website&utm_medium=referral&utm_campaign=Capterra
Ecomply https://ecomply.io/
E-z audit http://www.ezaudit.net/features/
Fern Wifi
https://n0where.net/fern-wifi-cracker
Cracker
Form.com https://form.com/
IDEA https://www.casewareanalytics.com/products/idea-data-analysis
https://informer.freshdesk.com/support/solutions/articles/5000665438-
Informer
auditfile
InfoZoom https://www.softlakesolutions.com/
Isolocity https://www.isolocity.com/
Iubenda https://www.iubenda.com/en/gdpr
Janco https://sourceforge.net/projects/janco/
Lynis https://cisofy.com/lynis/
Manage engine https://www.manageengine.com/products/active-directory-
ADAUDIT Plus audit/download.html
MetricStream. https://www.metricstream.com/solutions/audit-management.htm
MKinsight http://www.mkinsight.com/functionality.aspx?id=9
MultCloud https://project-management.com/multcloud-software-review/
NetBase https://www.netbase.com/
Tool name Website URL reference
Netskope cloud https://www.netskope.com/
OneTrust https://www.onetrust.com/
Onspring https://www.onspring.com/#difference
Open-Audit https://opmantek.com/network-discovery-inventory-software/
Ormuco Stack https://ormuco.com/
PYXI http://www.pyxi.co.uk/
Quant cast https://alternativeto.net/software/quantcast/
Qwerks https://getqwerks.com/
Ramce ERP http://www.ramco.com/
R-CAP http://www.r-cap.com/
RightScale https://reviews.financesonline.com/p/rightscale/
SEMrush https://www.semrush.com/
Skeddly https://cloudcheckr.com/partners/skeddly/
SmartSolve https://www.pilgrimquality.com/
Symbiant tracker https://www.symbiant.co.uk/
Tailwind https://www.tailwindapp.com/
Taskle https://www.taskle.com/
Teammate http://www.teammatesolutions.com/audit-management.aspx.
Termly https://termly.io/
TrustArc https://www.trustarc.com/
WinAudit https://www.techspot.com/downloads/2307-winaudit.html
Xandria https://www.syslink-xandria.com/en
http://unbouncepages.com/reciprocity-zengrc-risk-management-
ZenGRC
gdm/?directory=Risk_Management&source=SoftwareAdvice