ADDIS ABABA UNIVERSITY

SCHOOL OF INFORMATION SCIENCE

Department of information science and

system (Information science)

Literature review assignment

 Done by: Bikila Boja & Barnabas Dereje

 Submitted to: Dr. Temtim(Phd)
 ARTIFACT EVALUATION IN INFORMATION SYSTEMS
DESIGN-SCIENCE RESEARCH (DSR)

1. Abstract
Design science in Information Systems (IS) research pertains to the creation of artifacts to solve
real-life problems. Research on IS artifact evaluation remains at an early stage. In the design-
science research literature evaluation criteria are presented in a fragmented or incomplete
manner. This article mainly focuses on two main components on evaluation of artifacts.

1. Hierarchy of evaluation criteria for IS artifacts organized according to the dimensions of

a system (goal, environment, structure, activity, and evolution),
2. A model providing a high-level abstraction of evaluation methods, and

2. Introduction
Design-science research (DSR) supports a pragmatic research paradigm promoting the creation
of artifacts to solve real-life problems (Hevner et al., 2004; Simon, 1996).

This paper highly focuses on artifact evaluation in DSR. As specified on the paper, DSR, a
distinction can be made between design science (which reflects and provides guidance on artifact
construction and evaluation) and design research (which constructs and evaluates specific
artifacts) (Winter, 2008).

Evaluation methods are also presented in a fragmented way and are unrelated to evaluation
criteria. There are many methods through which information system artefact can be evaluated.
IS artifacts are systems to be evaluated. The specific research questions addressed in this paper
when evaluating information system artifacts are:

 Which criteria are proposed in the DSR literature to evaluate IS artifacts?

 Which ones are actually used in IS published research?
 How can we structure this set of criteria?
 Finally, which evaluation methods emerge as generic means to evaluate IS artifacts?

The paper is organized as follows.

 Describing the research.

 Evaluating the artifact resulting from this research and
 Finally conclude evaluation system of information system artifacts.
 3. Artifact evaluation criteria to generic evaluation methods
As a design-science research this research describes, DSR process and the resulting artifact. The
main components of this artifact are a model of generic evaluation methods, and examples of
evaluation methods (instances of this model).

3.1. Design research process

The fundamental approach of this research is to combine DSR evaluation theory with practice.
When we say theory, it was provided by the IS design-science literature and Practice was
provided by a sample of design-research papers.

4. Artifact evaluation methods

Simon (1996) considers design artifacts as systems. He characterizes artifacts in terms of
functions (activities), goals, and adaptation (evolution). He also distinguishes the structure of the
artifact from the environment in which it operates. Information system artifacts, as specific
design artifacts, can therefore be considered as systems.

4.1. Hierarchy of criteria for IS artifact evaluation

As in this paper, there are five dimension of hierarchy for information system artifact evaluation.
Which are listed explained as below:

a. Goal: Characterized by three criteria. Which are:

 Efficiency: the degree to which the artifact produces its desired effect.
 Validity: the degree to which the artifact works correctly with out error.
 Generality: the scope of the system.

b. Environment: the environment of IS artifact comprises three components. Which are

 People: Is the artifact is ethical, understandable, easy to use and easily utilized.
 Organization: shows either the artifact is utilized, fit with the organization and has
low side effect in the environment.
 Technology: this shows either the system is easily copied up with recent
technology or not.

c. Structure: can be accessed in terms of:

 Completeness: either the system is completed or not
 Clarity: ease of clarity i.e the artifact is clear to understand and explain
 Simplicity: not complicates artifact is preferable
 Level of detail: shallow or deeply explained artifact.
 Consistency: continuity of the system in the future
d. Activity: can be evaluated in terms of the following criteria.
 Completeness: is the system complete to transfer needed information?
 Consistency: in terms of the consistency or continuity of the system.
 Accuracy: is the system or artifact is accurate or free from error?
 Performance: is the system performance is good?
 Efficiency: is the system is efficient in terms of resource (time, money)

e. Evolution: can be evaluated in terms of:

 Robustness: the ability to respond to fluctuations of the environment
 Learning capability: the capacity of a system to learn from its experience and the
reactions of the environment.

5. Conclusion
In conclusion information system artifact can be evaluated based on systems criteria’s like,
systems robustness, reliability, validity, efficiency and effectiveness of the system,
performance of the system, consistency of the system, simplicity and clarity of the system
and other attributes.

6. References

1. https://aisel.aisnet.org/pacis2014/23/
2. Hong, S.-Y., Kim, J.-W. and Hwang, Y.-H., "Fuzzy-semantic information management
system for dispersed information", 52:(1), 2011, 96-105.
3. Bera, P., Burton-Jones, A. and Wand, Y., "Guidelines for designing visual ontologies to
support knowledge identification", 35:(4), 2011, 883-908.
4. Cleven, A., Gubler, P., Hüner, K.: Design Alternatives for the Evaluation of Design
Science Research Artifacts. In: Proceedings of the 4th International Conference on
Design Science Research in Information Systems and Technology (DESRIST 2009).
ACM Press, Malvern (2009).
5. Nunamaker, J.F., Chen, M., Purdin, T.D.M.: Systems Development in Information
Systems Research. Journal of Management Information Systems 7, 89–106 (1990/1991)