Using ACT-R for Human Error Identication

Romke van der Meulen (R.van.der.Meulen.5@student.rug.nl)

Department of Articial Intelligence, Nijenborgh 9 9747 AG, Groningen, The Netherlands Abstract
Traditional techniques for Human Error Identication mostly consist of hand-written, high level task models, which often focus on goals and less on perception. Cognitive modelling could avoid this problem by modelling human performance at a much lower level. This would not only rigorously identify possible errors, but also their cause. ACT-R is a cognitive architecture that could be used for this purpose. It can model human behaviour and reproduce human performance data. It can also reproduce human errors, by selecting incorrect production rules or retrieving incorrect chunks. It is also in a position to offer insight into the inuence of strategy on errors. However, it would still require a human modeller to create the ACT-R task model, before error identication can take place. It is unclear whether automatically generated ACT-R models will be successful in error identication. More research is needed. Keywords: ACT-R; cognitive modelling; HEI; human error identication; error reproduction; strategy

makes use of production rules with an IF-THEN structure. Each production rule has an associated level of activation, which determines which rules are considered rst. Several cognitive systems, including perceptual and motor modules, are connected to this production system, allowing the modelling of complete tasks.

ACT-R and partial matching

At rst, the implementation of the ACT-R production system used an all-or-nothing strategy; Production rules only red if all their preconditions were met. This allowed for errors of omission1 : if the activation of a production rule was too low it might not be retrieved (in time). However, it did not allow for errors of commission2 . This problem was solved by allowing partial matching within the production system (Lebi` re, Anderson, & e Reder, 1994; Anderson, Reder, & Lebi` re, 1996). For e each missing or differing precondition, a production rule receives an activation penalty. Then the production rule with the most activation is red, even if this rule does not fully match current conditions. Some noise is added to the activation values of the rules, to model extraneous inuences. Under these circumstances, it can occur that an incorrect production rule is red, if it is close enough to the correct rule. In this case, the model exhibits an error of commission. The total level of activation that can be used in ACTR remains constant. If, therefore, the cognitive load is increased by, for example, a secondary task, the amount of activation for each item drops. In this case, errors of omission or commission are more likely, mirroring what we nd in human experiments in multitasking. This version of ACT-R using partial matching was later used by Lebi` re, Anderson, and Bothell (2001) to model e the Air Trafc Control task. This model did well in reproducing human performance, and even did very well in matching the number of errors made in 8 different error categories.

Introduction
No one who cannot rejoice in the discovery of his own mistakes deserves to be called a scholar. Donald Foster Errors occur every day, in any kind of situation, for any kind of reason. To learn from our mistakes, we must learn what went wrong. When the error occurred while man and machine were working together, one can look for the cause of the error in the machine (systems approach) or in the man: this is the eld of Human Error Identication. In the eld of Human Error Identication, a great number of techniques has been developed (Kirwan, 1992). A trait shared by many of these techniques, however, is that they rely heavily on the insights of the modeller, and are therefore subjective and difcult to automate. These techniques often focus on goals, and sometimes completely disregard perceptual factors, factors often relegated to the eld of Human Factors research. Some of these shortcomings could be avoided by using cognitive modelling to identify human errors. These models have the advantage of modelling human performance very closely, and at a very low level. If such models can be used for human error identication, the results could be quite rigorous in identifying not only possibilities for human error, but also what causes such errors and in modelling alternate approaches to the task that might avoid these errors. ACT-R (Anderson, 1993) is a cognitive architecture that can be used to model human task performance. ACT-R

1 The Technique for Human Error Rate Prediction (THERP) recognizes three types of error: errors of omission, where a required action is not performed, errors of commission, where the wrong actions is carried out or the right action is carried out incorrectly, and extraneous errors, where wrong (unrequired) actions are carried out (Kirwan, 1992) 2 At least not in the production system. It was still possible for an erroneous chunk to be retrieved; more on this later.

Chung and Byrne (2008) used this modied ACT-R model in an attempt to reproduce the post-completion error3 patterns they found in their experimental data. They did this by introducing two rules that can re once the main task has been completed. One was the correct rule, which stated that the post-completion task must rst be completed, and the other the incorrect rule, stating that a return to the main control sequence was warranted. Using the ACT-R partial matching system, it can occur that the incorrect rule is accidentally red. In a second condition, they added a visual cue, which is automatically picked up by ACT-Rs visual module, leading to the activation of additional knowledge that avoided the postcompletion error. Using fairly standard model parameters, and without doing excessive parameter-tting, they found that the ACTR model produced a 14.1% PCE frequency over 200 trials, where their target had been 5-15%, and that no PCE occurred when a visual cue was presented. They concluded that this was a successful application of ACT-R as a model of their experimental data. This shows that ACT-R with partial matching is successful in reproducing not only human error rates, but even human error types. However, in both cases the ACT-R model was used to postdict experimental data. It remains unclear whether ACT-R can be used to predict human errors.

cess, this can lead to an incorrect chunk being created. (Anderson & Lebi re, 1998) e In the second case, it could be that the information in the chunk is actually correct, but it is retrieved in the wrong situation. For example, a chunk encoding 3 + 5 = 8 might be retrieved when the task at hand is to compute 3 + 4, leading to the wrong answer. This is possible because the ACT-R memory retrieval system is not exact: it works through association. In this example, the chunk in question is associated to the task because both task and chunk are in the domain of mental arithmetic, and both relate to the concept 3. (Anderson & Lebi re, e 1998) In either case, ACT-R can learn to prevent such errors through feedback. Whether the retrieved chunk was unsuitable for the task or simply incorrect, through feedback the association between the task and the chunk can be lowered. This will ensure that, possibly after several trials, the association falls to the point where the chunk is no longer retrieved, and the correct chunk may be found.

Ease of use
One drawback in the use of ACT-R is that to model a task for the purpose of error identication, the production rules must rst be found. This can be done by hand by a human modeller, but this requires a large amount of time and efforts, not to mention extensive domain knowledge. Fortunately, there are other ways to create ACT-R task models. ACT-R can learn a task, given enough initial knowledge. However, whether such learned models will be successful in identifying possibilities for human error is an unanswered question. John, Prevas, Salvucci, and Koedinger (2004) provide another alternative. They have attempted to make the creation of ACT-R models simple enough for usability experts to do, by demonstrating the execution of the task on an HTML mock-up. This is then translated into a Keystroke-Level Model. Like the learned models, however, it remains unclear whether such generated models will be successful in identifying possible errors.

Errors in retrieval
ACT-R can produce errors not only through selecting an incorrect production rule, or failing to retrieve the correct one: it may also be that the production rule is correct, but the output is not. This happens when incorrect knowledge is present in the system in the form of chunks, or when the wrong chunk is retrieved for a specic situation (Anderson & Lebi re, 1998). e In the rst case, a chunk in an ACT-R model contains incorrect facts. For example, a chunk may encode the spurious fact that 1 + 2 = 4. Such a chunk can come from several sources. It could have been explicitly created by the modeller, in an attempt to model human error or when trying to see how ACT-R handles the erroneous data. It can also be the case that the chunk was create through the ACT-R learning process. This learning process compiles the result of a complex production procedure into a memory chunk, so that next time the result can be retrieved without having to be reproduced. If any of the production rules that rst produced the result are incorrect, or if noise entered during the production pro3 A post-completion error is a special type of error of omission: it occurs when some main task has been accomplished, and the subjects neglects to perform an extra required action. An often cited example is forgetting to retrieve the original after making a photo copy.

Strategy
Another problem that all HEI techniques, including any ACT-R approach, have to deal with, is the issue of individual strategy. Not all users approach a task with the same strategy. Some will make different decision from others. This implies that users of strategy A are exposed to different error possibilities than users of strategy B. It is also possible for one individual to sometimes use one strategy, and sometimes another. It is even possible for an individual to change strategies during the task, which

often leads to a number of errors. To completely identify all possible errors, models of each strategy need to be created, and the implications of switching strategy understood. Both ACT-R and traditional HEI techniques would need to identify each strategy and model them. Parts of the task that are the same for all strategies can be reused in the model, but the differing parts must be individually modelled. It is possible for the ACT-R modelling approach to gain an advantage in this respect. All strategies can be included in the same ACT-R model, by collecting the production rules of the different strategies. To model individual preferences, the activation levels of production rules in one strategy can be heightened or lowered. It might even be possible to model in ACT-R the effects of an individual changing strategy at a given point in the task. This would entail lowering the activation of the current strategy, and heightening that of the chosen alternate strategy. ACT-R is in a good position to model the kinds of errors that can occur in this situation, since it is precisely clear what information is available at this point, and which is lacking. The implementation of such an ACT-R model and the possibilities it offers for error identication can make a good subject for future research.

needed to determine whether generated ACT-R models are successful in Human Error Identication. It is my opinion that the use of ACT-R in Human Error Identication is a desirable goal. It is rigorous, quantiable, reproducible and produces measurable results. One cannot model errors without modelling the task, and in such a way that the ACT-R model can sufciently reproduce human performance data. This will lead to more complete and objective models than current HEI techniques can hope to attain. I therefor encourage more research into the use of ACT-R for Human Error Identication.

References
Anderson, J. R. (1993). Rules of the mind. Hillsdale, NJ: Erlbaum. Anderson, J. R., & Lebi re, C. (1998). The atomic come ponents of thought. In (pp. 297342). Mahwah, NJ: Psychology Press. Anderson, J. R., Reder, L. M., & Lebi` re, C. (1996). e Working memory: Activation limitations on retrieval. Cognitive Psychology, 30, 221256. Chung, P. H., & Byrne, M. D. (2008). Cue effectiveness in mitigating postcompletion errors in a routine procedural task. International Journal of Human-Computer Studies, 66, 217232. John, B. E., Prevas, K., Salvucci, D. D., & Koedinger, K. (2004). Predictive human performance modeling made easy. In E. Dykstra-Erickson & M. Tscheligi (Eds.), Proceedings of acm chi 2004 conference on human factors in computing systems (pp. 455462). Vienna, Austria: ACM Press. Kirwan, B. (1992). Human error identication in human reliability assessment. part 1: Overview of approaches. Applied Ergonomics, 23(5), 299318. Lebi` re, C., Anderson, J. R., & Bothell, D. (2001). e Multi-tasking and cognitive workload in an act-r model of a simplied air trafc control task. In Proceedings of the tenth conference on computer generated forces and behavior representation. Norfolk, VA: SISO. Lebi` re, C., Anderson, J. R., & Reder, L. M. (1994). Ere ror modeling in the act-r production system. In Proceedings of the sixteenth annual meeting of the cognitive science society (pp. 555559). Hillsdale, NJ: Erlbaum.

Conclusion
Human Error Identication is at present still a subjective process. This can be improved by the use of cognitive modelling. One possible modelling approach is the use of the cognitive architecture ACT-R. ACT-R can produce errors of omission and errors of commission by using partial matching in its production system. This version of ACT-R has been used successfully to postdict experimental data on different kinds of human errors. ACT-R can also produce errors through the retrieval of incorrect knowledge. Unfortunately, the creation of an ACT-R task model is an involved process, and still involves a human modeller with extensive domain knowledge. There are ways of automatically generating ACT-R models, but it is unclear whether such models will be successful in identifying (all) possibilities for human error in the task. ACT-R is in a good position to predict errors that can occur due to a change in strategy during task execution. All task strategies can be held in the same ACT-R model, using production rule activation levels to choose between them. This makes a good subject for future research. In closing, it can be said that ACT-R has a number of advantages as a technique for human error identication. However, practical applicability is still far off, as long as ACT-R models must be built by hand. More research is