Exploring the Use of Avatars to Sign Examination Questions

Mhairi McAlpine, Scottish Qualifications Authority.

Introduction
Background and Context
Since the establishment of the Scottish Government in1999, much progress has been made to recognising the value of Scotlands indigenous and community languages. In 2003, the executive committed to develop[ing] a new focus for Scotland's languages recognising both our heritage and our diversity (Scottish Executive, 2003) which not only committed them to providing secure status for Gaelic through a Gaelic Languages Bill (which passed into law in April 2005), but also committed them to introduce[ing] a national language strategy to guide the development and support of Scotland's languages, including British Sign Language (ibid) to cater for the approximate 5,000 people who use BSL as their first or preferred language. (Scottish Executive, 2005) BSL is structured in a completely different way to the English language and has its own specific grammar, additionally there are substantial regional variations. Scottish BSL signers use different signs for many words than English signers and there are also regional and religious variations within Scotland. BSL, like other sign languages, uses manual communication, body language and lip patterns instead of sound to convey meaning. Natural spoken language is however phonetically based, giving hearing children an advantage when learning to read. As a consequence, levels of literacy are typically several years behind hearing people of the same age (Powers et al 1998) and it is estimated that the average literacy of deaf adults in Scotland does not exceed the skills of a nine-year old hearing child (Scottish Executive, 2005). The Disability Discrimination Act (1995) requires that reasonable adjustments are made for disabled students taking examinations, and under the Disability Equality Duty (2006), all public sector bodies are required to take into account disability even where this favours disabled people, however care must nonetheless be taken in as accurate a translation as possible. The Joint Council for Qualifications (JCQ, 2005) notes that many signs are iconographic and therefore explain the meaning of the subject-specific word being assessed: for example the sign for perimeter draws the outline of the shape in space and so indicates that the perimeter is the distance around the outside of the shape. Going on to suggest that such words should be finger spelled or an alternative way of interpreting used.

Drivers within SQA

The Scottish Qualifications Authority is is an Executive Agency of the Scottish Executive responsible for the development, accreditation, assessment and certification of qualifications, other than academic degrees, in Scotland. It is a non-departmental public body, partly funded by the Scottish Executive Education Department, employing 650 staff, based in Glasgow and Dalkeith. It offers a range of qualifications through schools, colleges and training providers in both academic and vocational subjects. Additionally it also runs the annual Scottish Survey of Achievement for children in the 5-14 age range under contract from the Scottish Government. Several of the examinations offered by the SQA are constructed from banks of items which are written in advance and then compiled into tests in time for the annual diet, and in some cases using an on-demand model, it is anticipated that the use of both itembanking and ondemand assessment will increase over time. Approximately 25% of hearing impaired candidates (not all native BSL signers) as for an interpreter for their examinations, however it

is recognised that there is a national shortage of suitably qualified interpreters (Scottish Executive, 2005). As a result some centres are using interpreters below the minimum recommended standard in BSL. These interpreters in particular are struggling to interpret question papers produced from itembanks as there is no access to the question paper beforehand and no way of telling which questions may come up. Difficulties were also experience by interpreters as a number of subject specific terms are very difficult to sign without revealing the answer to the question as the signing is frequently iconographic (JCQ, 2005) and/or has wide variations. The use of video was considered, however rejected as prohibitively expensive as well as requiring high specification internet connections which could not be always be assumed particularly in the rural areas of Scotland.

Using an Avatar Signed Approach

Another solution was sought, and the possibility of using avatars to sign the exams was considered. This has a number of advantages over video production: good quality video needs relatively sophisticated recording facilities; ensuring continuity is problematic as materials are updated since the same signer, clothing, lighting, and camera settings must be maintained; stitching sequences together is impractical, requiring a complete re-shoot if minor changes are needed; despite improvements in network bandwidth and storage capacities, transmitting and storing high quality video consumes significant resources. The University of East Anglia (UEA) had been researching the use of avatars to provide signed support for native BSL signers for a number of years, and has been used in materials commissioned by the BBC to develop BBC JAM resources for Literacy for the Deaf (BBC, 2007) and after initial contact it was felt that given the need for support in this area, and SQAs intention to expand itembanking within examinations, this approach would be worth exploring. Together with the UEA, SQA was awarded 60K from the Teaching and Learning Research Programme, a fund distributed by the Economic and Social Research Council (ESRC) - supplemented by a contribution from SQA to build a demonstrator system.

Methodology
Ethical Concerns and Approach
The Economic and Social Research Council sets out a research ethics framework(REF), which includes minimum requirements for researchers funded through their programmes to adhere to (ESRC, 2005). This document outlines six core principles which inform this framework and which researchers should consider in relation to any research undertaken. Issues were reviewed under this framework; risks were identified and evaluated and then consideration was given to how these risks may be minimised or eliminated. Table 1: Ethical Principles, Issues Identified and Actions taken REF Principle Issue Identified
Focus groups with Deaf participants should be conducted in the most accessible form in order to derive high quality feedback Questionnaires given to Deaf participants must be clear and accessible The consent of participants was requested prior to the session

Action Taken
The focus group and feedback sessions were led by a native BSL signer. Some difficulties were nonetheless experienced, particularly by older participants as the researcher was English and regional variations in sign language made it more difficult to understand than a Scottish signer. The feedback questionnaires were written in Plain English and designed to require minimal writing. They were reviewed by the RNID prior to distribution to ensure that they would be suitable. : In both cases with the school pupils, formal signed consent was requested; in the case of the adult research subjects, verbal consent only was required

Research should be designed, reviewed and undertaken in a way that ensures its integrity and quality Confidentiality of information supplied by

Consent for filming of participants for write up purposes was sought.

research subjects and the anonymity of respondents must be respected

Consent of the participants for filming of the focus groups were requested at the start. With hindsight, it may have been advisable to request this permission in advance from the school participants, however as formal consent had been sought for the overall purpose, the children were above the age of Gillick competency and verbal consent was requested prior to the session, it was regarded that consent could be considered freely given The TLRP project requested that all primary data sources were deposited in the archives. We had neglected to ask permission for this, and were reluctant to go back to participants to request this, having assured them that their anonymity would be protected (as far as was reasonably possible within a small and enclosed community). We therefore declined to submit the tapes of the feedback and focus group sessions. Given that this research was being associated with an awarding body, any confusion over this matter may lead to understandable concern. We were careful therefore the emphasise at the start of each session, that this was purely research at an experimental phase; that no results from these trials would be used for anything other than research purposes and that there was no current plans to introduce such a system on an operational basis

Provision of filmed data into research funders archives

Research staff and subjects must be informed fully about the purpose, methods and intended possible uses of the research; what their participation in the research entails and what risks, if any, are involved Research participants must participate in a voluntary way, free from any coercion Harm to research participants must be avoided The independence and impartiality of researchers must be clear, and any conflicts of interest or partiality must be explicit

Concern was noted from the initial stakeholder focus group that this approach may be used to replace communication support workers for candidates to access examinations

To encourage feedback, a 25 gift voucher incentive was offered to all participants in the feedback sessions .No issues were identified in this area Focus group and feedback sessions were attended by a member of the project team from SQA. Focus group sessions were led by a staff member from the RNID who had been encoding the avatars.

We considered the issue of coercion however it is normal practice in both the SQA and RNID to offer a small gift to people who have given up their time to assist us with evaluation. We did not thus regard it as coercion No action was taken

It was explained that the SQA staff member attending was a member of the SQA Computer Assisted Assessment team, and as such had no influence over special assessment arrangements or examination marking This was explained during the initial briefing prior to the focus group taking place together with an explanation that someone who had been directly working on the encoding was in our opinion the best people to elicit the required information.

Question Selection and Creation

The avatar signing process is done by first recording a video of signs used A set of over 400 questions were given to RNID by SQA and made public using a creative commons licence. A two day session was organised in London with professional interpreters and a teacher of the Deaf. 150 questions were chosen as being most appropriate for interpretation to BSL and these were recorded to video. The video was then passed to learning technology specialists at the RNID and common vocabulary from these questions was then notated in a form of signwriting called HamNoSys, which records the movements that the signer is making to perform each particular sign. These signs were again licenced under a creative commons agreement and added to the existing lexicon that is held by the RNID. Figure 1: Example of HamNoSys

To produce the questions, the individual signs were stitched together along with any contextual amendments, before being translated into an XML Schema (SiGML) which encodes those

movements. The XML schema is passed to specially developed avatar software which then moves the avatar according to the encoded instructions. QTI 2.0 encoded questions are then annotated with the SiGML notation. A specially designed delivery vehicle based on the R2Q2 development with incorporated avatar reads these questions and uses the SiGML to sign the question to the candidate. In this manner approximately 50 questions were produced. A BSL specialist in Scotland was then engaged to evaluate and comment on the overall quality of the questions particularly in relation to regional variations.

Application Creation and Avatar Incorporation.

The architecture of the assessment system development by the project uses a standard web browser to display questions from an assessment engine with an embedded panel containing a Java based plugin. Pages are generated by an internet server running an assessment engine developed from R2Q2 (Wills et. al., 2006) which conforms to the QTI international specifications for the structuring of assessment data. a modification was added to R2Q2 to add buttons to the displayed web page which triggered a script to display the corresponding signing using the avatar plugin on finding sign (SigML) gesture files embedded in the QTI. The original text is still preserved, so traditional questions are displayed in conventional style, with the avatar to one side.

Figure 2: Anna the Avatar

As R2Q2 in its base form only handles presentation of a single QTI question, additional functionality was added to manage the concept of an assessment Centre with associated candidates and to allow selection of one of a number of tests. The assessment engine created also collected statistics on the performance of candidates including logging how the avatar was used. It had been hoped that these statistics could be analysed as part of the evaluation, however during the user trials it became apparent that the novelty of the approach and the need for familiarisation with the system would have rendered such an analysis invalid. Additionally as requested by a number of the participants, a dictionary was created using the individual signs for various computing concepts. It had been hoped that this this could have been used in the light of the feedback whether we had successfully captured signs which could be understood regional variations not withstanding, unfortunately however this was not possible in the timescales.

Figure 3: Screenshot from assessment engine

Feedback and Focus Groups

An initial focus group session was held with SQA stakeholders invitees included internal SQA staff, teachers of the Deaf and representatives of the Deaf community. A short presentation was given on the overall project and a demonstration of the software in action. An exercise was then held asking the participants to write their thoughts on post-it notes under defined headings. Following this a semi-structured discussion was lead by the researcher from RNID based on what had emerged from the post-it note exercise. Following the initial stakeholder focus group, an additional three learner feedback sessions were held. We aimed to gain feedback as far as possible from the range of learners and learning situations that candidates may find themselves presented for assessment as well as having a representative sample from across the country, given the known issues with regional signing variation. To that end, we held one feedback session at Dingwall Academy in the Highlands a mainstream High Schools with a specialist Deaf support unit; one at Donaldsons College in Edinburgh a specialist school for Deaf children and one at Deaf Connections in Glasgow a Deaf community resource, which among other activities provides education and training for Deaf adults. The feedback sessions were in three parts. First of all, participants were asked try out the software. Initially we had hoped to gain some comparative data for analysis by getting the candidates to try some of the questions with and some without the avatar, however the novelty of both the situation and the software, as well as technical difficulties experienced meant that this could not be considered a realistic test situation for analysis purposes. Once they had completed the test, candidates were asked to complete a short questionnaire (Appendix 1) in English, which asked about a variety of aspects of the software. A semistructured focus group was then held through the medium of BSL, led by the researcher from RNID. In total 19 people, including two tutors (one at Donaldsons, one at Deaf Connections)

participated in the focus groups. All of the participants were Deaf. Key demographics of the participants were captured from the questionairres. Demographics of Feedback session participants Gender female male Age 14-18 19 - 35 19-35 36-50 51+ Computer experience beginner? Intermediate int/adv advanced unknown Native Language BSL BSL/English DSL1 English SSE

9 10

11 1 1 1 5

3 11 2 1 2

Research Findings
Attitudes towards Avatars in Examinations
Probably influenced by the poor quality of signing as is noted below, one key concern was the accuracy of the BSL and the confidence that the candidates would have in relying on them in an examination context. Participants mentioned that with human communication support workers you were able to see clearer hand shapes multi-channel signals2 and non-manual features, some also mentioned the relationship that they had with the signer who would be able to adapt the signing to ensure that it was understood by the candidate and that boosted your confidence while in the exam. It was suggested that a communications support worker should be on-hand when avatar signing is being used as a backup. Several said that they would be keen to see the avatar again once the quality of the signing had improved. Advantages noted with the use of the avatar in exams included the equality of access of BSL signers who may currently have variable quality of communication support workers, that it would give candidates more control over what is signed, and also that it would be less intrusive, allowing Deaf students to access the examinations directly rather than through a third party. The participants at Donaldsons School, where we experienced significant problems with the technology, were understandably concerned about the robustness of the software, and feared that it may crash; however students at other schools highlighted the reliability of the avatar that you can be sure that it will turn up on the day and you can ask it to repeat things as many times as you would like without annoying it as advantages of avatar based signing For these trials, we had used the avatar to sign only the questions, not the answers as well on the basis that this would not be necessary as it was the grammar of English, rather than the words themselves which candidates tended to struggle with. Several focus group participants however pointed out that BSL is a language choice, and therefore it made no sense to sign the questions but not the answers, especially if candidates had chosen the avatar over a communications support worker and there was no one there to ask if they did not understand the English answer. Training in the use of avatars prior to the examination was also raised by a number of people who highlighted the need to be familiar with how the avatar is signing and how to control it. One suggested that there should be practice work using the avatar prior to the examination, while another suggested that a dictionary of BSL signs available in the exam may overcome the difficulties with regional variations. There was also a general feeling that the timing of exams also should be carefully considered as signing may take longer than reading. Outcomes from the questionnaires are not terribly encouraging, with less candidates reporting confidence in taking the test with avatar support than with no support at all, and also that it was the enjoyable form of test taking. One interesting outcome however is that candidates reported that they enjoyed taking the test less with human or with no support, even though
1 2

Danish Sign Language Emotionally loaded signs that have no direct English equivalent.

they had more confidence in the medium, whereas they reported more enjoyment in test taking with an avatar than they had confidence suggesting perhaps that the poor quality of signing noted below was a major factor in rejection of an avatar signed solution. Once the difficulties with the clarity of signing are addressed, confidence in the solution may well grow. Test confidence with avatar support 1 9 2 2 3 4 4 1 5 3 Average 2.32 Test confidence with human support 1 0 2 2 3 3 4 4 5 10 Average 4.16 Test confidence with no support 1 3 2 4 3 5 4 2 5 5 Average 3.11

Test enjoyment with avatar support 1 6 2 2 3 3 4 2 5 2 no answer3 4 Average 2.47

Test enjoyment with human support 1 0 2 1 3 3 4 5 5 5 no answer 5 Average 3.73

Test enjoyment with no support. 1 2 2 3 3 6 4 3 5 1 no answer 4 Average 2.87

Clarity of Signing
In general there was a feeling that the quality of signing was poor. Several participants commented that they were unable to follow the signs without using the text, and it was noted that it was a struggle to follow rather than clear and relaxing. This is clearly a major issue which needs attention. Of particular concern was the non-manual features and the lip speaking. The poor lip pattern in particular was something which was raised in every group, although there were differences of opinion about whether the lip pattern was actually incorrect or whether it was just so unclear as to render it meaningless. Several participants also commented on the lack of facial expressions, which may have contributed to the impressions given above, that the avatar appeared cold. The signing space as mentioned above, was criticised by several participants who felt that this added to the difficulty in understanding. It was also suggested that the avatar needed to be able to point more to make its meaning clearer The fluency of the signing was also criticised being described as stilted and robotic and that this contributed to the difficulty in understanding the signing. Several said that this had given them difficulty understanding the signs, as it did not appear to be language as much as individual signs strung together, although a few participants suggested that this simply took some adjusting until they were able to tune in to the avatar Another pertinent issue, which was raised in every group, was regional variations. The interpreter who had initially translated the questions was a Scottish signer living in London, which was perhaps on reflection not a good choice. Several participants queried whether the signing was English or Scottish. The regional variation within Scotland was also seen as a significant challenge, which would make it hard for people to follow the avatar as they were not using the local signs that people were used to. This is however contrasted with those, particularly teachers, who welcomed the idea of standardised signing for schools, although
3

At Donaldsons, difficulties with the technology meant that we werent able to allow the candidates to trial, but instead we demonstrated it on a large screen

they did point out that SQA must work with schools to ensure that the same signs are used consistently. The BSL vs Sign Supported English (SSE) issue was a particular concern among teachers who felt that the native BSL used by the avatar would not be suitable for some of their children who use SSE, and pointed out that most tended to fall somewhere within the BSL/SSE continuum and thus would not be suitable for all students, however this was not a concern raised by students themselves who appeared to be happy with the native BSL. The results from the questionnaires also suggest that there is a particular problem with the lip movements and the non-manual features, although people were happier with the handshapes. The overall signing was considered to have higher clarity than any of the particular features on its own, suggesting perhaps that candidates could contextualise and assimilate to obtain a higher level of clarity. Was the overall signing clear? 1 2 3 4 5 Average Were the lip movements clear 1 2 3 4 5 Average 1.68 Were the non-manual features clear? 1 6 2 6 3 5 4 1 5 1 Average 1.74 Were the hand shapes clear? 1 2 2 8 3 5 4 2 5 2 Average 2.21

1 4 10 2 2 3.00

10 5 3 1 0

Layout, Appearance and Control of the Avatar

Most felt that the screen was well laid out, with appropriate proportions given to the avatar and the question, although it was noted that it was not clear how the avatar could be positioned, and that attention should be drawn to this functionality. There were also several participants who noted that it would be advantageous if the avatar could keep the same position when moving onto the next question, rather than moving back to the default. The potential to modify the screen, in particular the font and background colour for people with visual impairments was also raised. With the exception of one participant, everyone liked the appearance of the avatar, although there was a feeling among some participants that the signing may be easier to follow if the avatar was wearing lighter coloured clothes. A few participants also suggested that it looked too dim - one participant suggested this made the finger spelling particularly difficult to follow. Most regarded the avatar as being of an appropriate size, however among adult learners there was a desire for the avatar to be enlarged. The positioning of the hands appeared generally acceptable, although there were a few comments suggesting that the distance from the hands to the face was too far; that the arms were too long or that the hands were too long making the signing space too large to see all of the movements comfortably there was also a suggestion that a box which would allow you to see the hand movements close up would be useful. One participant also felt that the fingers were too long, while another felt that the hands were very flat. The lips also came in for some criticism - one participant suggested it looked like the avatar was wearing braces as the teeth were so close together and there was no tongue visible, while another thought that the lips were overly tight, which had contributed to the poor lip speaking. The impression that the avatar made in general wasnt so favourable, it was compared to a puppet by one participant who felt that its movements were too stiff, while another described it as cold. Although another participant acknowledged that this was to some extent a cultural issue, and that it took adjusting to.

The questionnaire responses suggest favourable reactions to the avatar appearance and control, and the overwhelming majority of respondents indicated that the proportionate size of the avatar was just right. Was the avatar big enough 2 7 2 8 Too big Just right Too small 1 16 2

Was the layout easy to use? 2 1 3 6 4 3 5 9 Average 4.05

Was the avatar easy to control and move? 2 3 4 5 Average

3.85

Participants who tried adjusting the angle and positioning of the avatar valued that option, however several said that once they had positioned it in a manner that suited them, it was annoying to have to re-position it again for the next question. People also commented that it wasnt obvious that the avatar positioning and angle could be adjusted. One participant also mentioned that it was inappropriate to allow the avatar to be turned a full 180 as that breaks BSL rules and is considered rude. Several participants mentioned that they would like more control over the avatar; with options to choose gender, ethnicity and background colour, however another pointed out that in the context of an examination that would waste time and be a distraction. Perhaps consideration could be given to a level of personalisation which could be provided in advance to ensure that candidates had an avatar that they felt comfortable with. Control over the speed of the signing was raised several times, and there was a feeling that the pace of signing used was somewhat slow, and the desire to skip parts particularly where there is long text was also raised.

Conclusions
Prior to operational use, it is clear that substantial technical improvements need to be made to the software deployed. Lip movements are a particular source of concern and although work has now been started to improve lip movements through matching lip pattens to speech phenomes, this work is still at an early stage and there was not the capacity or time within this project to fully explore how successful this approach would be. Not withstanding the need for technical improvements, given that the participants interactions with the avatar were fairly limited (under 30 minutes), further research is also required to identify how significant an issue the identified lack of fluency in signing is, and how much is the need to tune in to the technology. Further consideration must also be given to regional variation and to whether the technology should support either BSL or SSE. There was a sharp difference here between teachers of the Deaf, who considered such provision crucial and Deaf people (both adults and children) who did not raise it as an issue. The concerns of the teachers appeared to be twofold firstly that some lower ability children would not understand pure BSL, and also that for more able children allowing them to have an avatar signing BSL for learning and/or assessment may not support their English literacy. Teachers of the Deaf generally welcomed the initiative as a potentially inclusive technology and in particular saw the advantages that it could bring, particularly if standardised vocabularies and learning support materials provided, minimising regional variations in signing and providing a definitive version of BSL. This is not an issue which was particularly raised by deaf participants (despite difficulties experienced at times in communication between the London based researcher and the Scottish participants). There is precedent for

the SQA leading a standardisation of minority language, as it has done in Gaelic to overcome the difficulties in regional spelling variation through the SQA Gaelic Orthographic Conventions (2005) has. In 1988 when the convention was first made compulsory such imposition was a controversial move (Leland, 1991) within the Gaelic community, however the conventions are now accepted. The acceptability of an avatar signed solution within the Deaf community is another critical issue. At the start of the project there was some unexpected hostility at this research which became more muted over time once the early stages of this work became apparent. How much this reflects the poor quality of the signing, and a perception that the research is being driven by financial motives and how much is a genuine dislike for avatars in education is unclear. It is not the intention that an an avatar would ever replace a communications support worker in an examination, should that be the choice of the candidate, although perhaps that was not made clear enough in the introductions; this technology would be used only for those who requested this technology in addition to, or as a replacement for a CSV. We are hoping to explore funding streams to develop further work in this area to scope the potential for operational use within a limited area in association with examination teams. Interest has also been raised within SQA about the feasibility of using avatars to provide support for customers in all three languages of Scotland, minimising the need for special arrangements for BSL users, or Gaelic speakers to access our website.

References
Disability Discrimination Act (2005) London HMSO ESRC(2005) Research Ethics Framework Economic and Social Research Council, London. JCQ (2005) The use of BSL and other sign languages and definitions of a communicator or interpreter, Joint Council for Qualifications, February 2005. Scottish Executive (2005) Investigation of Access to Public Services in Scotland using BSL Scottish Executive (2003) Partnership for a Better Scotland Leyland L (1991) Critique of the "New" Gaelic Orthographic Conventions, Stornaway Gazette, August. Powers, S., Gregory, S., Thoutenhoofd, E. D. (1998) The Educational Achievements of Deaf Children. DfEE Research Report 65. London: DfEE. SQA (2005) Gaelic Orthographic Conventions Scottish Qualifications Authority. Wills, G., Davis, H., Chennupati, S., Gilbert, L., Howard, Y., Jam, E. R., Jeyes, S., Millard, D., Sherratt, R. and Willingham, G.2006) R2Q2: Rendering and Reponses Processing for QTIv2 Question Types Proceedings of 10th International CAA Conference, Loughborough University, UK

SQA Avatar Signing Pilot

About you
Name First Language Age Gender Computer experience? ____________________________________ BSL 14-18 Male Beginner English 19-35 Female Intermediate Advanced 36-50 51+

About the avatar signing

Was the signing clear?
5
very clear

1
not clear

Were the hand shapes clear?

5
very clear

1
not clear

Were the lip patterns clear?

5
very clear

1
not clear

About the layout

Was the layout easy for you to use?
5
very easy

1
not easy

Was the avatar easy to control and move?

5
very easy

1
not easy

Was the avatar big enough?

too big about right too small

Compare the three ways of doing the exam

How confident were you that you understood the questions?
With avatar support 5
very confident

1
not confident

With human support 5

very confident

1
not confident

With no support 5
very confident

1
not confident

How much did you like taking the test?

With avatar support 5
liked very much

1
did not like

With human support 5

liked very much

1
did not like

With no support 5
liked very much

1
did not like

If you were given the choice which option would you prefer?
In a classroom to test yourself? Avatar support Human support No support

In a practice assessment (like a prelim)? Avatar support In an examination? Avatar support Human support No support Human support No support