Professional Documents
Culture Documents
Standards-Referenced Assessment For Vocational Edu
Standards-Referenced Assessment For Vocational Edu
net/publication/228427015
CITATIONS READS
21 994
3 authors, including:
Some of the authors of this publication are also working on these related projects:
The design and validation of assessment, reporting, and planning materials to support foundational skill development for children and young
people with additional learning needs View project
All content following this page was uploaded by Patrick Griffin on 05 June 2014.
T
his study examined a model of assessment that could be applied nationally
for Year Twelve Vocational Education and Training (VET) subjects and
which could yield both a differentiating score and recognition of com-
petence. More than fifty colleges across all states and territories of Australia field-
tested the approach over one school year. Results showed that the model allowed
for a standards-referenced model to be used: that the approach was compatible
with the diverse range of senior secondary assessment systems in use throughout
Australia and that there were considerable cost benefits to be had in adopting the
logic of item response modelling for the development of rubrics for scoring per-
formances on units of competence from National Training Packages. A change in the
logic of competency assessment was proposed, in that the performance indicators
were not rated using a dichotomy but with a series of quality ordered criteria to
indicate how well students performed specified tasks in the workplace or its
simulation. The study validated the method of assessment development, demon-
strated the method’s consistency, and showed how the method could address the
issue of consistency across states. The study also proposed a set of principles for a
joint assessment of both quality and competence.
Assumptions
This does not mean that it is necessary to jettison the idea of competence as a
dichotomy or the practice of confirming competence on the basis of performance
on discrete tasks as long as the quality of the performance can be taken into
account. There is, however, a need for the tasks to be a coherent set ordered in
terms of skill demand if they are to define a continuum of competence. It also
requires a set of assumptions to be explicated.They also apply to the dichotomy of
competence but are almost never considered (Griffin, 1997).
The study
The study addressed the issue of differentiation among students on the basis of
relative quality of performance and was conducted in four industries using a total
of fifty-six competency units: seventeen units in Metal and Engineering, fifteen
units in Information Technology, twelve units in Business Administration and four-
teen competency units in Hospitality. Sixty schools nationally participated in the
project on the advice of each of the state vocational education jurisdictions.There
were nine schools in the Australian Capital Territory, twelve in New South Wales,
five in Queensland, six in South Australia, thirteen in Victoria, five in Tasmania and
ten in Western Australia. The schools focused on VET subjects and units of com-
petency distributed over the four industries.A total of 3,672 students were assessed,
493 in Metals, 1,215 in Hospitality, 884 in Business and 1,080 in Information
Technology. The students were assessed against each of a series of units of com-
petence in their relevant VET study area. The assessments were conducted by
teachers who were provided with a rating sheet on which they recorded their
observations of the relative quality of the student performance in the tasks associ-
ated with each of the units of competency. This was then developed into a score
for the subject.The derivation of the score is described below.
Criterion 1.1.1.1
Indicator 1.1.1
Capability 1.1 Criterion 1.1.1.2
Domain 1
Indicator 1.1.2 Criterion 1.1.1.3
Capability 1.2
Indicator 1.1.3
Capability 1.3
Indicator 1.1.n
Framework
Capability 1.n
Domain 2
Domain 3
how well each task was completed, the issue of ‘how well’ could be addressed.
The descriptions of ‘how well’ were called quality criteria. Some performance
indicators might have two identifiable quality criteria, some three, some four, but
in all cases a set of procedures for defining the criteria would have to be followed.
When a set of quality criteria were combined with the performance indicators as
rating scales the composite is called a rubric.
The panellists were shown how to write rubrics according to the rules that
quality criteria should: reflect levels of [workplace] performance quality not pro-
cedural steps; avoid counts of things right or wrong; avoid all comparative terms;
discriminate among ordered quality performances; enable assessees to verify their
performance assessment; focus on a series of single performances within the same
domain; and yield reliable and consistent judgments (Griffin, 1997).
The number of quality criteria for each rubric had to be defined by the
panellists, who could each call on expertise outside the sphere of their own
experience in order to avoid the restrictions of self referencing.
Score development
The model used for the study consisted of several steps.The first step was defining
the framework or the broad area of study, as shown on the left in Figure 1. In this
case it represented the training package itself and represented the overall VET study
area within which the assessment was conducted.
The second level of the model described the key performance areas, or skill
sets required within the framework. In the training package these were called the
‘units of competence’.
Criterion 1.1.1.1 1
Establish rapport
when working with
others from a range
of social, cultural
and ethnic
backgrounds?
Criterion 1.1.1.2 2
Area 1: Indicator 1.1.1 Apply a range of
Work with Capability 1.1 Recognise and cummunication
colleagues and Work in a team accommodate strategies when
customers cultural dealing with team
differences members from
within the team diverse
backgrounds?
Criterion 1.1.1.3 3
Use the diverse
background
composition of the
Hospitality team to assist
colleagues achieve
work group goals
Criterion 1.1.2.1
Criterion 1.1.2.3
Criterion 1.1.3.1
Criterion 1.1.3.3
Area 2
Area 3
and quality criteria would score highest, and the relative importance of a unit could
be enhanced simply by defining a larger number of quality criteria.This approach,
in addition to being flawed, would generally lead to an equally incorrect practice
of insisting that all criteria had the same number of levels or score points and it
would require an irrelevant method of weighting process to establish the relative
importance of the rubrics. If there were more levels of quality, the importance or
influence of the criterion would be increased. Applying the logic of item response
modelling was an important procedure in making this assessment approach acces-
sible. It also provided a way of defining the underlying continuum in terms of a
Works as a Demonstrates limited Communicates and Deals with difficult Displays cultural Anticipates, monitors
member of ability to work with interacts with others situations in a sensitivity and high and resolves difficult
team colleagues and customers in a positive and positive and quality sevice situations when dealing
supportive manner sensitive manner with others
0 1 2 3 4
Work in a Requires support Uses various Displays cultural Avoids, and when
socially to work in a communication strategies awareness and required,
diverse socially diverse when dealing with sensitivity resolves cultural
evnironment environment diverse groups misunderstandings
0 1 2 3
Follow Requires support Identifies and reports Follows correct health, Contributes to the
occupational to follow occupational OHS issues safety and security management of workplace
health, safety health, safety requiring attention procedures and can deal health, safety
and security and security procedures with emergency situations and security
procedures 0 1 2 3
Follow Demonstrates ability Follows workplace Applies corrective action Understands hygiene
hygiene to follow basic standards for handling to minimise or remove regulation and its impact
procedures hygiene procedures and storage of foods hygiene risks on the industry
0 1 2 3
Develop and Demonstrates limited Accesses specific Maintain knowledge of the Demonstrates awareness
update ability to develop and information on industry, including and understanding of a
hospitality update hospitality relevant sector of work legal, ethical, and current range of industry related
industry industry knowledge when required issues of local concern issues including current
knowledge and emerging
0 1 2 3
Organise and Demonstrates ability to Follows correct equipment Prepares a range of food Uses logical and time- Improvises ingredients
prepare food assemble equipment and safety procedures and quickly and accurately efficient workflow in when required, applies
prepare some ingredients assembles ingredients with consideration to preparation of food and cutting and shaping
for menu items quality, hygiene, suitability displays a range of knife techniques appropriate to
consistency and wastage handling, cutting and the style of cuisine, and
shaping techniques cleans and prepares
seafood with consideration
to hygiene and OHS
0 1 2 3 4
Present food Demonstrates ability to Portions and presents Presents food using
select appropriate food according to classical and innovate
garnishes/sauces when standard recipes and styles with consideration
presenting food instructions, OHS and to colour, contrast and
hygiene regulations and temperature
food presentation
0 1 2
Process Requires support to Receives cash payments, Follows enterprise Conducts timely Processes a range of
financial process financial issues correct change procedures when transactions, counts cash, non cash transactions in
transactions transactions and records transactions processing automated calculates non cash accordance with enterprise
in timely manner receipts and cash documents and maintains and financial
payments, and removing accurate records institutions
and recording takings
from register/terminal
0 1 2 3 4
Promote Requries support to Supplies acurate and Actively researches and Evaluates products, Applies conflict resolution
products and promote products and readily available maintains product/service services and promotional strategies
services to services to customers information to customers knowledge initiatives. Successfully
customers on products employs upselling and
and services cross selling techniques
0 1 2 3 4
Prepare and Demonstrates limited Prepares and services a Drinks can be customised Prepares ingredients for an
serve non ability to prepare and range of beverages in to meet specific requests extensive range of hot and
alcoholic serve non alcoholic drinks a logical, efficient and and quality control is cold beverages and
drinks presentable manner maintained during busy maintains and monitors
periods equipment usage and
functionality
0 1 2 3
Receive and Follows standard Prioritise storage Identifies, records and Manages stock to
store kitchen procedures for inspecting requirements of various reports incoming stock ensure timely use
supplies storing and recording foods and mantain stock variations and and replacement
incoming stock with consideration to discrepancies and of goods
usage, safety and disposes damaged or
hygiene expired stock in
accordance with OHS
and industry regulations
0 1 2 3
School
Based 14 0.84 9.35 8.10 44 4 9 22 33 40 44
and it helped to illustrate that the procedure could be carried out and that statis-
tical moderation of the school-based assessment is possible with item response
modelling analysis.
From Table 2 it can be seen that there were variable numbers of performance
indicators for each unit. Ni, the number of performance indicators within a com-
petency unit, ranged from twenty-six indicators (unit THHCOR01B) to six (unit
THHGHS01B). Measures of reliability were obtained using the scores assigned to
indicators within units and a score from which levels of competence could be iden-
tified. Reliability estimates ranged from 0.95 to 0.83. Maximum scores for each of
the units are also shown and these were obtained by summing the maximum
quality criteria ratings over all indicators within a unit. Panellists defined the cut
scores for the change of competence level.
Consistency
Classical measures of consistency include the indices of reliability known as
Cronbach’s alpha. This index ranges in value from 0.0 to 1.0. Zero indicates that
there is no consistency and that the assessment is entirely random. More recently,
item response theories have added indices of ‘separation’ reliability and they pro-
vide some interesting interpretations. An item separation reliability index is closely
related to the Cronbach index but indicates the extent to which the items or, in
this case, the rubrics or quality criteria were separated along the underpinning con-
struct or continuum. A value of zero indicates that the rubrics are all clustered at
the same point and each indicates the same level of performance. Adding more
items or rubrics at the same level does not add value to the assessment.As the items
are increasingly separated, the value of the index rises to 1.0. At 1.0 the rubrics or
3
Expected Score
0
-3 -2 -1 0 1 2 3 4
Person Location (logits)
how many units are used. Figure 4 relates student ability or competence to the
standards-referenced band assigned for the each unit. The close alignment of the
lines shows that there is little difference in which or how many units are used to
assess overall competence.
Figure 5, however, shows that the assessment can differ considerably depend-
ing on the location of the assessment. In the Australian Capital Territory, for
example, a student had to demonstrate a high level of performance quality in order
to be assessed as competent (represented by the horizontal line at a value of 1.0.)
In most other states, a much lower level of performance quality was required in
order for the student to be assessed as competent. In this study, the difference due
to location was controlled through the application of item response modelling
analysis but in most assessments, where the weighted differentiating score is not
used and a decision of competence is made, this difference is uncontrolled and the
consistency of decision-making is variable. This underpins the point that there is
no fixed level of competence. Not only is the level of performance expected in the
workplace variable, but the competence decision varies according to the demands
of the curriculum in the school system. The important thing is whether the
student can meet the expectations of the workplace or the school system. This
analysis has exposed the variability in expectations demanded for attributing
competence.
Figures 4 and 5 show that there were different views of competence across
systems and action needs to be taken to address this lack of consistency.The lack of
consistency was not an artefact of the current procedure demonstrated in this
article; it was a hidden aspect of competency assessment, which relied on judgment
3
Expected Score
0
-3 -2 -1 0 1 2 3 4
Person location (logits)
in context.We expect the effect would be exacerbated across workplaces and across
assessors. The lack of consistency is not a weakness of the proposed assessment of
competence and quality; on the contrary, the identification of this inconsistency is
a strength of this analysis.
Previous investigations of consistency of competency assessment have not
focused on outcomes or on the performances of the assessees (Victorian Education
Training and Assessment Services, 2000). They tend to have examined the proce-
dures and materials. Even with constant process and materials, differences exist in
the interpretation of competence. If a consistent national process were to be used
it would be possible for national standards to be set and monitored in competency
assessment and to control effects of location, localised curriculum effects and
judgment inconsistency. Hence, the method reported in this article not only pro-
vides an opportunity for scored assessment, it adds the notion of quality to com-
petence and allows monitoring of standards and the identification of any bias in
competency assessment.
Discussion
In this study the capacity of the student or the level of performance has been con-
trolled for the effect of the location.Without the weighted differentiating score and
item response modelling calibration, this effect would have to be controlled
through moderation. Consistency of competence assessment is an issue that still
needs to be resolved. The methods displayed in this article have shown a possible
approach to resolving or at least identifying such an issue.
References
Bateman, A. (2003). A validation of multi source assessment of higher order competency assessment.
Unpublished masters thesis, Faculty of Education, The University of Melbourne,
Australia.
Connally, J. (2004). A multi source assessment of higher order competencies. Unpublished
doctoral thesis, Faculty of Education, The University of Melbourne, Australia.
Curwood, M. (2006). A case study of the implementation of competency based assessment and
training in Australian industry. Unpublished doctoral thesis, Faculty of Education, The
University of Melbourne, Australia.
Glaser, R. (1981). The future of testing: A research agenda for cognitive psychology and
psychopathology, American Psychologist, 36(9), 9–23.
Greaney, V., Khandker, S.R., & Alam, K. (1999). Bangladesh: Assessing basic skills. Dhaka:
University Press.
Griffin, P. (1995). Competency assessment: Avoiding the pitfalls of the past. Australian and
New Zealand Journal of Vocational Education, 3(2), 33–59.
Griffin, P. (1997, September 18). Developing assessment in schools and workplace. Paper pre-
sented at the Inaugural Professorial Lecture, Dean’s Lecture Series, Faculty of
Education, The University of Melbourne.
Griffin, P., & Gillis, S. (2001, May 4). Competence and quality: Can we assess both? Paper pre-
sented at the Upgrading Assessment: A National Conference on Graded Assessment,
Kangan Batman Institute of TAFE, Melbourne.
Griffin, P., & Gillis, S. (2002). Scored assessment for Year 12. (Report of the pilot study).
Melbourne: The University of Melbourne, Assessment Research Centre.
Griffin, P., Gillis, S., & Calvitto, L. (2004). Connecting competence and quality: Scored assess-
ment in Year 12 VET. Melbourne: The University of Melbourne, Assessment
Research Centre.
Griffin, P., Gillis, S., Keating J., & Fennessy, D. (2001). Assessment and reporting of VET
courses within senior secondary certificates. In Creating expanded opportunity for youth:
Greater recognition for VET courses in industry and university. Sydney: New South Wales
Department Vocational of Education and Training
Masters, G.N. (2002). Fair and meaningful measures? A review of examination procedures in the
NSW Higher School Certificate. Melbourne: Australian Council for Educational
Research.
McCurry, D., (2003). But will it work in theory? Theory, empiricism, pragmatics and the key com-
petencies: the place of theory and research in the development of a notion of a work related skills
and the whole school assessment of generic skills. Melbourne: Australian Council for
Educational Research.
McGaw, B. (1997). Shaping their future: Recommendations for reform of the Higher School
Certificate. Sydney: NSW Department of Training and Education Coordination.
Messick, S. (1992). The interplay of evidence and consequences in the validation of performance
assessments: Research report. Paper presented to the Annual Meeting of the National
Council on Measurement in Education, San Francisco, CL, USA.
National Assessors and Workplace Trainers Body. (1998). Training package for assessment and
workplace training. Melbourne: Australian National Training Authority.
Authors
Professor Patrick Griffin is Director of the Assessment Research Centre, Chair of Edu-
cation (Assessment), Deputy Dean, and Associate Dean of Innovation and Development at
the University of Melbourne.
Email p.griffin@unimelb.edu.au
Dr Shelley Gillis is a lecturer and research fellow at the Assessment Research Centre.
Leanne Calvitto was a research associate at the Assessment Research Centre at the time of
this study.