Assessor 3

Assessor 3
Standardisation Setup User Guide

This guide assumes you already have an understanding of how to mark in

RM Assessor3 – refer to the Examiner User Guide if necessary.

V2.1
For use from January 2022 onwards
 Assessor 3 Standardisation Setup User Guide V2.1

© RM Education Ltd. 2021

rmresults.co.uk 2
 Assessor 3 Standardisation Setup User Guide V2.1

Contents
1. Overview of Standardisation Setup    4

2. Responses available notification    4

3. Accessing the Standardisation Setup area    5

4. Choosing provisional responses    7

5. Marking provisional responses    11

6. Definitively marking responses    15

7. Re-using responses     19

8. Classifying responses     21

9. Completing Standardisation Setup    26

10. Standardisation related reports    28

11. Logout and Settings    30

12. Support     31

rmresults.co.uk 3
 Assessor 3 Standardisation Setup User Guide V2.1

Overview of Standardisation Setup

As a member of the Standardisation Setup team you will be involved in the process to create definitively marked
practice responses, 1st qualification responses, 2nd qualification responses, and seeding responses.

All members of the team are able to access the Standardisation Setup area, to view candidate responses that are
available to be marked, and select responses to provisionally mark. Once provisionally marked responses can then
be shared with the rest of the team.
The shared provisionally marked responses will be discussed during the standardisation meeting, and the Principal
Examiner will select those that will be used, set the definitive marks, and then classify them as practice, qualification
and seeding responses. The rest of the team members can view the definitive marks of unclassified responses, and
once they have been classified they can continue to view all responses except those chosen to be seeds.
Once the Standardisation Setup process is complete, Examiners in the QIG are moved onto their first marking target
(usually practice marking).

Responses available notification

If you are the Principal Examiner for one or more
QIGs, you can choose to receive an automatic email
to advise you when responses are first uploaded to
RM Assessor for the components in which you have
this role. This email alerts you to when responses are
available, you will then know that the Standardisation
Setup process can commence as agreed with your
Subject Manager.
To receive this notification:

1 Enter your email address in the top portion of

the Settings panel.

2 Turn on the Notify me when responses are

available option in your RM Assessor settings.
This option only displays if you have the
Principal Examiner role in at least one QIG
you are assigned to mark, and for at least
one of these QIGs no responses are as yet
uploaded into RM Assessor.

Note that if you undertake these steps after the

first responses are uploaded for a component,
you will not receive an email alert for that
component.

rmresults.co.uk 4
 Assessor 3 Standardisation Setup User Guide V2.1

Accessing the Standardisation Setup area

Navigating to Standardisation Setup from the Home page
You will be presented with the Home page displaying your QIG list when you login for the first time or if you were
on this Home page when you last logged out. All of the QIGs you are assigned to mark and manage are listed on
this Home page.

1 2 3 4

1 Where no responses are uploaded into RM
Assessor for a component, an Awaiting
Responses status will show against the QIG(s) in
the component. At this time you cannot access
the Standardisation Setup area for that QIG.
2 For QIGs where previous session responses are
available to select for re-use, whilst awaiting
responses from the current session an Access
previous session responses link will show for
the Principal Examiner of the QIG. Clicking
this will navigate to the standardisation setup
area to allow for working with previous session
responses (see the Re-using Responses section
for further details).
3 Any QIGs where you are a Standardisation
Team Member, there are candidate responses
uploaded, and Standardisation Setup is not yet
complete, will have this button. Click on it to
open the Standardisation Setup area.
The button will have an indicator on it as
follows:
You have not yet selected any responses for
provisional marking
You have at least one response in your
Provisional worklist
4 Any QIGs where you are a Standardisation Team
Member, and Standardisation Setup is complete,
will have a link option beneath the marking
progress indicator – clicking the link will open
the Standardisation Setup area.

rmresults.co.uk 5
 Assessor 3 Standardisation Setup User Guide V2.1

Navigating to Standardisation Setup via the Menu option:

The top right Menu option is visible at all times, and
allows you to quickly navigate between the three most
recent QIGs you have worked on during your current
login session, as well as to access the Home page,
Reports, your Messages, and providing the logout option. If you are a Standardisation Team Member for any of the
three most recent QIGs you accessed, then an option will show beneath the QIG name to access that area.

Note that this is with the exception of if you accessed Standardisation Setup prior to live responses being
available from the current session (in order to work with previous session responses available for re-use). In this
scenario the Standardisation Setup option will not show – you can only access the area from the Home page
link.

Hints and Tips

If you were in the Standardisation Setup area of a QIG when you logged out of RM Assessor, you will be returned to
that area on next login.

rmresults.co.uk 6
 Assessor 3 Standardisation Setup User Guide V2.1

Choosing provisional responses

The first step in the Standardisation Setup process is the preparation of marked provisional responses which can then
be discussed in the Standardisation meeting.
The Select responses worklist allows you to view all currently available responses by centre. Browse through these
to find suitable responses for use in the standardisation process, and then choose to provisionally mark them. Once
you have chosen a response to provisionally mark, the rest of the Standardisation team can only view it, they cannot
also choose to provisionally mark it.
After provisionally marking a response you can choose to share it with the whole Standardisation team.

Notes:
• If configured for the component, you may also have permission to choose to re-use definitively
marked responses from the same component from previous sessions. See the section on Re-using
responses for more details.
• For On-screen examination QIGs, responses flagged as Specialist are not included in the count
of Responses for a centre in the Select Responses screen, nor are they displayed within the list of
responses for the centre available to select for Provisional marking.

Finding and viewing responses

On accessing the Standardisation Setup area, the Select Responses screen will be displayed by default.

1 Against each Centre you can see how many

responses are currently uploaded into RM 2 1
Assessor (this number may therefore increase as
further responses are uploaded), how many are
currently available to be selected for provisional
marking (i.e. excluding those already selected
for provisional marking by you and other
Standardisation Team Members), and the date
the first response for that Centre was uploaded
into RM Assessor. A Centre will only show in
this list once candidate responses have been
uploaded into the system.
You can choose to sort the Centre list by any of
the columns by clicking on the column header.
The Centre you have selected and the sort order
you have applied will persist when you navigate
away from the Select responses worklist and
back to it, within your current logged in session.
Select a Centre from the top panel, and
responses from that Centre will display in the
bottom panel.
2 Click on a Script ID link to open a response (it
will open read only). If a response cannot be
selected for provisional marking the Status will
show as Unavailable, but this does not stop you
from viewing it.
For structured components, if a response
contains additional pages then an icon will show
to the right of the status column.

rmresults.co.uk 7
 Assessor 3 Standardisation Setup User Guide V2.1

Having opened a response from the Select responses screen:

3 1 2 4

1 Click Select responses to close the open are left in Full Response View and need to click
response and return to the list of centres and the Response button to go back to the main
responses. response view.
2 You can use the previous and next responses
buttons to navigate through responses. Once
you get to the last response in the currently
selected centre, clicking next will take you to the
first response in the next centre.
3 Click to view the response in Full Response
View.
If you open a response that has additional pages
it will automatically open in Full Response View
4 Note if the response is not available for
with an alert that there are additional pages. You
provisional marking this button will not display.
will not have to manage these additional pages
at this stage, but on accepting the alert then you

Notes:
• At this stage it is not possible to raise an Exception, or send a message, in relation to a response.
• Where a response is unavailable as it has already been selected by you or another Standardisation
Team Member for provisional marking, no marks will display.
• If the QIG is configured to use the seed categorisation feature, the first time in a login session that you
open a response from the Select Responses worklist a prompt will appear to remind you to ensure
responses selected for standardisation setup cover a full range of candidate abilities. For further details
on the seed categorisation feature, see the Viewing seed categorisation information section.

rmresults.co.uk 8
 Assessor 3 Standardisation Setup User Guide V2.1

Select Responses & ebookmarking QIGs

For e-bookmarking QIGs, candidate responses which have unknown zones are not displayed in the list of responses
for a Centre. For each response that is displayed, an additional column shows which Question Items have been
identified in each response.

If required, you can search for responses with particular questions answered. The search is not case sensitive and
will search for any questions with the entered value present (e.g. if you searched for “a” it would find 1a, 2a, 3aii
etc..). You can enter multiple questions by using a comma as a delimiter (e.g. 1a,4b), and the system will search for
responses with at least one of those values (e.g. 1a OR 4b) – note, do not include any space after a comma. The
applied search persists until you clear it, even if you choose a different centre.

Creating a provisional response

1 In an open response click Select to mark.

2 Choose Mark now to provisionally mark the

response immediately, or select to Mark later.
In either case the response will be added to
your Provisional worklist ready for marking. If
you choose to mark later then you are taken
to the next response for the selected centre
(or back to the Select response worklist if it
was the last response available in the centre
that you just selected to mark).

rmresults.co.uk 9
 Assessor 3 Standardisation Setup User Guide V2.1

If the QIG is from a multi-QIG component, it is

possible for you to create a provisional response in
one or more QIGs (you must have a Standardisation
Team Member role in a QIG and the response must
not already be selected for provisional marking in that
QIG).
By default the currently open QIG will be selected.
Only QIGs where it is possible for you to provisionally
mark the response will be listed. Tick all required
before choosing to Mark now or Mark later.
Note that if multiple QIGs are selected, you still
provisionally mark each QIG separately – this
selection step simply creates the provisional response
in the worklist of the other QIGs selected, as well as
the currently open QIG.

rmresults.co.uk 10
 Assessor 3 Standardisation Setup User Guide V2.1

Marking provisional responses

Provisional marks worklist

1 2 3 4 5 6 7

1 The count shows how many responses you visible in the Standardisation Setup area only
currently have in the Provisional worklist (it – they are not part of the definitive marking
excludes any already shared). comments.
2 Click the Response ID to open a response. If you
have a small screen resolution this column is
frozen as you scroll to the right.
Each uploaded response has a Script ID that
identifies the response in the Select responses
screen. When you choose to mark a response 5 Click to apply a coloured Tag to the response if
as provisional it is assigned to you and given a desired (as a way of subdividing or highlighting
Response ID associated with your marks. Both responses). Tags are personal to you, and
ID numbers are shown in the worklist. cannot be seen by any other Standardisation
Team Member. When you place a tag during
By default the Provisional worklist is sorted by
provisional marking, once the response is shared
Response ID, in ascending order, but you can
the Tag will also be visible to you when you view
click on any column header to change the sort
it in the Unclassified or Classified worklist.
order to one of your choice.
6 The Modified date is only updated when the
3 The status field is blank if marking has not
marks and annotations in the response are
started for a response, a percentage value if
saved. If a response is opened and only notes
marking has started but is not complete, and
are updated then the modified date does not
shows a Share button if the response is 100%
change.
marked. For further information see the section
on Sharing your provisional response. 7 Click View mark by question to view the
Provisional worklist at a question mark level,
4 Any Note entered against a response when
rather than viewing the total marks.
provisional marking is visible from the worklist.
Click on the note to see the full text. Notes are

rmresults.co.uk 11
 Assessor 3 Standardisation Setup User Guide V2.1

Mark by Question view

The Marks by question view may be useful to check that you have selected responses over a good spread of
candidate abilities. When View marks by question is chosen, only the Response ID, Marks, Status and Note columns
show in the Provisional worklist, in addition to a column for every question item in the QIG. If there are more
questions than fit on your screen, you can scroll to the right to see them (the Response ID column is frozen and
always remains in view). If there is any optionality within the QIG, any question marks not included in the total mark
will show with a line struck through them.

Click View total marks to return to the previous view.

Note: The Unclassified and Classified worklists also have a Mark by Question view, in the same manner as
the Provisional worklist. Your current view setting in the Provisional worklist automatically applies to these
worklists, and vice versa. Your view setting will also apply across all QIGs you are involved in Standardisation
Setup for, and persists when you log out of RM Assessor and back in again.

Marking a provisional response

You mark a provisional response in the same way as a normal live response (this includes first managing any
additional pages in a structured response before you can mark it). The only differences are highlighted below.

1 If you create a message about the response 1 2 3 4

the recipient will automatically be the Principal
Examiner (and as such the message option is not
available to the Principal Examiner).
Note that whilst Standardisation Setup is in
progress if you create a message from your Inbox
(i.e. not related to a response) then it will also
default to be sent to the Principal Examiner, and
you cannot message others until Standardisation
Setup is complete (this is with the exception of
if you are the Principal Examiner, in which case
you can always send a message to anyone in the
marking team for the QIG.
2 Click to add a Note (see section below)
3 Click to discard the response if during
marking you decide it is not suitable for use in
Standardisation Setup. A pop up confirmation
message will appear. Discarding a response will
make it available for provisional marking selection
again from the Select responses worklist.
4 Click Provisional to close the response and
return to the Provisional Worklist.

rmresults.co.uk 12
 Assessor 3 Standardisation Setup User Guide V2.1

5 Once a response has been 100% marked a

Share button will display. Click this to share
the response with the whole Standardisation
Team. For further information see the section on
Sharing your provisional response.
By default on Sharing the response you will be
navigated to the next provisional response in
your worklist (if you have one). If however you
prefer to be navigated back to your Provisional
worklist, turn On the “Navigate to Provisional
Worklist on Submit” user option.

Note that it is not possible to raise an exception

when provisional marking.

Adding Notes to a provisional response

The Note functionality is a useful way to communicate your thoughts on a particular response to your Principal
Examiner and others in the Standardisation Team. Notes entered are visible in the Provisional worklist (and the
Unclassified worklist once the response is shared). Notes are only visible within the Standardisation Setup area.

1 Click the Note icon on the left hand menu bar

2 Add note text as required

3 Click outside the Note area to close the

note window and save the note you entered.
The Notes icon changes to show a note is
present. You can click the icon again to view
the note entered or to edit it if required.

rmresults.co.uk 13
 Assessor 3 Standardisation Setup User Guide V2.1

Sharing your provisional response

Once you’ve completed provisionally marking a response, choose to share it with your Principal Examiner and other
Standardisation Team Members. Once shared the response will disappear from your Provisional worklist and display
in the Unclassified worklist. You can view it in the Unclassified worklist, and still choose to amend the Notes in the
response.

Classifying a response whilst Sharing

If you are a Principal Examiner, on sharing a provisional response you will have the option to choose
to classify it immediately. If you choose to Classify the response it will move directly to the Classified
responses worklist. It will still be possible to re-classify it if required. This option is recommended to save
time if you are the only member of the Standardisation Team.

rmresults.co.uk 14
 Assessor 3
Definitively marking responses
All shared provisional responses display in the Unclassified worklist. All Standardisation Team Members can view this
worklist, but only the Principal Examiner has the ability to definitively mark and classify each response.
Reviewing shared provisional responses
1 2 3 4
1 Select to view the Unclassified worklist.
2 This summary panel shows the targets for how
many Practice, Qualification, 2nd Qualification
and Seed responses should be created and
progress towards this.
Practice, Qualification and 2nd Qualification
targets must be met (and cannot be exceeded)
before Standardisation Setup can be completed,
and the icon to the left of each item will change
once the target is met.
The number of Seeds to create is not
systematically mandatory however, but it is
strongly advised to meet the seeding target
during Standardisation Setup. More can be
created after the process completes.
rmresults.co.uk
Standardisation Setup User Guide V2.1
3 By default the worklist is sorted by Response ID
in ascending order, however you can change
the order by clicking on a column header (and
to reverse the order you click on a header a
second time). Your chosen sort order will persist
until you log out.
4 The status shows as either:
• Provisional - if the response has not yet
been definitively marked by the Principal
Examiner.
• A percentage value (if definitive marking has
started but is not complete).
• As 100% complete if fully definitively marked
and you are not the Principal Examiner OR
with a Classify button if the response is
100% definitively marked and you are the
Principal Examiner.
15
 Assessor 3 Standardisation Setup User Guide V2.1

5 6 7 8

5 The Last marker will either be the provisional
Examiner, or if definitive marking has started,
then the Principal Examiner.
6 Any Notes entered against the response are
visible from the worklist. Notes are visible in the
Standardisation Setup area only – they are not
part of the definitive marking comments.
7 Click and select a coloured tag to apply to
a response here if you wish (e.g. perhaps to
identify how you wish to classify it later on).
Note that tags are personal to you. When you
place a tag against an unclassified response, it
will also display once the response is Classified.
8 Click to view the Unclassified worklist at a
question mark level, rather than just viewing the
total marks.
See section on Mark by Question view for more
information.

Note that where the seed categorisation feature is configured for a QIG, a Category column will display at
the end of the worklist, showing the category associated with the total marks given to a response. Be aware
that the category values will always be in English, even when RM Assessor is set to display in French or
Spanish.

rmresults.co.uk 16
 Assessor 3
Viewing a shared provisional response
All Standardisation Team Members can open and view responses in the Unclassified worklist by clicking on a
Response ID.
2 3 6 4
1 Toggle to view or hide the annotations on the
response. You may want to hide them if they are
making it difficult to see the candidate response
underneath.
2 Click to view any existing Notes against the
response and edit these if required, or to enter
a Note if none exists. Notes are visible in the
Standardisation Setup area only – they are not
part of the definitive marking.
3 As Principal Examiner click to discard the
response if you decide it is not suitable for use
in Standardisation Setup. A pop up confirmation
Note when viewing a provisionally marked e-coursework response, all response files will automatically show
as read.
rmresults.co.uk
Standardisation Setup User Guide V2.1
7 5 1
message will appear. Discarding a response will
permanently delete any provisional or definitive
marks entered. The response will be available
for provisional marking selection again from the
Select responses worklist.
4 You can navigate between responses in the
Unclassified worklist by using the previous and
next response arrows.
5 As Principal Examiner this button will show.
6 Click Unclassified to close the response and
return to the Unclassified worklist.
7 If you don’t want this message to show every
time you open a shared response, you can turn
the hint display off in your Settings. See section
on Logout and Settings.
17
 Assessor 3 Standardisation Setup User Guide V2.1

Setting definitive marks

Only the Principal Examiner can choose to set
definitive marks and annotations.
To definitively mark a response, click Mark as
definitive and a pop up message will appear -
choose to copy the provisional marks as definitive,
or to clear the provisional marks and enter definitive
marks from an unmarked response. Whichever
method is chosen, you can always continue
to edit definitive marks up until the point that
Standardisation Setup is completed.

Note: If you choose to clear the marks and annotations and the original provisional marking was not done
by you, then it will still be possible to look back at the provisional marks while entering the definitive marks.
If you entered the provisional marks and choose to clear them when marking definitively, your original
provisional marks will be lost.

1 As the Principal Examiner, on opening a

definitively marked response that was not one 2 1 3
that you provisionally marked, you can choose
to view the provisional marks (and annotations
if required) as well as the definitive marks, by
clicking on the Examiner dropdown and ticking
the checkbox against Provisional marks. When
viewing both the provisional and definitive sets
of marks, any differences are highlighted.
2 On opening an unclassified response as the
Principal Examiner, if you did not provisionally
mark it then you can choose to send a message
to the provisional Examiner by clicking the
message icon. The message recipient will
default to the provisional Examiner and cannot
be changed.
3 Once a response is 100% definitively marked
then as Principal Examiner a Classify button will
appear at the bottom of the marks panel. At this
time you can:
• Choose to classify the response (see section
Classifying responses below).
• Click on the Unclassified link on the top bar
to return to the Unclassified worklist (your
edits to this response are saved).
• Click on the previous or next responses
buttons in the top bar to move to a
different response in the Classified worklist
(which will also save your changes to this
response).

Note you can return to a definitively marked response – classified or unclassified - and edit the marks and
annotations at any point up until Standardisation Setup is completed.

rmresults.co.uk 18
 Assessor 3 Standardisation Setup User Guide V2.1

Re-using responses
RM Assessor can be configured to allow the re-use of responses as quality responses between components and
sessions, where components have the same structure. Where configured, on Standardisation Setup completion RM
Assessor will add definitively marked responses into a Re-use list, where they can be viewed and chosen for use in
subsequent Sessions, and any seeds later created are also automatically added to the Re-use list. Additionally the
Principal Examiner can promote other responses into the Re-use List if required – this is covered in the Principal
Examiner and Team Leader guide.
It is possible to access the Re-use list even before responses from the current session are available. Choosing a
response from the Re-use list helps to reduce the time taken at the standardisation meeting as fewer new responses
may need to be provisionally and definitively marked.
The Principal Examiner has permission to choose responses for re-use and will see an additional Previous sessions tab
in the Select Responses screen, as illustrated below, through which they can view the responses available for re-use.

1 2 5 3 6 4

1 Click the Script ID to open up the response in
read-only mode. This Script ID column is frozen if
you scroll to the right to view additional columns
of information.
2 Responses not already re-used in the current
session have a Re-use button against them.
Click this to re-use the response. See Choosing
a response from a previous session for further
details.
3 If required, toggle to hide the response in this
worklist. You may choose to hide a response if it
is unsuitable for re-use. For example, it has had a
high number of mark changes. Note, a response
which has been hidden in this worklist will remain
hidden in future sessions.
4 Toggle to show or hide responses in the worklist
that have been flagged to be hidden. This setting
will be remembered even after logging out of RM
Assessor and back in.
5 The Mark for the response is the current mark for it
within the Re-use list.
If a response is picked for re-use, and the marks are
then updated, it is only on Standardisation Setup
completion that the revised marks are saved back to
the Re-use list and reflected in this column.
6 You may need to scroll to the right to see all the
columns in this worklist (the Script ID and Reuse
button columns will remain in place when scrolling).
7 The Total Times Re-used column shows how many
times a response has been re-used. If you hover
over the count the tooltip will show the classification
types the response was set as, how many times the
response has been classified as each type, and the
session it was last classified as that type in.

rmresults.co.uk 19
 Assessor 3 Standardisation Setup User Guide V2.1

Choosing a response from a previous session

You can choose to re-use a response in two ways:

• By clicking the Re-use button against a response
from the worklist
• By opening a response and clicking the Re-use
button at the bottom of the marks panel
Whichever method is chosen a pop-up message
appears. You can choose to move the response to
the Unclassified worklist to discuss along with your
team, or to immediately classify the response and
move it to the Classified worklist. In either case a
copy of the response is created, and allocated to a
dummy candidate and centre number.

Notes:
• You can only re-use a response once within a component and session.
• If you later decide you don’t wish to re-use a response, you can discard it prior to Standardisation
Setup completion. It will then show in the previous sessions tab as available to choose to re-use again.
The original marks will not be lost.
• If standardisation setup required targets are met using only re-used responses, the Complete Setup
button will not become available until there has been at least one response from the current session
loaded into RM Assessor. Additionally on choosing to complete setup, a warning message will be
shown, to ensure it was intended that all practice, qualification and 2nd qualification responses were
taken from previous sessions and none from the current session candidate responses.

rmresults.co.uk 20
 Assessor 3 Standardisation Setup User Guide V2.1

Classifying responses
Definitively marked responses need to be classified as Practice, Qualification, 2nd Qualification or Seeding within RM
Assessor. Only the Principal Examiner has permission to classify responses. The Classification Summary panel in the
bottom left of the Standardisation Setup area shows how many of each response type are required, and progress
towards meeting this target.
• Indicates that the target has been met
• Indicates that the target has been exceeded.
You may temporarily classify more than the
target number but it will not be possible to
complete Standardisation Setup whilst any of
these targets is exceeded.
• Indicates that the target has not yet
been met and must be completed in order to
complete Standardisation Setup.
• If a response type has no icon then the
target is not mandatory in order to complete
Standardisation Setup. However, it is strongly
advised to meet the seeding target during
Standardisation Setup.

To classify a 100% definitively marked response

1 Click the Classify button against the 2 From the Classify response pop up message,
response – either in the worklist, or in the select the required classification type and
open response. click OK.

If you chose to Classify from the open response, the response will then close, returning you to the Unclassified
worklist. The classified response will no longer be visible in this worklist, and will show in the Classified worklist.

rmresults.co.uk 21
 Assessor 3 Standardisation Setup User Guide V2.1

Viewing Classified responses

All Standardisation Team members have visibility of the Classified worklist, however the types of response visible will
depend on your role. Only the Principal Examiner is permitted to see the Seed responses.

1 2 4 3 5 6 7

1 To view the classified responses, at any time,
click the Classified worklist.
2 The classified responses are grouped under their
respective classification types. The title headers
will display even if there aren’t any responses
classified of that type.
3 Click on the Response ID to open a classified
response. If you are the Principal Examiner
you will be able to edit the definitive marks if
Standardisation Setup is not complete. If you
edit marks then you cannot leave the response
unless it is 100% marked – if you remove marks
and try to leave the response a warning message
will display.
Note: the Response ID column remains fixed
in view if you need to scroll to the right to see
more information in this worklist.
4 The first column shows an order number and
this is the order that the responses will be
allocated to Examiners once Standardisation
Setup is completed.
The Principal Examiner can drag and drop a
response within the same classification type to
change the response allocation order if required.
The order number will remain visible if you
scroll to the right to see more information in this
worklist. Seeds do not have an order number as
they are always allocated in a random order.
5 Any Notes entered against the response are
visible from the worklist. Notes are visible in the
Standardisation Setup area only – they are not
part of the definitive marking comments.
6 Click and select a coloured tag to apply to a
response here if you wish. Note that tags are
personal to you.
7 Click View marks by question to view the
Classified worklist at a question mark level – this
view shows the marks for each question rather
than just viewing the total marks.
See section on Mark by Question view for more
information.

Note that when you open a classified response, you can see in the top header bar what type of response
it is and the order number assigned to it, if applicable. You can also use the previous and next buttons to
navigate through all classified responses.

rmresults.co.uk 22
 Assessor 3 Standardisation Setup User Guide V2.1

Viewing Seed categorisation information

A QIG may be configured to use seed categorisation. When this is the case IB will have specified categories that
apply to total mark ranges (e.g., a category of High may be given if the total mark is between 70 and 100) and also
the maximum percentage of seeds that should fall into each category (where the maximum percentages do not
have to add up to 100%). The aim of this feature is to encourage the creation of seed responses over a required
range of candidate abilities, however the system does not enforce this and exceeding a maximum percentage does
not prevent standardisation setup being completed. The categories and percentages used can vary per QIG.

Where this feature is in use, then:

• All responses in the Unclassified and Classified worklist will show the category associated with each response .

• In the Classified worklist, click next to the

classification type- Seed to see what percentage of
seeds currently classified are from each category,
against the maximum percentages set.
• Once the seed creation target has been met, if the
maximum percentage for any category has been
exceeded, then when you are in the Classified
worklist a message will appear to alert you that the
responses selected are not as desired.

Note: When a response is promoted to be seed during live marking, information on the categorisation
spread is also shown.

rmresults.co.uk 23
 Assessor 3 Standardisation Setup User Guide V2.1

Re-classifying a response
Principal Examiners can choose to re-classify a response from the Classified worklist if required, up until
Standardisation Setup completion. A response can be re-classified in one of two ways:

1 Select the response you want to re-classify from

the Classified worklist, and drag and drop it to
the classification type section in the worklist
that you wish to change it to. A confirmation
pop-up message will display when you drop the
response in the new location.

Note: If dropping the response to the new classification type exceeds the target set for that classification
type, then the response row that is displayed at the bottom of the category will be displayed in amber colour.

2 Open a response by clicking the Response ID. At the bottom of the marks panel, as long as the response is
100% definitively marked a Reclassify button will display. Click on this and select the required classification
from the pop up message.

rmresults.co.uk 24
 Assessor 3 Standardisation Setup User Guide V2.1

To de-classify a response
There may be occasions when you want to return some previously classified responses to the 100% definitively
marked state. Only the Principal Examiner is permitted to do this, although this option is not available on all
components. If it has been enabled, as the Principal
Examiner you will see a Declassify option against the
responses in the Classified worklist if Standardisation
Setup has not yet completed. Click this option
against the required response and respond to the
confirmation prompt.
The de-classified response is moved back to the
Unclassified worklist.

Note as the Principal Examiner you can also discard a classified response prior to Standardisation Setup
completion. This can be achieved by clicking on the ‘Discard’ button from within the classified response.
It will then show as available for provisional marking once again in the Select responses worklist, all marks
and annotations will be lost.

rmresults.co.uk 25
 Assessor 3 Standardisation Setup User Guide V2.1

Completing Standardisation Setup

Completing Standardisation Setup fixes the definitive marks for the Practice, Qualification, 2nd Qualification and
Seed responses, moves the Examiners out of Simulation mode (if this was configured for the QIG), and allocates
practice responses (assuming a practice target has been given). Only the Principal Examiner can mark the
Standardisation Setup process as complete, but this can only be done once all required targets are met for each
classification type, and should only be done once the IB Subject Manager has confirmed they are happy for this step
to be completed.
• In the majority of QIGs, the Complete setup button will not show at all for the Principal Examiner until the
IB Subject Manager has checked they are happy for setup to be completed, and have then enabled the
display of the button (which they do via the Administrative Interface).
• In other QIGs the button will always be displayed to the Principal Examiner. It will initially display in a greyed
out state, and only show as enabled once all required classification targets are met for the QIG. In this
scenario the Principal Examiner must wait until the IB Subject Manager gives them approval to mark setup
as complete (verbally or via email).
As Principal Examiner, you MUST ensure you are happy with your selection of classified responses and definitive
marks before completing standardisation setup as you have no ability to edit the marks or classifications once
Standardisation Setup is completed.

To complete Standardisation Setup:

1 Click the Complete Setup button available

beneath the Classification Summary.

Note: If the Complete setup button is

disabled, when you move your pointer over
the button a tooltip will appear to explain
what needs to be done for it to be enabled.
If the button is not visible, and mandatory
targets have been met, you must wait for an
IB Subject Manager to give you access to
use it.

2 Click Yes to the confirmation prompt if you

are satisfied with your response classifications
and definitive marks.

Notice that the QIG status shown on the Home page changes from a Standardisation Setup button to a link and the
‘My Marking’ button is now displayed against that QIG.

rmresults.co.uk 26
 Assessor 3 Standardisation Setup User Guide V2.1

What happens next?

Once the completing setup is confirmed, RM Assessor will:
1 Make practice responses available to all Examiners in the QIG (assuming they have a practice target).
2 Return the definitive marks for all classified responses to the IB.
3 In general an IB QIG is configured so that the marks of shared provisional responses that have not been
definitively marked, and the marks of 100% definitively marked responses that have not been classified, are kept
and these marks returned to the IB. The shared provisional marks are credited to the provisional Examiner and
will show in their closed Live worklist. The definitively marked responses are credited to the Principal Examiner,
and appear in their closed Live worklist.
However this configuration may not be on for all QIGs. Where it is not turned on then the marks of provisional
responses not definitively marked, and any 100% definitively marked responses not classified, are lost, and the
responses will be returned to the marking pool.
4 Any provisional responses not shared are removed from the Provisional worklist and returned back to the
marking pool.
5 If Re-use RIG functionality is enabled, all definitively marked responses are copied to the Re-usable responses list
for future use when standardising components that share the same structure.
As a Standardisation Team Member you can still access the Standardisation Setup area to view and remind yourself
of how classified responses were definitively marked. As a Principal Examiner you may wish to return to the
Standardisation Setup area to create additional Seed responses during the session.

rmresults.co.uk 27
 Assessor 3 Standardisation Setup User Guide V2.1

Standardisation related reports

A number of reports are available to allow you to monitor the Standardisation Setup progress and to monitor your
teams progress in undertaking qualification responses once Standardisation Setup is complete. To access these click
on Menu, and choose Reports. The reports available will depend on your marking team role.

1 Select the report you wish to run.

2 Select the required report parameters.

5 4 3 6

3 Click Execute Report Now to run the report To return to the main report use the blue back arrow
on the report toolbar.
4 You can export the report (in a variety of
formats) or print it, if you wish
Note: When using a tablet device, printing
reports is not available and the export options
are also more limited.
6 Click the Back button to return to the report list.
5 If the report has a sub-report, click on
underlined links to access it.

rmresults.co.uk 28
 Assessor 3 Standardisation Setup User Guide V2.1

Requesting a report for later access

If you know the report will contain a lot of data and may
take a while to run, you may wish to instead choose
Request Report after completing the report search
criteria. Enter the name for your report and choose a
format and click Submit request for Report. The report
will then generate in the background whilst you carry on
with other marking and team management tasks.

To later view the report, choose Requested Reports from the report list, and when it is available then click on the
icon in the Download column to save the report. It will remain available from this screen for 3 days.

Standardisation Set-up Report

This report shows the Standardisation Setup targets and the progress made to meet these targets. The report is
useful to the Standardisation Team in the period between when responses are first uploaded to RM Assessor for a
component, and the Standardisation meeting, to ensure that sufficient provisional marking occurs. It can also be
used during the Standardisation meeting to monitor progress towards the targets.

Standardisation – Optional Questions

For e-bookmarking components with optionality, this report provides details of how many responses have each
question answered, and it is then possible to drilldown on a question to see the details of responses where that
question was answered. This is useful to identify responses you may wish to consider for provisional marking.

Standardisation Progress Active Examiners Only Report

This report monitors the progress of Standardisation as Examiners work towards approval by marking and submitting
their qualification samples.
The report can be run for a single component or for all components an Examiner is associated with. At the top-
level, it displays a row per QIG and a count of the number of Examiners that have reached each milestone in the
standardisation process. It is possible to drill down for an individual QIG and view a line per Examiner to review what
stage each individual Examiner is at.
The report is most useful in the early stages of marking to ensure that Examiners are making progress towards being
approved. This report shows the Examiner side of the approval process while the Supervision Progress report shows
the team view.

Standardisation Feedback Report

The Standardisation Feedback report is available when Examiners have marked and submitted their qualification
responses. The report displays a summary of how many of a team member’s qualification responses were accurate,
within tolerance and outside of tolerance.
A detailed analysis of each submitted qualification response can also be viewed, showing:
• For each of the questions within the QIG and for the total mark, a display of the definitive mark, the Examiner’s
mark, and the deviation between these (where the mark deviation also shows if the Examiner’s mark was under
or over the definitive mark).
• The total deviation of the Examiner’s response.
Your team members can see their own standardisation feedback report in RM Assessor.

rmresults.co.uk 29
 Assessor 3 Standardisation Setup User Guide V2.1

Logout and Settings

2 1 3

1 Click to logout or change the settings.

2 Click Edit settings to expand the window to
view all the user settings grouped under different
headings (as illustrated here).
3 Turn this option off if you no longer wish to see
the blue pop up on screen hints whilst using RM
Assessor for Standardisation Setup.

Note: After 20 minutes of inactivity you will

be automatically logged out.

rmresults.co.uk 30
Support

Send us an email:
emarking@ibo.org

Telephone us:
+44 (0)29 20517114

©RM Education Ltd 2021