Peer Grading Guide

Software Design
Bachelor of Computer Science
2nd Year
February - March 2024

Introduction
This year, we will use peer grading for the first time. This means that each student team will
have to provide assessments, i.e., feedback comments and initial calibrated grades, of other
team submissions. Empirical evidence shows that systematic peer grading of software projects
leads to evaluations comparable to TA grading [1]. Additionally, it has the following advantages:
1. Peer grading encourages students to reflect on the quality of other submissions and to
compare them to their own. This allows students to practice and improve their analysis
and evaluation skills. Understanding and evaluating the design or code of others is also
a very important competence to have in Industry.
2. Reviewing other submissions allows students to get inspired by the design and/or coding
convention of others, exposing them to potentially beneficial concepts they might not
have come in contact with otherwise.
3. Peer grading lowers the considerable effort of individual staff members or TAs that
comes with grading assignments in courses with several hundred students, while
simultaneously facilitating beneficial learning outcomes [2]. In short: the teaching team
will be able to dedicate more time to directly support you.

Overall process
Our general peer grading process looks as follows:

1
Each team has to provide 3 peer assessments per assignment, i.e., 9 assessments in total over
the whole course. The assessments will always be about project tracks that the respective team
is not working on. For example, if team A works on project track 1, they will only assess
submissions of project tracks 2 and 3. For each submission, the respective TA will read and
compare the 3 assessments, inspect the submission, aggregate, refine, and extend the
feedback, and then finalize the grade per team. The peer grading is single-anonymous, i.e.,
graded teams do not know which other teams provided feedback for their assignment. Providing
the 9 peer assessments is mandatory, i.e., failing to provide them will lead to failing the course.
However, if done well, the peer assessments can also boost a team’s grade (see section at the
end). If you are unsure about how you should assess a specific detail, ask your Mentor-TA
about it.

Structure of peer assessments

Each assessment consists of two parts:
- Textual feedback in the form of PDF annotations or a separate PDF
- A calibrated grade for each separate part of the assignment
- Assignment 1: one grade for “description of the system” and one for “features
identification”
- Assignment 2: one grade per sub-heading of the document (see also Team
Project Guide)
- Assignment 3: one grade per sub-heading of the document (see also Team
Project Guide)
The textual feedback should point out mistakes and inconsistencies and make suggestions for
improvements. It should be formulated in a constructive manner, e.g., saying that some aspects
of the project are wrong is not enough, you will need to also briefly state how to fix them. Try to
be fairly neutral and objective. If you think something is very good, you can also make
comments about this.
Calibrated grades for A1 are pass or fail. For A2 and A3, the following ordinal scale is used:
1. Very poor: the assignment part is of very low quality and doesn’t fulfill the threshold for
passing, e.g., ignoring most assignment instructions, abundance of mistakes or
inconsistencies, not understandable, extremely low presentation quality, etc.
2. Poor: the assignment part is of low quality that is closely around the threshold for
passing, e.g., ignoring several assignment instructions, lots of mistakes or
inconsistencies, hard to understand, low presentation quality, etc.
3. Sufficient: the assignment part is clearly above the threshold for passing, but is still
impacted by several quality issues, e.g., it contains mistakes, inconsistencies,
understandability issues, presentation issues, etc.
4. Good: the assignment part is of good quality, with only some issues, e.g., regarding
mistakes, inconsistencies, understandability, presentation, etc.

2
 5. Very good: the assignment part is of very good quality, with no major issues and only
very few minor issues, e.g., regarding mistakes, inconsistencies, understandability,
presentation, etc., the assignment could only be objectively improved marginally.

Impact on the final grade

Each of the 9 peer assessments created by team A is graded by the TAs of the assessed
submissions, i.e., not team A’s own Mentor-TA. The following ordinal scale is used:
1. Missing: no assessment has been provided
2. Poor: the assessment is of insufficient length or quality, e.g., only 1-2 comments have
been provided in addition to the calibrated grade, the textual feedback is mostly useless
or incorrect, the feedback is very aggressive or offensive, etc.
3. Good: the assessment has sufficient details to help students improve the submission,
the feedback is mostly accurate and balanced
4. Excellent: the assessment has many helpful details and provides above-average
suggestions for improvement that are nearly all accurate, the feedback could only be
objectively improved marginally
If a team accumulates a single missing, they fail the course.
If a team accumulates 4 or more times poor, they fail the course.
If a team accumulates 3 to 5 times excellent, their final grade is increased by 0.2.
If a team accumulates 6 or more times excellent, their final grade is increased by 0.5.

References
[1] M. Aniche, F. Mulder and F. Hermans, "Grading 600+ Students: A Case Study on Peer and
Self Grading," 2021 IEEE/ACM 43rd International Conference on Software Engineering:
Software Engineering Education and Training (ICSE-SEET), Madrid, ES, 2021, pp. 211-220,
doi: 10.1109/ICSE-SEET52601.2021.00031.
[2] B. Tenbergen and M. Daun, “Calibrated Peer Reviews in Requirements Engineering
Instruction: Application and Experiences,” in Hawaii International Conference on System
Sciences, 2022. doi: 10.24251/HICSS.2022.106.