===Audit Workflow 1===

Goal: to look at the jobs completed by other raters, and determine if they are correct/incorrect.
Schedule: est Tue 2/8 - Tue 2/16

Instructions:
Refer to the sample screenshots for details.

● Step 1: Go to the Audit Assignment tracker here. You will see your name and your
auditor number in column A & C. You will also see the name of raters you’re assigned to
audit in column J, as well as the number of audit jobs assigned in column L.
Note: We have specifically assigned the audit jobs so that there are no overlaps
between raters. Kindly maintain a respectful working environment by working on
your own assignment. Please pay attention and notify the project team if you see
that someone is doing the jobs that are already assigned to you.
● Step 2: Turn on SRT proxy. Open the audit job page. On the “Filter by” dropdown,
choose the name of the rater you are assigned to, in order to see their jobs. By default
you will see the most recent jobs at the top. You will also see black or yellow dots next to
each job, please only click on the black dots.
Note: If you filter for a rater and you see an error message, refresh the page and it will
load. If the proxy is busy, you may also not be able to see anything on the screen, so
refresh the page and wait until it goes back up.
● Step 3: As you know, each job contains full body boxes and head boxes, so sometimes
the jobs can look very ‘crowded’. For example, this screenshot below can be really hard
to audit. Let’s click the icon circled in red.
● Step 4: If you uncheck the ‘Supplement’ box and keep everything else the same, you will
be able to see only the head boxes, which will help you focus on the audit task. If you
turn off the ‘Show labels’ option, you can hide the black labels. It is helpful when the
object is small and blurry and it is blocked by the floating text.

● Step 5: This step is not a must, but in case you need to see the original full body boxes:

● Step 6: click the tag button on the left, choose "Approved" if the image is perfect, but
choose "Incorrect Annotation" if it contains at least 1 mistake (mistake = violation of the
guidelines). Even if you see 7 boxes, 6 correct but 1 wrong, please click “Incorrect
 Annotation”

● Step 7: for every job that is labeled as 'Incorrect Annotation', please click the right button
button and click Send to Re-do. Click "Send with Annotation" and also write the reason
in sentences? Something like "missing the child in the background" or "missing the arm".
We're trying to get another group of auditors to re-do the images that you rejected. This
is the reason why, when you reject a job, please click 'Send with the original
annotation" and please write a brief clear summary of the errors, so that the other
auditors who picked them up can clean up the job faster, by following your direction. Of
course, if the original annotation is so bad, it doesn't make sense to "send with the
original annotation", because the auditors can do it faster by re-annotating from scratch,
so please exercise a good judgement based on that.

Bad comment example:

3rd box from the left: Right & bottom sides are drawn too tight or big. 4th box from the
left: Bottom & right sides are drawn too big. 6th box from the left: Right side is drawn too
tight. 7th box from the left: Bottom side is drawn too tight. 8th box from the left: Left side
is drawn too tight. 9th box from the left: Right & bottom sides are drawn too tight or big.

Good comment example:

Multiple tightness issue on 3rd, 4th, 6th, 7th, 8th, 9th boxes from left.

Bad comment example:

The left box is too tight on the right and top cutting the head. The right box is too wide on
the right side and it exceeds the edge of the picture. The box in the back is too tight on
the left side and top cutting the head.

Good comment example:

left box -> error on right & top side. right box right side -> too big. box in back -> too tight

● Step 8: Please fill in the details for all of the jobs you audited in your own auditor
worksheet. Go to the Audit Assignment tracker here. You will see your name and the link
to your own worksheet in column F. As a best practice, please keep a respectful working
environment by working only on your own worksheets. It is also good to check if the
number in the worksheet is the same as your own auditor number. You can also add
your own name in the file to make it easier for you.
Worksheet instructions

Fill in the details as follows:

● Column A: just copy paste the task URL from your browser.
● Column B: please write ‘1’ to indicate that this is a correct job
● Column C, F: write ‘1’. Column C and F are mutually exclusive–you can’t choose both,
you have to choose either one.
○ The 4 examples in the red rows have been added for reference. For example, in
the first example, it means that “This job (URL here) is a rejected job and as an
auditor, I consider this job to be incorrect. The reason is, the original rater
rejected it because the job contained no heads, but actually the correct rejection
reason was that the job was missing some full body boxes from the beginning”
● Column D, E, G, H: choose the correct description from the dropdown
● Column I-L: write the number of boxes that fall into the category. It is very possible that
a job contains multiple mistakes, so you can choose multiple categories.
○ The 4 examples in the red rows have been added for reference. For example, in
the second example, it means that “This job (URL here) is a submitted job and as
an auditor, I consider this job to be incorrect. The reason is, the original rater
mapped 1 head box to the wrong full body box and I found 2 head boxes which
were imprecisely drawn”
● Column M: optional comments

Reward: The same hourly rate applies. Your reward will be calculated based on the ‘Job Count’
x 90 seconds AHT.

Job count: The total job count is self-reported and it is based on your worksheet, as Halo
doesn’t track how many audit jobs have been completed in this method. Please note that we will
do a random spot check periodically to check if the URL that you wrote indeed belongs to you.

AHT: This audit method isn’t tracked by SRT timer, therefore we guarantee a fixed 90 seconds
AHT for everyone. The reason is that the task consists of two parts: auditing in SRT & filling in
the worksheet, which is estimated to take 90 seconds each. Please note that the task of auditing
the job in SRT itself should be done quickly. If you take a long time to audit, it’s very likely that
you are still using an annotating mindset while auditing, which unfortunately does not really
apply in this case. You are not requested to fix the job, only to say 'correct' or 'incorrect'. The
annotator mindset applies to the next audit task (re-doing the incorrect jobs), but not this one.
Do not go too deep on each mistake, but identify where all the mistakes are & leave short
comments for the other team to fix. Do not write long comments, be concise but clear.
===Audit Workflow 2===

Goal: For all incorrect jobs identified during the audit period, please re-do them according to the
instructions.
Schedule: est Tue 2/16 - finish. Audit workflow 2 will only begin as the team finished Workflow
1, please kindly wait for further instructions.

Instructions:
● Step 1: In the previous audit workflow, every incorrect image which was sent for 'Re-do'
would go to an audit queue called ______You will see it when you log in to your Home
page in SRT.
● Step 2: Every day we expect there are going to be new jobs that are sent to this audit
queue.
● Step 3: The first auditor who flagged this image as incorrect was asked to write a brief &
clear comment about the error found; as well as sending the original annotations
together. The intention here is to make sure that you don’t waste time re-annotating 10
boxes from scratch if the original annotator only made minor mistakes. Please do expect
that some jobs will not come with original annotations because if the original boxes are
so bad, auditors can re-do the jobs faster by re-annotating from scratch, so please
exercise a good judgement based on that. Please also remember the original auditors’
comments are not going to be correct 100%, so please keep in mind and be confident in
fixing the jobs according to your understanding.

Reward: The same hourly rate applies. Your reward will be calculated based on the Total
Handling Time with 90 seconds AHT rule, just like the production queue, because this audit
workflow will be tracked by SRT. The AHT will reset to 0 as we begin and it will not be mixed
with your production data.