Whale Keypoint 
Handbook 
 

 
 
 
 

History / Change log

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Table of contents
● Workflow
- Step by step guide

● Whale Keypoint ​Spec Document

○ Labels & Attributes Table 
○ Correct Task Examples 
○ Annotation Rules 
○ Annotation Logic 
○ Critical vs. Non-Critical Errors 
○ Labels & Attributes Description/Examples

● Human Keypoint Annotation in 3D

○ Whale_keypoint_pilot_05min_single 
○ Whale_keypoint_pilot_05min_10frame  

● Webinars

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Workflow

Whale​ ​Spec Document 

“Lidar Cuboid Cleanup”​ | ​IN PROGRESS 
 
 
Contents 
● Annotation Rules​ -- needs to be locked by end of Qualification Run 
● Critical vs. Non-Critical Errors​ - needs to be locked by end of Qualification Run 
● Edge Cases​ - iterative 
● Labels & Attributes Description/Examples​ - needs to be locked by end of Qualification Run 
 
 Annotation Rules 
 
Rule  Description  Example 

 Delete non-pedestrian cuboids  Please delete all cuboids that don’t  E.g. delete a car cuboid 
have the label “pedestrian”   

v 

 Max distance pedestrian cuboids   Pedestrians should be within 25  Keep: 

meters away from the sensor 
(SDC/ego vehicle). 
 
Please delete any pedestrian cuboids 
that have a max distance greater than 
25 meters. 
  
 
Delete: 

 Pedestrians in camera view  Please delete pedestrians that don’t  Delete - pedestrian not in camera 
appear at all in the camera view -  view: 
either fully occluded or are not in the 
range of the camera. 
 
It’s okay to keep pedestrians that are 
partially occluded.   

 
 
Delete - pedestrian is fully occluded: 
  
 
Keep - pedestrian is only partially 
occluded: 
 

Cuboid orientation  Please be sure you have looked at the  Incorrect: 

following instructions before adjusting 
cuboid orientation: 
 
Delete non-pedestrian cuboids  
Max distance pedestrian cuboids  
Pedestrians in camera view 
 
Fix the cuboid orientation for all 
pedestrians within max distance. 

 
  
 
Correct: 

Camera view  Check both camera view and lidar to   

locate objects listed in the label table 
Rely on lidar view for sizing and 
positioning. 

Cuboid sizing  There should be no more than 20cm   

of space between the cuboid border 
and the closest point on the object. 

Point drift or echo points  Most LiDAR will return a type of error   
for static objects - where points drift 
off the front or back of a stationary 
object.  
When this occurs - fix the cuboid to 
the front-most points of the object, 
even when it is clear these are 
duplicate points. 
 
 
 
 
Edge Cases 
 
 
Edge Case  How to Handle It  Example 

     
 
 
 
Critical vs. Non-Critical Errors 
 
 
Critical Errors  Non-Critical Errors 

Missing Keypoints  Critical error message will automatically show, upon 

submission of task especially when certain attributes are set 
to ​Invisible. 

 
 
 
 Labels & Attributes Description/Examples 
 
Labels/Attributes  Description  Example 

 Label 1  Label 1 description  Insert image here 

 Attribute 1  Attribute 1 description  Insert image here 

 
 
 
 
 
 
 
 
 
 
 
Whale​ ​Spec Document 
“Lidar Point Annotation”​ | ​IN PROGRESS  
 
 
Contents 
● Labels & Attributes Table 
● Correct Task Examples 
● Annotation Rules 
● Annotation Logic 
● Critical vs. Non-Critical Errors 
● Labels & Attributes Description/Examples 
 
In this task, you will draw ​points​ on parts of pedestrians in the ​3D view​. Draw points by clicking the 
“Key Point Annotation” button in the left side panel. 

 
 
 Labels & Attributes Table 
 
Class  Label (Keypoint)  Attribute 

Human Skeleton  Center of Head  is_visible   

 
  Nose 

Shoulder-Left 

Shoulder-Right 

Elbow-Left 

Elbow-Right 

Wrist-Left 

Wrist-Right 

Hip-Left 
 Hip-Right 

Knee-Left 

Knee-Right 

Ankle-Left 

Ankle-Right 
 
 
 Correct Task Examples 
 
2D View  3D View 

 
  ^Notice 
1. The key points are ​where the bones 
meet​ (not on the outer part where the 
skin is). 
2. The points are ​inside​ the pedestrian, not 
on the skin. 
 
 
 
  
 
 
 
Annotation
  Rules 
 
Rule  Description  Example 

Max distance  Pedestrians should be within 25   

meters away from the sensor 
(SDC/ego vehicle). 

Add keypoints where joints  The keypoints are locations  Key Points 
meet   inside​ the human body where 
two bones meet. They are ​not 
surface/skin points, except ​Nose​. 
Pedestrian present in camera  Pick a human that’s also shown 
image  in a camera view (i.e. present in 
the lidar point cloud and a 
corresponding camera image). 
However, not all keypoints have 
to be visible in the camera image. 
 
Use the 3D view to place points 
in the correct spot. 

Left and Right labels relative to  Please use corresponding 

the pedestrian in the frame  Left/Right labels to be relative to 
the pedestrian in the image. 

Omit occluded keypoints   If the person is not visible in the  Add​ keypoints for Knee-Left 
2D camera at all  and Knee-Right and mark as 
OR  “invisible”. 
 
It is impossible to guess where 
Omit​ keypoints Ankle-Left 
the keypoint should be  and Ankle-Right: 
THEN   
Do NOT draw a keypoint. 

 
  

 
 
 
 
Annotation
  Logic 
 
Situation  How to Handle It 

Obvious where the keypoint should be  Annotate point 

Mark visible 

Not obvious where the keypoint should be, but  Annotate point 
we can guess with high confidence based on  Mark ​in​visible 
knowledge of human shape 

NOT obvious where the keypoint should be  Don’t annotate 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Labels Description/Examples 
 
Label  Description  Example 
 Center of Head  Center point between the ears 
and inside the head. 

 
  
 Hip   Hip keypoint should be where 
the leg and hip bones meet - 
inside the body, ​NOT at the 
skin​. 

 
  
 Nose  Tip of the nose. 
 
The Nose and Center of Head 
point together create an 
imaginary line in the direction 
that the person is looking. 

 
  
 
 
Attributes Description/Examples 
 
Attribute  Description  Example 
 is_visible  If a keypoint is present in an  Wrist Left is visible since it is 
image and its ​location is  obvious where the keypoint 
obvious ​it should be marked as  should be 
visible​.  
 
If it is not visible on the image 
or it's location is not obvious for 
a labeler, but ​could be roughly 
guessed ​based on prior 
knowledge about the human 
body and other visible 
keypoints it should be still 
labeled, but marked as 
invisible​.  
   
Sometimes it is not obvious if a 
keypoint should be labeled as 
visible or not because of image 
blur, occlusions colored similar 
to a person's closing. In such 
cases it is OK to label a point 
either way. 
 
 
 
 Human Keypoint Annotation in 3D

 Take Note:

● Keypoints should be labeled only if a pedestrian is visible on a camera. A keypoint should be labeled 
only if there are some lidar points around it to "reference" the location. 
● if you cannot see the pedestrian at ​all ​in camera -- p​ lease keep unlabeled 
● If you do see parts of the pedestrian in camera -- label as normal, try to add keypoints according 
to both camera and lidar data. A keypoint should be labeled only if there is enough lidar+camera 
data to "reference" the location. Use the "is_visible" as appropriate when keypoints are not visible 
in camera, but known from human knowledge 
● We know some tasks have obvious camera calibration issues, Whale is working on it. 

 
Whale_keypoint_pilot_05min_single
 
 Issues: 
● hips are marked on the surface and not inside the body 

 
 
● hips are marked on the surface and not inside the body 
● hips are way too high 
● Other points looks good 
 
  

 
 
 
● Hips are good! 
● Right shoulder and elbow are on the surface and not inside the body 
 

   
● right hip is on the surface 
● head should be marked as invisible, location is on the surface 
 
  
 

 
 
● looks like this task has some camera calibration issues 
● left wrist is mislabeled (should be the left hip) 
● all points on the right are off 
 

 
 
 
 
 
 
 
 
 
whale_keypoint_pilot_05min_10frame 
 Frame #1   
 
 
● Looks reasonable 
 

 
 
Other frames: looks like keypoints were not moved at all 
  
 
● First frame is OK 
● The rest of the frames there is no changes in keypoints location 
 
 

 
 
  
● Pedestrian is not visible on any cameras, but still was labeled somehow 
 

     
 
● Camera calibration or camera-lidar sync issues 
● Hips are on the surface