Professional Documents
Culture Documents
Handbook
Table of Contents
Table of Contents 1
Attributes 6
Occlusion 6
Occlusion from Fences / Poles 9
Light Attributes 11
Glass 11
Blurry Image of Object 11
Brake Lights Workflow 12
Indicator Lights 16
Multiple Indicator Lights on One Vehicle 16
Indicator Lights Workflow 17
Any Lights 21
Any Lights - Pedestrians 22
Light Attribute (Any Lights) - Non-Vehicle Objects 22
Reverse Lights 24
Can’t Tell [Reverse Lights - Brake Lights] 26
1
When the SDC is in a parking lot 33
Annotating Apartment Parking Spaces 34
2
Recent Updates & Changelog
April 2020
- Added clarifications on Ghost - Sensor Artifact
- Added clarifications on Ghost - Retroreflectors
- Added clarifications on Overpasses vs. Underpasses in Incorrect No Label Zones
- Added clarifications on including echo in all object sizing
- Added clarifications on embedded NLZs and how to navigate them.
March 2020
- Added clarifications about Car Dealerships and Auto Repair shops
- Added clarification about Temporary Construction barriers.
- Added example of highly blurred object w/ attributes
February 2020
- Added more examples of attributes
- Added guide on reporting Shifting Data & Missing Lidar Sections
January 2020
- Adding New Bee Attributes Examples
September 2019
- Recreated Bee Handbook from older version
- Removed outdated sections
3
Tips for Working on Bee Tasks
● To quickly add a label to multiple frames for a single object (i.e., occlusion attribute)
○ Select the initial frame which this change will be applied to.
○ While holding down the “shift” key, click on the frame where you want the change
to happen
○ After you have chosen the frames to modify select the desired label to be set to
those frames.
■ What this will do is - Everything that is inside the first selected frame and
the last selected frame will change and will be set to the selected label.
○ Example: https://share.getcloudapp.com/kpum6ZRO
○ Reference: MULTISELECT TOOL SCREENSHOTS (REFERENCE)
● Use the “L” hotkey to cycle through the attribute labels of the objects in the scene in lidar
view instead of checking each object’s attributes one by one.
○ Familiarize yourselves with the colors and what they correspond to per attribute to
speed up workflow
○ Example: https://share.getcloudapp.com/BluNdDPk
● Use the “O” hotkey to see every objects’ pathways (cuboids) in the scene or to see one
selected object's pathway through a scene.
○ The “O” hotkey also allows you to see the label changes for all of the attributes.
This is extremely helpful when working with global attributes which should remain
the same in all the frames as well as making sure that the objects path alignment
is correct.
○ Example: https://share.getcloudapp.com/Z4uw1rEq
● Using the “O” and “L” hotkeys together is a very efficient way of checking an object’s
attributes throughout a scene. This is very useful in checking non-global attributes
(occlusion, brake lights, and signal indicators)
○ Remember to always make your decision, and double-check, an object’s attribute
label with what you see in camera view.
● When deciding which occlusion label to use for an object that appears in multiple camera
views, make sure to use the camera view with the least % of the object occluded,
● When on a freeway or road, you should check if any of the cars behind the SDC passes it
in any of the frames within the task, if they do not it is usually safe to put “can’t tell” for the
reverse and brake lights attributes
● Turn signal indicator lights can be found on the back, front and side-mirrors of the car.
○ When you can only see the front turn signal indicator make sure to base your
choice of label off which direction the driver of that vehicle is signaling towards
4
Understanding Global and
Non-Global Attributes
● What is a Global Attribute
○ Global attributes are labels that will remain the same in all the frames - What this
means is that if at any point in the scene, let’s take for example, “Any Lights” - if at
any point in the scene you can see the objects brake lights turn on - the label for
“Any Lights” should be set to “On” in ALL THE FRAMES. In other words these labels
will always be the same in all the frames.
● List of All Global Attributes - Label Remains the Same in All Frames
■ “Is the object parked in every frame?”
■ “Are any lights on? (in at least 1 frame)”
■ “Are the object’s reverse lights on? (in at least 1 frame)”
■ “Is the vehicle towing or being towed?”
■ “Does the vehicle or animal have a rider?”
5
Attributes
Occlusion
● 0-20%: Object is fully visible in camera view OR object is slightly blocked but the
majority of it is still visible in camera view.
● If the object is coming straight towards the SDC, meaning that the front of the
object is facing straight towards the SDC, and there is nothing blocking the object,
the label should be set to 0-20%.
● 20-40%: Object is slightly (more) blocked - Majority of the object is still visible in
camera view OR More than 50% of the object is still visible in camera view and
only a section of the object is occluded/blocked.
6
● 40-60%: Object is partially being blocked to where half of it can still be seen OR the
object is more than halfway blocked but a good amount of the object can still be
seen in camera view.
● 60-80%: More than half of the object is occluded/blocked OR most of the object is
occluded/blocked but there is still a part of the object which is visible in camera
view.
● 80-100%: Object is almost fully occluded/blocked (small part of the object can still
be seen) OR object is fully occluded (object is not visible in any of the camera
images)
7
● Important! Objects that are visible through glass are treated the same way as
objects behind a wall - Label should be set to 80-100% if object is fully covered by
the “wall” or “glass”.
● Can’t Tell: Should be used for areas labeled as a “No Label Zone” and when an
object is too far away to reliably answer occlusion value
8
Occlusion from Fences / Poles
● There might be an instance where a vehicle object might pass behind a fence - This
creates occlusion in various parts of the vehicle.
○ In this scenario - Add up the surface area of the poles/fences (The amount of
space that the poles/fences cover) and treat that value as you would normally for
occlusion.
○ In the example below the vehicle is occluded by a fence - In this case we look at
how much the poles cover the vehicle - We then add up the amount of space that
the poles take (this is subjective - Try to estimate it the best you can) - In the case
of the image below an acceptable guess would be 0-20% OR 20-40% occlusion.
9
10
Light Attributes
Glass
- Glass DOES NOT affect any of the Light Attributes
- If a turn signal indicator, brake light, reverse light, or any other light is clearly On/Off
through glass, select the correct attribute.
11
Brake Lights Workflow
12
- If you can’t see the brake lights in this frame due to the angle of the object (ex. 1, 2), or
occlusion (ex. 3) mark the brake lights as Can’t Tell in this frame
- When vehicles are faced perpendicular to the camera view, the Brake Lights should be set
to 'Can't Tell', unless there is an obvious change between frames that indicates active
Brake Lights.
13
- If you can clearly see the brake lights on and active, mark the Brake Lights attribute as On
14
- If you can clearly see the brake lights off, mark them as Off
- If you can’t tell whether the brake lights are on or off due to blurriness, glare, or any other
uncertainty, mark the brake lights as Can’t Tell
15
Indicator Lights
Multiple Indicator Lights on One Vehicle
If a vehicle object has multiple turn indicators which are signaling at different times the attribute
should correspond to which lights are currently on or off.
Headlight and Side mirror display a turn signal light - both of them activating with different
timings - label should be set to "Off" only when both signals are off.
Signal lights can be found on the front and back lights.
16
Indicator Lights Workflow
Follow this workflow when working on the Indicator Lights Attribute:
17
18
- If no indicator lights are visible on the object due to occlusion, mark the Indicator Lights
Attribute as Can’t Tell.
- If you can clearly see the indicator lights Off or On, mark the Indicator Lights Attribute with
the corresponding selection
19
- If we can’t clearly see the indicator lights on the vehicle due to blurriness, glare, or angle,
we should mark the Indicator Lights attribute as Can’t Tell
20
Any Lights
- Set as “On” if you can definitely see the lights (brake, turn-indicator, reverse, headlights,
etc.) on in any frame.
- Set as “Off” if you can see that the lights don’t turn on in any frame.
- Set as “Can’t Tell” if you can never see the vehicles lights on any point in the sequence.
This means that there is never a clear image of the objects headlights, backlights, turn-
lights, etc.
- Important! If the vehicle displays any external OR internal lights which are on - Label
should be set to “On” in all the frames.
21
Any Lights - Pedestrians
- Pedestrians which are holding cell-phones, flashlights or any object which may give off
light should be considered for the “Are any lights on?” attribute. This also includes any
animals that may have lights attached to them in any way.
- Example - Pedestrian Holding Cell Phone
- If a pedestrian is holding a cellphone and you can clearly see the screen is on
attribute label should be set to “On”.
- If a pedestrian is holding a cellphone and you can clearly see the screen is off
(black screen) attribute label should be set to “Off”
- If a pedestrian is holding a cellphone and you can’t see the screen in any image
attribute label should be set to 'Can't Tell’.
Pedestrian holding cell phone: We can’t see the screen light - “Any Lights?” attribute should be set to “Off”
Pedestrian holding cell phone: We can see the screen light - “Any Lights?” attribute should be set to “On”
22
● Objects which have reflectors on them should be labeled as “Can’t Tell”
23
Reverse Lights
Follow this workflow when working on Reverse Lights
24
- If you can never see an object’s reverse lights at any point in the scene due to angle,
occlusion, blurriness, or glare, mark the Reverse Lights attribute as Can’t Tell.
25
- Set as “On” if you can clearly see the reverse lights on at any point in the scene.
- Set as “Off” if you never see the reverse lights on at any point in the scene.
26
27
Annotating Specific Areas
Auto Repair Shops
● Vehicles inside of auto repair shops should be annotated individually
● The parking lots of auto repair shops can be annotated with a No Label Zone
This auto repair shop has an interior area where cars are fixed, and an exterior parking lot
The cars being worked on inside the garages should be annotated individually.
28
The objects in the parking lot outside can be covered with a No Label Zone.
29
Car Dealerships
● Cars inside of car dealerships (where cars are sold) should not be annotated
● Cars outside of car dealerships (parked in the parking lot in front of the store)
should be annotated with a No Label Zone
Annotate the cars parked outside of the dealership with a No Label Zone
30
31
Fast Food Drive-Thrus
● Fast food drive thrus ARE NOT part of the parking lot, and any cars that are in the drive-
thru need to be annotated with individual cuboids.
● Even if the restaurant is in the corner of a parking lot, all objects in the drive-thru area (e.g.
cars and pedestrians) need to be annotated.
32
When the SDC is in a parking lot
● If the SDC is driving inside, exiting or entering a parking lot all cars, pedestrians, and other
objects need to be annotated.
● If there are any other parking lots in the scene which aren’t connected to the parking lot in
which the SDC is in, we then annotate the separate parking lost as a “No Label Zone”.
33
Annotating Apartment Parking Spaces
● Parking spaces meant for apartment buildings should be considered similar to parking
lots and should be annotated with a ‘No Label Zone’.
● Important! It is unnecessary to annotate the driveway leading into this parking area as a ‘
No Label Zone’ because vehicles can be found entering or exiting the NLZ from these
areas.
34
Annotating Specific Objects
Towed Objects: Semi-Trucks
Annotate semi-trucks with two separate cuboids:
35
Utility Trucks: Truck
Utility Truck - Modified pickup trucks which are larger-size than a regular pickup truck should be
labeled as “Vehicle - Truck”
36
2 Pedestrians with 1 Large Object
- Split the large object between the two pedestrians, and label both cuboids as “Pedestrian
with Object - Adult”
37
Trailers/Large Trucks - Limited Views
● If the SDC is behind a large truck trailer and you can only see the shipping container (the
back of the Truck) and the object is clearly moving but you cannot tell if there is an object
towing it - Label should be set to “Being Towed”
38
Labels Changing Mid-Scene
● There might be cases in which objects might need a label change at some point in the
scene. This is mostly seen on pedestrians which might be holding objects and then
setting them down. In this case we might need to change labels and annotate the object
which was being held by, in this case, the pedestrian.
● If an object needs to change labels mid-scene, create a new cuboid in its path with
the new label, and new sizing, if necessary.
● If a person is holding a large object the label should be set to “Pedestrian with
Object - Adult”.
● If they throw that object we start two new cuboids - one “Pedestrian - Adult” and
one as “Other Dynamic Object”
● SCENARIO:
○ If a construction worker picks up a cone in one of the frames - delete the
‘Pedestrian - Specialty’ cuboid for the frame where the object gets picked up
and add the new cuboid with label ‘Pedestrian with Object - Specialty’.
○ If they set down the cone, remove the ‘Pedestrian with Object - Specialty’
cuboid, and create a ‘Construction Object - Construction Cones and Poles’
cuboid for the cone, and a ‘Pedestrian - Specialty’ cuboid for the
construction worker.
● Important! This example also applies to multiple cases - For example: When a
pedestrian (adult) who is on a bicycle gets off and pushes the bicycle.
39
○ The label would change from “Vehicle- Bicycle” with the rider attribute
selected as “With Adult Rider” to “Vehicle-Bicycle” and “Without Rider”. When
this happens a new cuboid will be created for the pedestrian who is pushing
the bicycle “Pedestrian With Object - Adult” label.
● When a bicycle has a rider the cuboid should be made to fit both the rider and
bicycle, but when it changes to a bicycle without rider the cuboid should be sized
to ONLY include the bicycle.
40
Vehicles with Large Objects on the Roof
● Any ,and all, objects on top of a vehicle’s roof should be included within the vehicle object
cuboid.
● This, for example, includes trucks hauling lumber, passenger vehicles with luggage bins
on top, other self driving vehicles with LIDAR devices, etc.
41
42
Forklifts
● Forklifts should be labeled as ‘Vehicle - Other’. Include all parts of the vehicle into the
cuboid when annotating (e.g. the forks, or the hydraulics).
43
Mail Truck, Mail Van, or Mail Lorry
● Mail trucks, like the one pictured below (1st image) should be labeled as ‘Vehicle - car’.
● For larger mail trucks or semi-trucks annotate them with the proper label for the object.
44
Wood Chippers
● Wood Chippers, or mulchers, which are mounted on trailers should be annotated as ‘
Vehicle - Trailer’. If the object is being towed, make sure to give the object the attribute ‘
Being towed’.
● Important! If there are any tree branches or other debris, it needs to be annotated with ‘
Static Road Object - Other’.
45
Mobile Traffic Control vs Construction Object Signs and
Sign Boards
46
● Electronic constructions signs differ
from mobile electronic signs -
Stationary electronic signs that warn
about construction are to be labeled
“Construction Object - Construction
Zone Signs and Construction Sign
Boards”.
● Important! Any electronic signs that
its purpose is to redirect traffic
should not fall under this label even if
next to a construction zone.
Ghost - Retroreflector
LiDAR points associated with retroreflective road signs or on-road markers.Ghost points will be <1m
away from the physical retroreflector.
47
Common Places to Look For
Missing Vehicles
Behind rows of stopped cars at intersections
48
● Look for gaps and spacing between cars which are stopped at stop lights. These cars are
heavily occluded but there may be enough points to annotate the vehicles which are in-
between.
● Watch out for other vehicles that are occluded by gas pumps. Check for the normal areas
that cars may be pumping gas as well.
49
Frequent and Common
Errors
Global Attribute Errors
● Reverse Lights - If in one frame the objects reverse lights are on - Label is set to “On” in
every frame.
● Parked Vehicles - If a car is parked in a parking lot and at any point in the scene it moves
the label should be set to “No” in every frame. Vehicle MUST be parked in EVERY FRAME
OF THE SEQUENCE for the label to be set to “Yes”.
● Do not label cars at a stoplight as parked.
S
Small Passenger Van with Label: Vehicle - Car
50
Vans used for transportation of equipment or more than 8 people with Label: Vehicle- Van
51
Incorrect No Label Zones
● Be careful when placing ‘No Label Zones’. Use as few cuboids as possible to cover the
parking lot. Try to avoid having overlap between cuboids as much as possible. Do not
include any buildings or sidewalks within the cuboid.
● DO NOT label any other classes within the ‘No Label Zone’ - UNLESS it is an object
that enters, exits or is waiting to exit a NLZ (examples).
● When creating a ‘No Label Zone’, create one large cuboid to cover the entire area.
52
○ If the Parking Lot has an abnormal shape, use as many cuboids as
necessary to cover the entire area without going over buildings, sidewalks,
or any other area that does not represent the zone being annotated
○ If the Ego Vehicle is on an overpass, and when labeling a lower road with
‘No Label Zone’, use one ‘No Label Zone’ to cover the left side of the lower
road and one ‘No Label Zone’ to cover the right side of the lower road.
○ If the Ego Vehicle is driving below an overpass, DO NOT use a NLZ for
vehicles above the Ego Vehicle
● Cuboids are permitted to overlap slightly in order to fully contain the zone but try to
avoid overlapping as much as possible.
● DO NOT cover Gas stations, auto repair shops (ie. oil changers), fast food drive
thrus and house driveways with no label zones - We should always label all objects
in these areas individually.
● DO NOT make small strips to cover rows of parked cars
● DO NOT cover bus stop roundabouts with a ‘No Label Zone’
● DO NOT label vehicles parked along the road with a ‘No Label Zone’
● DO NOT annotate driveways as no label zones.
● DO NOT annotate parking lots that do not have any vehicles in them
● DO NOT annotate parking lots that are right next to the street (without driveway or
curb separating them) as ‘No Label Zone’.
53
● The ‘No Label Zone’ should cover all stationary vehicles, alleys, or driveways within
the parking lot.
● DO label objects in parking lot driveways as they are exiting or entering parking
lots.
○ Objects that enter a "No Label Zone" need to be fully annotated all the way
through.
○ Start labeling objects when part of the object exits the "No Label Zone".
○ EXCEPTION: Objects that are intending (waiting or moving toward) to exit
the "No Label Zone" (but do not exit during the sequence) need to be fully
annotated all the way through.
54
Do Not Annotate Reflections
● Important! Do NOT annotate any reflections. This could happen due to building reflecting
points off their windows or vehicles reflecting points off the body of the vehicle.
● When the SDC is near a building, usually by a building with lots of windows, it will pick up
the lidar points reflecting off of objects in front of it. Be careful not to annotate these false
lidar points.
55
● The pedestrians on the left of the orange line are reflections of pedestrians on the right.
● The building is causing a reflection, picking up additional lidar points of the objects
between the building and the SDC.
56
● It is good practice to check for reflections using “Camera View” - By using “Camera View”
we can view the area where we think the reflections are happening. It is also important to
remember that we should NOT annotate any objects that are indoors.
57
Incorrect 'Ghost - Retroreflector'
If seeing more than 2-3 labeled in a task, it’s Incorrect.
58
Incorrect 'Ghost - Sensor Artifact'
Not only will 'Ghost - Sensor Artifact' appear in the front and back of the SDC, but will also appear
on the sides. These typically follow the ego vehicle around.
The ‘Ghost - Sensor Artifact’ should not be confused with the hood of the SDC. Do not include
points from the hood in labeling of the ‘Ghost - Sensor Artifact’.
59
Reporting Missing LiDAR Sections
A task has a Missing Lidar Section if there is a pie-shaped area of black where lidar points should
be. This is an indicator that one of the lidar sensors is broken.
Any task with a MIssing Lidar Section should be Reported & Skipped immediately. You can
report tasks with missing lidar sections on Zendesk here.
60
MULTISELECT TOOL SCREENSHOTS (REFERENCE)
61
62
Occlusion References - Sectional Values
63
64
65
66
OCCLUSION EXAMPLES
67
40-60%: Object is moderately - more than half occluded
68
Handling Occlusion on Foggy Scenes
69
If it is too difficult to interpret Occlusion on an object due to darkness, fog,
or other elemental factors, select Can't Tell.
70