2Dbox Task

Overview
• General instructions
• Special instructions
– Common Mistakes
– Common Object Definition
– Special Rules
– DO NOT Label Objects if
• Labels
• Attributes

General instructions
• Draw bounding box around all objects from labels list, and label them
according to the instructions below.
• Do not apply more than one box to a single object.
• Some tasks may contain no labels.
Bounding Box color convention in example images: + Green: Objects like this
should be annotated. + Red: Objects like this should not be annotated

Special instructions
Common Mistakes
• Person Inside Vehicle: People inside/on vehicles should not be anno-
tated as pedestrians. They are considered as appendages of vehicles.

1
2
• Traffic Cone Shape: Traffic cone does not necessarily have to be cone-
shaped.
– Exclude: Do not annotate pedestrian yield sign as traffic cone.

3
4
• Non Movable Objects: Movable objects do not include pedestrians,

5
bicycles, wheel-chairs and strollers.

6
7
 • Stones And Rocks: If individual stones (>5cm) are on the driveable
surface, then each should be segmented as foreground object (aka. other
moveable object). Pebbles and small stones (<20cm) not on the driveable
surface should be terrain. Big static rocks (>20cm) should be vegetation.

Common Object Definition

• Movable Small Barrier: Movable small barrier should be labeled as
other movable object.

8
• Permanently Mounted Pole: DO NOT label permanently mounted
poles with a box.

9
• Orange Striped Barrel: Orange striped barrels should be labeled as

10
 traffic cones.

Special Rules
• General Occlusion: If an object is occluded, then draw the bounding
box to include the occluded part of the object according to your best guess.

11
12
• Vehicle Occlusion: If a vehicle is mostly occluded, then add box to it
if you can tell very confidently the object is a vehicle (e.g. When you can
see part of the wheel).

13
14
15
16
17
18
19
• Glass Buildings: If an object is inside or behind a glass building, an-
notate it as usual. This rule does not apply to reflections in the glass.
See the special rules for reflections.

20
• Reflection: If an object is reflected clearly in a glass window, then the
reflection should be annotated in the 2d box and linking tasks. Note that
for reflected objects, is_reflection attribute should be set to “Yes”.

• Extremities: If an object has extremities (e. g. arms and legs of pedes-

trians), the bounding box should include ALL the extremities.
– Include: If a vehicle door is opened, include it in the vehicle bounding
box.
– Exclude: Never include a vehicle’s side view mirrors or antennas.
– Exclude: Traffic signs are not considered extremities. For example, if
a traffic sign is attached to a traffic barrier, they should be annotated
separately.

21
22
23
24
25
26
27
28
29
30
31
• Nighttime Vehicles: In nighttime images annotate the vehicles only
when a pair of lights are clearly visible (break or head or hazard
lights).
– Note: If you are not 100% sure about the vehicle type select Car or
Van or Suv label.

32
33
• Mannequin: Mannequins should be annotated as Adult Pedestrian

34
• No Bike Riders: People on Bicycles and Motorcycles should
NOT be annotated in this task
• Carried Object: If a pedestrian is carrying an object (bags, umbrellas,
tools etc.), such object will be included in the bounding box for the
pedestrian. If two or more pedestrians are carrying the same object, the
bounding box of only one of them will include the object.
• Nighttime Pedestrians: In nighttime images annotate the pedestrian
only when either body part(s) of a person is clearly visible (leg or
arm or head etc.) OR the person is clearly in motion.
– Note: If you are not 100% sure about the pedestrian type (adult vs
child vs construction worker etc.) choose adult pedestrian label

35
• Pedestrian Bicycle Traffic Lights: For traffic lights for pedestrians
and bicycles we annotate the black surrounding frame with a bounding
box and the lenses/bulbs with attributes. We do not draw boxes for each

36
 individual lens. We annotate the state as stop (red), go (green/white), off
(all lights are inactive) or unknown (when seen from behind or at a large
distance).

• Vehicle Traffic Lights: For traffic lights for vehicles we annotate the
black surrounding frame with a bounding box and the lenses/bulbs with
attributes. We do not draw boxes for each individual lens. We annotate
up to 3 active lenses with color and shape attributes for each:
– The lens colors are red, amber/yellow, green and off (lens is not active
or not visible).
– The lens shapes are circles, arrows left/straight/right, other (any
other shapes) and off (in case the lens is not visible or not active).
– Annotate the active lenses row by row (from top to bottom) and
within each row from left to right. Color1 and shape1 are assigned to
the top-most lens, color2 and shape2 to the next, etc.
– If there are more than 3 active lenses, only annotate the first 3.
– If there are less than 3 active lenses, set BOTH color and shape of
the unused lenses to “off”.
– If the color or shape of a lens are not clearly visible, set the respective
attribute to off.

37
• Multiple Traffic Light Boxes: If a traffic light post has several physi-
cally separated traffic light boxes, then annotate each box individually.

38
39
40
41
42
• Traffic Sign Extremities: The bounding box for a traffic sign should
not include its base or stand (e.g. poles, sign stand, traffic barrier, etc.) or

43
other attached traffic signs.

44
• Multiple Traffic Sign Boxes: If a traffic sign consists of two independent
traffic signs (e.g. for different lanes), put a bounding box around each of
the individual signs.

45
• Traffic Sign Back Side: If you only see the back side of a traffic sign in
the current frame, set the attribute ‘back side of a sign’ to ‘yes’. If you
know what type of sign it is, use the specific traffic sign label. Otherwise
use the ‘other traffic sign’ label. If you are annotating an entire scene and
you can only see the back side of a sign in some frames, look at the other
frames to see the sign from the front and figure out its label.

46
47
• Traffic Lights 10Px: Even when the height of a traffic light box is less
than 10 pixels, it should be annotated as long as you can see an active
traffic light lens.

48
DO NOT Label Objects if
• Do not label an object if its height is less than 10 pixels.
– Even when you are not sure about the object’s height, you should
add a box if you can confidently tell what the object is.
• Do not label if the object is less than 20% visible. The clarity and
orientation of the object do not influence its visibility. An object can have
low visibility when it is occluded or cut off by the image.
– Include: Even when the object is less than 20% visible, you should
add a box if you can confidently tell what the object is.

49
• The object is too blurry to very confidently tell what it is.

50
51
Labels
For every annotation, include one of the following labels:
1. Car or van or SUV: Vehicle designed primarily for personal use,
e.g. sedans, hatch-backs, wagons, vans, mini-vans, SUVs, jeeps, and
recreational vehicles (RVs). If the vehicle is designed to carry more than
10 people, use bus. If it is primarily designed to haul cargo, use truck.

52
53
54
55
56
57
 Top
2. Pickup truck: A pickup truck is a light duty truck with an enclosed
cab and an open or closed cargo area. A pickup truck can be intended
primarily for hauling cargo and personal use.

58
59
60
 Top
3. Front of semi truck: Tractor part of a semi trailer truck. Trailers hauled
after a semi-tractor should be labeled as trailer.

61
62
63
64
65
 Top
4. Other truck: Trucks used to haul cargo. These are bigger than pickup
trucks and not for personal use. If the truck has two parts, label them as
front_of_semi_truck and trailer.

66
67
68
69
 Top
5. Bendy bus: Buses and shuttles comprising two or more rigid sections
linked by a pivoting joint. Annotate each section of the bendy bus individ-
ually. Buses are designed to carry more than 10 people.
• Exclude: If you cannot see the pivoting joint of the bus, annotate it
as rigid bus.
• Exclude: Does not include school buses.

70
71
72
 Top
6. Rigid bus: Buses and shuttles with a single rigid section. Buses are
designed to carry more than 10 people.
• Exclude: Does not include school buses.

73
74
75
76
77
78
79
80
 Top
7. School bus: A type of bus used to transport school children. School buses
are often yellow and can be rigid or bendy buses. Buses are designed to
carry more than 10 people.

81
82
 Top
8. Construction vehicle: Vehicles primarily designed for construction. Typ-
ically very slow moving or stationary. Cranes (movable ones, not those
mounted on buildings) and extremities of construction vehicles should al-
ways be included in annotations. Trucks used to hauling rocks or building
materials are considered vehicle.truck rather than construction vehicles.
• Exclude: Trucks used to hauling rocks or building materials are
considered trucks rather than construction vehicles.

83
84
85
86
87
88
 Top
9. Motorcycle with sidecar: Any motorcycles with sidecar. Motorcycle
refers to a gasoline or electric powered 2-wheeled vehicle designed to move
rapidly (at the speed of standard cars) on the road surface. This category
includes all motorcycles, sidecars, vespas and scooters. It also includes
light 3-wheel vehicles, often with a light plastic roof and open on the sides,
that tend to be common in Asia.

89
Top

90
10. Motorcycle without sidecar: Any motorcycles without sidecar. Motor-
cycle refers to a gasoline or electric powered 2-wheeled vehicle designed
to move rapidly (at the speed of standard cars) on the road surface. This
category includes all motorcycles, sidecars, vespas and scooters. It also
includes light 3-wheel vehicles, often with a light plastic roof and open on
the sides, that tend to be common in Asia.

91
92
93
94
95
96
97
 Top
11. Bicycle: Human or electric powered 2-wheeled vehicle designed to travel
at lower speeds, typically on on road surface, sidewalks or bike paths.
• Include: If there is a rider, include the rider in the box.
• Include: If there is a passenger, include the passenger in the box.
• Exclude: If there is a pedestrian standing next to the bicycle, do NOT
include in the annotation.

98
99
100
101
 Top
12. Personal mobility vehicle: A small electric or self-propelled vehicle or
non self-propelled vehicle, e.g. skateboard, segway, or scooters, on which the
person typically travels in an upright position. Driver and (if applicable)
rider should be included in the bounding box along with the vehicle.

102
103
104
 Top
13. Bike rack: Area or device intended to park or secure the bicycles in a
row. It includes all the bikes parked in it and any empty slots that are
intended for parking bikes.
• Exclude: Bicycles that are not part of the rack should not be included
in the box, instead they should be annotated as bicycle separately.

105
106
107
108
109
 Top
14. Trailer: Any vehicle trailer, regardless of whether currently being towed
or not. E.g. for trucks, cars and bikes.
• Exclude: A vehicle towed by another vehicle should be labeled as
vehicle (not as trailer).
• Exclude: For semi-trailers (containers) label the truck itself as front
of semi truck.
• Exclude: Do not label permanently installed mobile homes as trailers,
unless these are ready to move.

110
111
112
113
114
115
116
117
 Top
15. Four wheel police vehicle: All four or more wheeled police vehicles
including cars, pickup trucks, SUVs, trucks, trailers, etc.

118
119
120
121
122
 Top
16. Police bicycle: Police bicycles.

123
 Top
17. Police motorcycle with sidecar: Police motorcycles with sidecar.
• Note: See the Motorcycle With Sidecar class for more information on
how to annotate motorcycles.

124
 Top
18. Police motorcycle without sidecar: Police motorcycles without side-
car.
• Note: See the Motorcycle Without Sidecar class for more information
on how to annotate motorcycles.

125
 Top
19. Firetruck: All types of firetrucks.

126
 Top
20. Ambulance: All types of ambulances.

127
128
129
 Top
21. Train: Any vehicle that travels on rails e.g. light rail / tram / train.
• Note: For trains, that consist of several linked units, annotate each
segment with a bounding box.

130
131
 Top
22. Other vehicle: Any vehicle not defined here, e.g. farm vehicles, military
vehicles.

132
133
134
 Top
23. Adult: Adult pedestrian that is not a police officer or construction worker.
• Include: Mannequins should be annotated as adult pedestrians. If
they are missing large parts, such as upper body, do not label them
at all.
• Include: If a pedestrian is carrying an object (children, bags, umbrellas,
tools), such object will be included in the bounding box for the
pedestrian.
• Include: If two or more pedestrians are carrying the same object, the
bounding box of only one of them will include such object.
• Exclude: If a pedestrian is pulling or pushing an object, the pedestrian
and the object should be annotated separately.
• Exclude: If a pedestrian is inside/on a vehicle, do not label it sepa-
rately.

135
136
137
138
139
140
141
 Top
24. Child: Child pedestrian. Children are usually shorter than sedans (150cm

142
or 4’11")
• Include: If a pedestrian is carrying an object (children, bags, umbrellas,
tools), such object will be included in the bounding box for the
pedestrian.
• Include: If two or more pedestrians are carrying the same object, the
bounding box of only one of them will include such object.
• Exclude: If a pedestrian is pulling or pushing an object, the pedestrian
and the object should be annotated separately.
• Exclude: If a pedestrian is inside/on a vehicle, do not label it sepa-
rately.

143
144
145
146
 Top
25. Construction worker: A human in the scene whose main purpose is
construction work.
• Include: If a construction worker is carrying an object (children, bags,
umbrellas, tools), such object will be included in the bounding box
for the pedestrian.
• Exclude: Do not label construction workers inside/on vehicles.

147
148
149
150
 Top
26. Stroller: Strollers. If a person is in the stroller, include in the annotation.
• Include: If a person is in the stroller, include it in the annotation.
• Exclude: Pedestrians pushing strollers should be labeled separately.

151
152
153
154
 Top
27. Wheelchair: Wheelchairs. If a person is in the wheelchair, include in the
annotation.
• Include: If a person is in the wheelchair, include in the annotation.
• Exclude: Pedestrians pushing wheelchairs should be labeled sepa-
rately.

155
156
157
 Top
28. Police officer: Police officer.
• Include: If a regular person (that is not a police officer or construction
worker) is directing traffic, label it as a police officer (adults do not
have the directing traffic attribute).

158
159
160
 Top
29. Small animal: Small animals that do not pose a danger to the ego vehicle,
e.g. cats, rats, small birds.

161
162
 Top
30. Large animal: Large animals that pose a danger to the ego vehicle, e.g
dogs, deer.

163
164
165
 Top
31. Traffic cone: All types of temporarily placed traffic cones. Traffic cones
are usually used to re-direct vehicle or pedestrian traffic.
• Include: Traffic cones do not necessarily have to be cone-shaped.
• Exclude: Permanently mounted traffic delineator posts are NOT
temporary traffic cones.
• Exclude: Do not label traffic cones appended to vehicles.

166
167
168
169
170
171
172
173
174
175
176
177
178
 Top
32. Temporary barrier: Temporary road barrier placed in the scene in order
to redirect traffic. Commonly used at construction sites. This includes
concrete barrier, metal barrier and water barrier. No fences. If there are
multiple barriers either connected or just placed next to each other, they

179
should be annotated separately. If a barrier is a rectangle, the front of the
barrier should be the longer side that is closest to facing the road.
• Exclude: If barriers are installed permanently, then do NOT include
them.
• Exclude: If there is a traffic sign attached to the traffic barrier, do
not include the traffic sign.
• Exclude: Tight-knit metal fences are part of manmade, not temporary
barrier.

180
181
182
183
184
 Top
33. Pushable pullable: Objects that a pedestrian may push or pull. For
example dolleys, wheel barrows, garbage bins, or shopping carts. These
objects often come with wheels to facilitate pushing or pulling them.
Typically not designed to carry humans.
• Include: Any garbage bin is considered pushable pullable regard-
less of whether it is fixed to the ground or not.
• Exclude: This does not include large containers or garbage bags.

185
186
187
188
189
190
 Top
34. Other movable object: Any other type of movable object that is too
large to be driven over safely, e.g tree branch, full trash bag, backpacks
(not carried by a human), stones, etc. These are annotated regardless of
their location (e.g. sidewalk, road).
• Include: Movable basketball hoops.
• Include: Individual stones (>5cm) that are on the driveable surface.
• Exclude: Do not annotate objects that are less than 20cm in diameter.
• Exclude: Do not annotate containers (shipping containers, storage
containers, etc.) on the ground.

191
192
193
194
195
 Top
35. Pedestrian bicycle traffic light box: Traffic light box for pedestrians
or bicycles.
• Include: Annotate the traffic light box regardless of whether you see
the front, back or side.

196
197
198
199
200
 Top
36. Vehicle traffic light box: Traffic light box for regular vehicles except
bicycles.
• Include: Annotate the traffic light box regardless of whether you see
the front, back or side.

201
202
203
204
205
206
 Top
37. Stop sign: A mandatory stop sign, typically white letters on a red sign.

207
• Exclude: Never include the pole of a traffic sign in the annotation.

208
209
 Top
38. Yield sign: A mandatory yield or give way sign, typically a yellow or red
triangle.
• Exclude: Never include the pole of a traffic sign in the annotation.

210
211
212
213
 Top
39. Turn sign: A sign close to or at an intersection that indicates the direction
vehicles on a particular lanes must or must not drive.
• Include: ‘Turn sign’ includes negative signs. (e.g. ‘No left turn’, ‘No

214
 U-turn’)
• Include: ‘Turn sign’ includes one way signs.
• Exclude: Never include the pole of a traffic sign in the annotation.

215
216
217
218
219
 Top
40. Pedestrian crossing sign: Pedestrian crossing sign, typically showing
one or more pedestrians. Pedestrian crossing signs can be placed on or off
the road.
• Exclude: Never include the pole of a traffic sign in the annotation.

220
221
222
223
224
 Top
41. Start of work zone sign: A sign that indicates a work zone is beginning
here or further ahead. Typically says ‘Works Ahead’.
• Exclude: Never include the pole of a traffic sign in the annotation.

225
226
227
 Top
42. End of work zone sign: A sign that indicates a work zone is finished.
• Exclude: Never include the pole of a traffic sign in the annotation.

228
229
230
 Top
43. Left lane closed work zone sign: A sign that indicates left lane ahead
is closed to traffic due to construction works.
• Exclude: Never include the pole of a traffic sign in the annotation.

231
232
233
 Top
44. Right lane closed work zone sign: A sign that indicates right lane
ahead is closed to traffic due to construction works.
• Exclude: Never include the pole of a traffic sign in the annotation.

234
235
236
 Top
45. Other work zone sign: Any other work zone signs. These signs typically
show black font on an orange or yellow background.
• Exclude: Never include the pole of a traffic sign in the annotation.

237
238
239
240
 Top
46. Other traffic sign: All other traffic signs (prohibitory signs, speed limits
etc.) than yield, stop, turn, construction zone and pedestrian crossing.
• Include: Carefully read the definition of turn signs. Many signs
indicate the direction that cars should travel, but are not related to
lanes at intersections.
• Include: If you see the back side of a traffic sign and cannot tell what
kind of sign it is throughout the scene, annotate it as ‘other traffic
sign’.
• Exclude: Never include the pole of a traffic sign in the annotation.
• Exclude: Street signs (that indicate the name of the road) or other
signs that are not traffic signs should not be included. In general
these are not annotated with a bounding box, and labeled as manmade
in the segmentation tasks.

241
242
243
244
245
246
247
248
249
250
251
 Top

Attributes
1. Vehicle status: Indicates the vehicle status.

252
 • Parked: Vehicle is stationary (usually for longer duration) with no
immediate intent to move. Vehicles are considered parked if there is
no driver present.
• Stopped: Vehicle is stationary, but about to start moving. If a
vehicle has a driver and zero velocity then it is often stopped. Other
cues for parked versus stopped include activated turn signals and a
running engine.
• Moving: Vehicle is in motion.
• Unknown: The vehicle state cannot be determined. This attribute
may only be selected for far away objects that are not clearly visible.

Top
2. Rider status: Indicates whether the vehicle has an adult, child or no
rider. Typically applied to bikes, motorcycles and mobility devices. A
person standing next to bike is not a rider. If there are multiple riders in
a vehicle, select the rider status for the person in control of the vehicle.
• Adult: Vehicle has an adult rider.
• Child: Vehicle has a child rider. A rider younger than 12 years old
is considered a child.

253
• No rider: Vehicle do not have a rider.

254
255
256
 Top
3. Emergency lights: Indicates whether the vehicle is flashing emergency
lights.
• Flashing: Vehicle emergency lights are flashing.
• Not flashing: Vehicle emergency lights are not flashing.

257
258
 Top
4. Is reflection: Indicates whether the object is a reflection in a window or
a mirror.
• Yes: The object is a reflection.
• No: The object is not a reflection.

259
260
Top

261
5. School bus stop sign: Indicates whether a school bus is flashing yellow
or red lights or ejecting a stop sign. If yellow and red lights are flashing,
annotate as red flashing. If lights are flashing and the stop sign is
ejected, annotate as stop sign.
• Inactive: The school bus does not show any stop signs or flashing
lights.
• Yellow flashing: The school bus lights are flashing yellow (proceed
with caution).
• Red flashing: The school bus lights are flashing red (stop no closer
than 20 feet from bus).
• Stop sign: The school bus stop sign is ejected.

262
263
264
 Top
6. Human/animal pose: Indicates the pose of a human or animal.
• Sitting: Ped/animal is sitting down or kneeling. Sitting requires the
person to touch the ground/bench with their buttocks/behind.
• Lying down: Ped/animal is lying down. We say a human is lying
down when the body of the human is horizontal to the ground.
• Standing: Ped/animal is standing up. Squatting/crouching and

265
 bending are also included here. Standing requires the person to touch
the ground/bench with only their feet.
• Moving: Ped/animal is moving (running, walking or flying).
• Unknown: Ped/animal pose cannot be discerned. This attribute
may only be selected for far away objects that are not clearly visible.

266
267
268
269
270
 Top
7. Vertical position: Indicates whether the animal is on or above the
ground.
• On ground: Object is on the ground plane.
• Off ground: Object is not on the ground plane, e.g. flying, falling,
jumping or positioned in a tree or on a vehicle.

271
272
 Top
8. Is directing traffic: Indicates whether the police officer or construction
worker is actively directing the traffic in the scene.
• Yes: The human is actively directing the traffic.
• No: The human is NOT directing the traffic.

273
274
 Top
9. Ped/bicycle traffic light lens color: Indicates whether a pedes-
trian/bicycle traffic light is on and which color it has.
• Stop: Pedestrians/bicycles are not allowed to cross. Typically indi-
cated by a red pedestrian/bicycle traffic light lens.
• Go: Pedestrians/bicycles are allowed to cross the road. Typically
indicated by a green/white pedestrian/bicycle traffic light lens or a
countdown.
• Off : No lens is active.
• Unknown: The lenses are not visible and therefore cannot be deter-
mined.
Top
10. 1st traffic light lens color: The color of the first traffic light lens.
• Red: The first active traffic light lens color is red.
• Amber: The first active traffic light lens color is amber.
• Green: The first active traffic light lens color is green.
• Off : The first active traffic light lens cannot be determined or does

275
 not exist.
Top
11. 2nd traffic light lens color: The color of the second traffic light lens.
• Red: The second active traffic light lens color is red.
• Amber: The second active traffic light lens color is amber.
• Green: The second active traffic light lens color is green.
• Off : The second active traffic light lens cannot be determined or does
not exist.
Top
12. 3rd traffic light lens color: The color of the third traffic light lens.
• Red: The third active traffic light lens color is red.
• Amber: The third active traffic light lens color is amber.
• Green: The third active traffic light lens color is green.
• Off : The third active traffic light lens cannot be determined or does
not exist.
Top
13. 1st traffic light lens shape: The shape of the first traffic light lens.
• Circle: The first active lens has a circular shape.
• Arrow left: The first active lens shows an arrow pointing left.
• Arrow straight: The first active lens shows an arrow pointing
straight ahead.
• Arrow right: The first active lens shows an arrow pointing right.
• Other shape: The first active lens shows any other lens shape.
• Off : The first active lens cannot be determined or does not exist.
Top
14. 2nd traffic light lens shape: The shape of the second traffic light lens.
• Circle: The second active lens has a circular shape.
• Arrow left: The second active lens shows an arrow pointing left.
• Arrow straight: The second active lens shows an arrow pointing
straight ahead.
• Arrow right: The second active lens shows an arrow pointing right.
• Other shape: The second active lens shows any other lens shape.
• Off : The second active lens cannot be determined or does not exist.
Top
15. 3rd traffic light lens shape: The shape of the third traffic light lens.
• Circle: The third active lens has a circular shape.
• Arrow left: The third active lens shows an arrow pointing left.

276
 • Arrow straight: The third active lens shows an arrow pointing
straight ahead.
• Arrow right: The third active lens shows an arrow pointing right.
• Other shape: The third active lens shows any other lens shape.
• Off : The third active lens cannot be determined or does not exist.
Top
16. Whether the front or back of a sign is visible: Indicates whether
only the back side of a sign is visible.
• Back: Only the back side of a sign is visible.
• Front: The front side of a sign is visible.

277
Top

278