[Fashion] Triplet Labeling with Boxes

Contents
● Summary of Task
● Workflow
● Taxonomy Table
● Correctly Labeled Examples
● Annotation Rules
● Attributes Definitions
● Common Errors
● Edge Cases
● Appendix:
○ Decision Tree: How to manage cases where you’re unsure if an object in the
image to label exists in at least 1 context image (e.g. due to low visibility)?

Summary of Task
1. Compare prelabeled clothing items in the main image (“image to label”) to unlabeled
clothing items in the “context images”

2. For each prelabeled bounding box in the “image to label”, answer 1-2 attribute
questions:

Back to top

Important Updates:
● 11/29/22: Included decision tree explaining how to manage low visibility cases

Correctly Labeled Examples

Taxonomy Table
Attribute Option

Item exists in at least 1 True

context image?
False

Item is the same color(s) as True

the item in at least 1
'context' image? False

Workflow

STEP 1 Start by looking at the “context

images”
 STEP 2 Compare the image-to-label image to
the non-labeled context images.

Answer 1-2 attribute questions for each

prelabeled bounding box.

1. Determine if the item exists in at least

1 context image?

2. If the item exists in at least 1 context

image, continue & label whether the
item is the same color(s) as the item
in at least 1 'context' image?

Note, there may be more than 1 item for sale

in a given image.

STEP 3 Review the task for overall accuracy, check the common errors and edge cases before
submitting

Reminder: Check the context images before answering the attribute questions!

Back to top

Annotation Rules

Rule Description Example

Incorrect label If the label surrounding a

clothing item is incorrect (e.g. a
“pants” bounding box
encompasses a shirt), ignore
the label name and follow the
below rules:

1. If the item seen in the

image exists in at least 1
bounding box, set Item
exists in at least 1 context
image? = TRUE

2. If the item seen in the

context image does NOT
exist in at least 1 image, set
Item exists in at least 1
context image? = FALSE
 Please also see edge case for
rompers

Low Visibility: Rule #1: If there is enough Rule #1: The unique material, color, straps etc. provide
information about the garment enough context for us to confidently determine that the
Please see a for you to confidently determine item exists in at least 1 context image
supporting decision >> Correct Answer: Set item exists in at least 1
that an item in the
tree in the context image? To TRUE.
image-to-label exists in at least
APPENDIX, which 1 context image, select TRUE.
includes additional
examples Tip: Look for unique features of
the item (e.g hems, rips,
stitching, embroidery, sleeve
style / type, patterns, materials
etc), which often indicate if 2
items are the same.

Rule #2: Rule #2: Though the pants are highly truncated, the
If the image-to-label and image-to-label and context images appear to be different
context image(s) are different angles of the same scene/same model appears to be
angles of the same wearing the same outfit in all images.
scene/photoshoot OR if the >> Correct Answer: Set item exists in at least 1
item(s) are featured on the context image? To TRUE.
same model/mannequin

AND

The items in all images are

otherwise the same, defer to
selecting TRUE for all items
 Rule #3: If neither of the above Rule #3: We cannot confidently determine if the item in
is true OR if you cannot make the image to label exists in at least 1 context image.
a confident decision, select >> Correct Answer: Set item exists in at least 1
FALSE. context image? To FALSE.

Back to top

Attributes Definitions

Attribute Option / Description Example

Item exists in at True - if the garment is
least 1 context pictured in at LEAST 1
image? context image

See annotation
rules

False - if the garment is

not pictured in at least 1
context image

Select FALSE if either of

the following is true:
● It is not clear whether
or not the context
images contain the
garment (e.g. due to
low visibility of the item
in th image to label)
● You are deciding
between TRUE and
FALSE but are not
sure
Item is the same True - if the garment is the
color(s) as the same color as the
item in at least 1 corresponding garment in
'context' image? AT LEAST 1 context
image
Tip: Toggle the
bounding boxes - SHADE: Different
on / off in the shades of the same color
image to label to (e.g. light and dark blue)
get a clear idea of should be labeled FALSE.
the color of the Select TRUE only if the
item inside the colors are exactly the
image to label same.

- LIGHTING: If the color of

the item appears to be
distorted due to lighting
alone, err on the side of
selecting true - see
example

Lighting
 False - if the garment is a
different color as the
corresponding garment in
ALL of the context images

Select FALSE for different

shades of the same color
(ie light and dark blue)

If you are deciding

between TRUE and
FALSE but are not sure,
err on the side of selecting
FALSE

Back to top

Common Errors

Error Details Example

Incorrect selection If there’s a bounding box INCORRECT

for Item exists in at around an item in the
least 1 context image-to-label but the item
image? does NOT exist in at least
one context image, set
Item exists in at least 1
context image? = False
 CORRECT

Carefully compare the Description: In the below example, the CORRECT

item in the image to label answer is TRUE. We have enough context on the
to items in the context jeans (i.e. the rips) to confirm the item exists in at
images to accurately least 1 context image. Because the image to label is a
answer the question mirror selfie, the rip on the knee appears on the
opposite side of the rip on the item in the context
Look for subtle details image; however, we can confirm the rips exist on the
(e.g. similar rips, cuts, same side in real life since one is a mirror selfie while
shape, pattern, fit, material the other is not.
etc) to confirm if an item
exists in at least 1 context Image to label:
image
 Context Image:

Back to top

Edge Cases

Description How to handle it Example

How to answer “item Even if an item is labeled
exists in at least 1 incorrectly (ie the bounding
context image” if the box has the incorrect label
incorrect label name is name), we should STILL
used ANSWER whether or not
the boxed object exists in at
least 1 context image
In other words, you can
IGNORE the label name
and focus only on the boxes
object(s) in the image
Example
In the example, the glasses were labeled pants. This
DOES NOT MATTER, we still respond TRUE that the
boxed item DOES exist in the context image.
Whatever the item is in the box should be what we’re
confirming exists in the context images
Answer Key
“Pants”
Does Item Exist? TRUE
Pay attention to subtle
details when determining
if an item in the
image-to-label exists in at
least 1 context image
Please see the appendix
for a supporting decision
tree
When determining if an item Example
exists in at least 1 context
image, carefully compare REMEMBER IF YOU ARE UNSURE IF THE ITEM IS
the item in the THE SAME→ MARK FALSE
image-to-label to items in
the context images, paying The red arrows in the context image show the following:
close attention to the Sleeves - NOT A MATCH
following details Pockets - NOT A MATCH
Style (i.e. observe Hemlines - NOT A MATCH
different types of
sleeve styles) So we would answer the first question - “item exists in at
Length (e.g. pant least 1 context image” - as FALSE
length, sleeve length
etc) Answer Key
Material (e.g. leather Top
vs velvet) Does Item Exist? FALSE
Color
Pattern
Collar type
Pockets
Hemline
Fit (ie baggy, loose)
If the image-to-label is If it is obvious that the
from the SAME image-to-label and the
PHOTOSHOOT you can context images are from the
assume True for Attribute same photoshoot, it is
1 more often with less SAFER to assume the
visibility item/color are the same
without seeing the full item
If you can’t tell if the Always SELECT FALSE
items are the same for Attribute 1 [“Does Item
Exist in context images”] if
you ARE NOT 100% SURE
it’s the same Item!
Example:
In the example below, we only see the back of the shirt in
the image to label, but it is obvious the photos are from
the same photoshoot [hanger, lighting, shirt etc] so without
seeing the design on the front, it is safe to assume they
are the same shirt / same color
Answer key:
Top
Does Item Exist? True
Same color? True
Example 1:
In the example below, since the images are of different
photo shoots/models, we must be SURE the clothing item
is the same
We CAN tell the Top is the same because of the design
similarities [neck wrap and crop top style]
Answer key:
Shorts - False
Top - Item exist: True → Same color: True
Left vs Right Shoe If the image to label
features a left shoe and the
context image(s) contains a
corresponding right shoe, or
vice versa:

Set Item exists in at least 1

context image? = TRUE as
long as the shoes belong to
the same pair.

Rompers Answer TRUE for Attribute

1 [Does the item exist in
context images?] even if
the label on the clothing
in the image-to-label is
wrong

Example: If “top” and

“shorts” or “top” and “skirts”
are used as the prelabeled
bounding boxes in the
image to label, but you can
confirm the corresponding
item in the context image(s)
is clearly a
romper/jumpsuit/one, piece,
set Item exists in at least 1
context image? to YES for
BOTH the tops and
shorts/skirts
Back to top
 Appendix

Decision Tree: How to manage cases where you’re unsure if an object in the image to label
exists in at least 1 context image (e.g. due to low visibility)?

See associated examples in the table below.

Step Choice “Image to Label” Context Image Explanation

STEP 1 YES Item in question: Shirt Item in question: Shirt It’s obvious the shirt
exists in at least 1
Do you have context image.
HIGH
confidence that Set ‘item exists in at
the item exists least 1 context
in at least 1 image?’ to TRUE
context image?

NO Item in question: Pants Item in question: Pants Since the pants are
e.g. low highly truncated, we
visibility don’t have HIGH
gives you confidence that the
doubts item exists in at least
1 context image.

Do NOT select an
answer yet. Instead,
 move to STEP 2

STEP 2 YES Item in question: Pants Item in question: Pants The image-to-label
and context image(s)
Are the are from the exact
image-to-label same photoshoot
and context and feature the same
image(s) from model/mannequin
the exact same ALL images
scene/photosh
oot and/OR is Set ‘item exists in at
the same least 1 context
model/manneq image?’ to TRUE for
uin featured in the pants
ALL images
AND the items
in all images
are otherwise
the same

NO Item in question: Shirt Item in question: Shirt The image-to-label

and context image(s)
are NOT from the the
exact same
scene/photoshoot
and have different
models

Do not select an
answer yet. Instead,
move to STEP 3
STEP 3 YES Item in question: Shirt Item in question: Shirt The sleeves and
cuffs are clearly
Have you different.
compared
SUBTLE Now that you have
details of the this information,
item in the move to STEP 4.
context image
and items in
the image to
label? Subtle
details to
inspect include:

- Style

- Length

- Material

- Color

- Pattern

- Collar type

- Pockets

- Hemline

- Fit (ie baggy,

loose)

STEP 4 NO Item in question: Shirt Item in question: Shirt Since the sleeves
and cuffs are clearly
Do you now different, we have
have HIGH HIGH confidence
confidence that these are not the
the item exists same item.
in at least 1
context image Set item exists in at
after checking least 1 context image
subtle details? to FALSE
 YES Item in question: Shorts and shirt Item in question: Shorts and shirt If you look closely at
the pink shirt and
shorts, it features the
same white writing as
the writing on the
shorts and shirt in the
context image. This
subtle detail indicates
the items are the
same.

Set item exists in at

least 1 context image
to TRUE

Remember, if you’re
not sure / don’t have
high confidence,
select false.

Back to top