PETS

International Workshops on
Performance Evaluation of Tracking and Surveillance

James Ferryman
Computational Vision Group
Department of Computer Science
The University of Reading, UK

ETISEO, Nice, May 11-12 2005

 PETS
International Workshops on
Performance Evaluation of Tracking and Surveillance

Supported by

ETISEO, Nice, May 11-12 2005

Introduction
 Theme - Performance Evaluation of
Tracking and Surveillance
 Successful tracking of object motions key
to visual surveillance
 PETS started in Grenoble, France in 2000
as satellite workshop of FG2000
 Not a competition
 http://visualsurveillance.org
 ftp://pets.rdg.ac.uk
ETISEO, Nice, May 11-12 2005
PETS - History
 PETS’2000 was held at FG’2000, 31 March 2000,
Grenoble, France.
 PETS’2001 at CVPR’01.
 PETS’2002 at ECCV, Copenhagen, Denmark, June
1 2002.
 PETS2003 at ICVS, Graz; VS-PETS at ICCV2003
 PETS2004 at ECCV04
 WAMOP-PETS, CO, USA (Jan 05) as part of IEEE
Winter Workshop Series
 2005: VS-PETS at ICCV’05

ETISEO, Nice, May 11-12 2005

Datasets – Example – PETS2001
 Five separate sets of training and
test sequences.
 All datasets are multiview (frame
sychronised).
 Datasets were significantly more
challenging than PETS2000 (significant
lighting variation, occlusion, scene activity and use of
multiview data)

ETISEO, Nice, May 11-12 2005

Datasets

Dataset 1

Dataset 2

Dataset 3

ETISEO, Nice, May 11-12 2005

Dataset 1

ETISEO, Nice, May 11-12 2005

Dataset 2

ETISEO, Nice, May 11-12 2005

Dataset 4

ETISEO, Nice, May 11-12 2005

PETS - Prerequisites
 Tracking results reported
– should be performed using the test sequences, but the
training sequences may optionally be used if the
algorithms require it (for learning etc.)
– may be based on a single camera view of the scene, or
using multiple view data.
– can be based on the entire test sequence, or a portion of
it; the images may be converted to any other format
and/or subsampled.
– results must be submitted in XML format.

ETISEO, Nice, May 11-12 2005

PETS – Workshop Overview

 XX contributed papers
 ~3 sessions: e.g. appearance-based
tracking, people and vehicle tracking, multiview
tracking
 Y invited speakers
 Demonstration session
 Overall evaluation and discussion

ETISEO, Nice, May 11-12 2005

Quantitative PE - XML
 XML provides mechanism of setting up
“syntax” file in form of schema
 Schema used to automatically validate
object tracking results
 For PETS’2001, two schemas were used:
– low-level tracking results
– high-level surveillance
(understanding object motions and
interactions)
ETISEO, Nice, May 11-12 2005
Quantitative PE - XML
<?xml version='1.0' encoding='ISO-8859-1' ?>
<!-- -->
<!-- Example file for visual surveillance reporting -->
<!-- -> scene understanding with multiple cameras. -->

<!-- Edited by PETS2001.Reading.JMF (J.M.Ferryman@reading.ac.uk) -->

<people_tracker xmlns="http://www.cvg.cs.reading.ac.uk/PETS2001"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://www.cvg.cs.reading.ac.uk/PETS2001

http://www.cvg.cs.reading.ac.uk/PETS2001/XML/surveillance.xsd">

<!-- this is a comment ... add more as appropriate -->

<header>
<recording site="PETS2001 (Reading)" session="1" date="01/06/01">
<list_cameras num_cameras="2">
<camera camera_id="1"/>
<camera camera_id="2"/>
</list_cameras>
</recording> ETISEO, Nice, May 11-12 2005
Quantitative PE - XML
<video start_frame="1" end_frame="1450" step="2" fps="25"/>
<!-- step: stepping to read list of processed frames below -->
<!-- (for (i=start; i<=end; i+=step); i++) -->
<!-- fps: frames per second of original video -->

<image xdim="768" ydim="576" colour="1"/>

<!-- dimensions of video images and whether it is in colour (0,1) -->

<software name="Reading People Tracker" platform="Linux" version="0.03"

run_date="12/07/00">
<!-- information about the software this file originates from -->

<object_detector name="Reading People Tracker" platform="Linux"

version="0.03" run_date="08/06/01"/>

</software>

</header>

ETISEO, Nice, May 11-12 2005

Quantitative PE - XML
<sequence>
<!-- the actual data: a sequence of one or more frames. -->

<frame id="2" num_targets="1">

<target id="8" start_frame="1" end_frame ="2">

<!-- a "target" is any object which moves or may move, usually a
person, group of people, or a vehicle.
The target's id is GLOBAL to all the cameras defined in
"list_cameras" -->
<!-- start_frame and end_frame indicate when the target has been
tracked. end_frame may be unknown because it is in the future;
in this case the longest known time where the object was
tracked will be given -->

<track status="4" location="0" speed="200" trajectory="1" t_confidence="0.5"

num_parents="2">
<!-- track is part of a graph representing tracks of all targets -->
<!-- the status of a graph node explains how the node of current
target has been created or tracked. The following values may
be used and added together as appropriate:

0 :
default value, already tracked
1 :
new track (id did not exist before)
2 :
re-appearing object (id copied from last occurrence)
4 :
merging (more than one parent in graph)
8 :
splitting (at least one parent in graph has more than 1 child)
16 :
lost (object NOT found in current image, given position etc
are estimates (if available) or previous values)
32 : out of field of view (tracked object not "visible" as per
definition (see elsewhere)) -->
D1C1: XML output

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 1

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1 - 2

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1 C1- 3

ETISEO, Nice, May 11-12 2005

D1C1: XML output 1

ETISEO, Nice, May 11-12 2005

D1C1: XML output 2

ETISEO, Nice, May 11-12 2005

D1C1: XML output 3

ETISEO, Nice, May 11-12 2005

D1C1: XML output 4

ETISEO, Nice, May 11-12 2005

D1C1: XML output 5

ETISEO, Nice, May 11-12 2005

Performance Evaluation
 Evaluation of surveillance system can be
judged in a number of ways:
– object detection lag
– object centroid position error
– object area error
– track incompleteness factor
– accuracy of semantics of interaction
– object identity error
• maintenance of identity through occlusion
• …
ETISEO, Nice, May 11-12 2005
Image Format Processing Speed Processor
384 x 288 10 fps
(JPEG)
384 x 288 25-30 fps 600 MHz
Dual PIII
850 MHz
384 x 288 (AVI)
384 x 288 5 fps
768 x 576 5 fps
384 x 288 6.25 fps 800 MHz PIII
320 x 240 29.97 fps 1 GHz PIV
5fps 1.7 GHz PIV
768 X 576 5fps

ETISEO, Nice, May 11-12 2005

Discussion
 Evaluation criteria are application
dependent
 Training data – required or not?
 representative examples
 how much?
 Semantics of XML schema
 Ground truth
 difficult to obtain
 automatic evaluation - how?
ETISEO, Nice, May 11-12 2005
PETS Evaluation
 +ve: “Mindset” – engaging the community –
change of culture
 +ve: Repository of data (PETS01 most frequently
accessed)
 +ve: Discussion/presentation of methodologies,
metrics, tools …
 +ve: Filters through to conferences/published
literature
 -ve: For workshop, choice of dataset(s)
 + annotation
 -ve: More quantitative evaluation
ETISEO, Nice, May 11-12 2005 needed
PETS, ETISEO and the future …
 Online web-based evaluation service
 (Semi-)automatic validation of XML against ground
truth
 Repository of algorithms (incl. “strawman”), and
tabulated results (rank?)
 Methodology for evaluation
 Metrics
 More challenging datasets (e.g. multiview)
 Live workshop sessions on “unseen” data
 Expectation that ETISEO will support PETS
ETISEO, Nice, May 11-12 2005
PETS’05
 ICCV ’05, Beijing, China
 15-16 October 2005
 http://www.cbsr.ia.ac.cn/conferences
/VS-PETS-2005
 http://visualsurveillance.org/PETS2005

ETISEO, Nice, May 11-12 2005