Slides MC5

Raciocínio e Percepção Espacial: Uma abordagem
lógica
Paulo Santos
FEI - São Paulo
September 10, 2010
Paulo Santos ( FEI - São Paulo ) September 10, 2010 1 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)
6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Where is FEI?
S. Paulo, SP
FEI Campus
FEI is the largest engineering school in Brazil, with over 8,000
students
it is already a regional centre of scientific development for the
automotive industry and started investing intensively to become
also a regional centre for intelligent robotics.
Where is FEI?
S. Paulo, SP
FEI Campus
students
Where is FEI?
S. Paulo, SP
FEI Campus
students
Where is FEI?
S. Paulo, SP
FEI Campus
students
Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

reasoning about space is ubiquitous: it is necessary in situations

from tying a shoe-lace to urban traffic navigation;
automation of spatial reasoning has led to the development of a
number of application domains:
I geographical information systems (GIS)
I robotics
I commonsense reasoning
I natural language processing
I virtual world modelling and animation
I medical analysis and diagnosis systems
I computer vision

reasoning about space is ubiquitous: it is necessary in situations

from tying a shoe-lace to urban traffic navigation;
automation of spatial reasoning has led to the development of a
number of application domains:
I geographical information systems (GIS)
I robotics
I commonsense reasoning
I natural language processing
I virtual world modelling and animation
I medical analysis and diagnosis systems
I computer vision

We acquire knowledge about spatial relationships mainly in two ways:

sensory processing: intensively studied in Computer Vision and
Robotics;
being told, or reading, about spatial arrangements (high-level
reasoning). This is the kind of information processing we’re
concerned about here.
In other words, we’ll be talking about qualitative reasoning, in contrast
to numerical processing.
We will also present computer vision systems whose aim is to bridge
the gap between sensory processing and high-level reasoning:
Cognitive Vision systems.


Robotics;


Robotics;

Aim and inspiration for this tutorial
We follow the believe expressed in Takeo Kanade Keynote

Lecture (given at IJCAI 2003): it is now the time to combine
Computer Vision with Relational models and Reasoning.
This tutorial presents:
I tools and methodology of QSR;
I an overview of major QSR calculi;
I an overview of Cognitive Vision Systems;
I examples of QSR systems for Cognitive Vision.
Aim: make a brief overview of these areas, presenting the context
and foundations to kick start new projects.





Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Why a “logic-based approach”?
has a semantics
large variety of distinct logics for different kinds of reasoning
variety of inference mechanisms
provides a tool kit from which it is possible to characterise the
calculi
developing relational calculi is well understood

has a semantics
calculi

has a semantics
calculi

has a semantics
calculi

has a semantics
calculi

has a semantics
calculi

Logical Reasoning
Every man is mortal.

Socrates is a man.
Ergo: Socrates is mortal.

Logical Reasoning

Socrates is a man.

Logical Reasoning

Socrates is a man.

Logical Reasoning

Socrates is a man.

Logical Reasoning
∀x m(x) → mo(x).
m(Socrates).
Ergo: mo(Socrates).
A→B, A
MODUS PONENS: B

Logical Reasoning
∀x m(x) → mo(x).
m(Socrates).
Ergo: mo(Socrates).
A→B, A
MODUS PONENS: B

Logic programming
m(X ) :− mo(X ).
m(s).
?- m(X )
?- X = s
Computational processes: resolution, model checking, ...

Logic programming
m(X ) :− mo(X ).
m(s).
?- m(X )
?- X = s

Logic programming
m(X ) :− mo(X ).
m(s).
?- m(X )
?- X = s

Logic programming
m(X ) :− mo(X ).
m(s).
?- m(X )
?- X = s

Logical Reasoning
Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I ...
I Progol, HR, Claudien, FOIL, ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Logical Reasoning

I ...

Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993


I belief change
I vagueness
I spatial entities
14, 1993

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

What is QSR?
The formal representation of (qualitative) spatial knowledge in terms of

some basic entities and primitive relations in order to allow meaningful
and, sometimes, efficient inference methods about space.

Spatial Reasoning
“The basic stories we know best are small stories of events in space:
The wind blows clouds through the sky, a child throws a rock, a mother
pours milk into a glass, a whale swims through the water. These
stories constitute our world.” (M.Turner, The Literary Mind)

Philosophical origins of QSR
the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology


geometry (1829)


geometry (1829)


geometry (1829)


geometry (1829)


geometry (1829)


geometry (1829)


geometry (1829)


geometry (1829)

Philosophical origins of QSR: 20th century
Bertrand Russell and Alfred Whitehead: principles of logic to

phenomenological theories, describing the world as it is perceived
through sense data;
Whitehead (1920): a theory of the perceived world should have as
basic entities the very ’phenomena’: integral objects or events
I geometry becomes concerned with relationships between regions
occupied by bodies and dynamical laws with qualitative rules about
world events


through sense data;
world events


through sense data;
world events

Origins of QSR in Artificial Intelligence
phenomenological theories: closer to human reasoning

would it be possible to automate them ?
at least for some particular domains?
the construction of formal theories about the qualitative
relationships between basic spatial (“phenomenological”) theories
is the main goal of qualitative spatial reasoning in AI.
but perhaps we should talk about time before talking about

space...



space...



space...



space...



space...

Allen’s interval calculus
Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI2006

13 relations
any two intervals expressed by one and only one relation (JEPD)
temporal reasoning: derive facts about temporal intervals

Allen’s interval calculus: applications
represent activities and temporal knowledge

planning and scheduling
temporal databases
natural language understanding

Tools of temporal reasoning
Conceptual Neighbourhood Diagrams: graphs representing in

their vertices relations on some specific objects; whereas their
edges represent continuous transitions between these relations.
I Continuous transitions: in between adjacent vertices of the graph
there is no other relations that the entities in the relation’s
arguments can assume.
Composition (or transitivity) tables: given two relations on any
objects a, b, ad c (e.g., R1 (a, b) and R2 (b, c)), the composition
table entry for R1 (a, b) and R2 (b, c) gives the minimal set of
disjunctions R3 (b, c) of the possible relations between a and c.





Conceptual Neighbourhood Diagram: illustration
Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI2006

Transitivity table
< > d di ...

before (<) < no info <; o; m; d; < ...
s
after (>) no info > >; oi; mi; > ...
d; f
during (d) < > no info. d ...
contains <; o; m; di; >; oi; di; o; oi; dur; di ...
(di) fi mi; si con; =
overlaps < >; oi; di; o; d; s <; o; m ...
(o) mi; si
overlapped <; o; m; di; > oi; d; f >; oi; mi; ...
by (oi) fi di;si
meets (m) < >; oi; mi; o; d; s < ...
di; si
met-by <; o; m; di; > d <; o; m; di; ...
(mi) fi fi

Basic elements in a spatial theory
space: absolute or relative; global or local;

basic entities: points, regions, directions, bodies, shapes, things,
sense-data, ...
primitive relations: meet, between, connect, part-of, ...
I the set of relations is usually Jointly Exhaustive and Pairwise
Disjoints (JEPD)
formal tools: axiomatic (deriving axioms and proving theorems),
algebraic (encode knowledge with operators and equations),
purely logical (design of a spatial logic)

Tools of QSR
Conceptual Neighbourhood Diagrams

Composition (or transitivity) tables

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Region Connection Calculus (RCC)
many sorted, first order logic axiomatisation of spatial regions

based on a primitive binary relation about the connection between
two regions (C/2).
C(x, y ) (“x is connected to y ”), i.e., the topological closures of x
and y share at least one point
∀x C(x, x);
∀xy C(x, y ) → C(y , x);
∀xyz (C(z, x) ↔ C(z, y )) → x = y .

P(x, y ), “x is part of y”;

O(x, y ), “x overlaps y”;
PP(x, y ), “x is proper part of y”;
Pi/2 and PPi/2 are inverse relations of P/2 and PP/2, resp;
DC(x, y ), “x is disconnected from y”;
EQ(x, y ), “x is equal to y”;
PO(x, y ), “x partially overlaps y”;
EC(x, y ), “x is externally connected to y”;
TPP(x, y ), “x is tangencial proper part of y”;
NTPP(x, y ), “x is non-tangential proper part of y”;
TPPi/2 NTPPi/2 are inverses of TPP/2 and NTPP/2
{DC(x, y ), EQ(x, y ), PO(x, y ), EC(x, y ), TPP(x, y ), NTPP(x, y ),
TPPi/2, NTPPi/2 } is a JEPD set known as RCC8.
RCC: conceptual neighbourhood diagram
x TPP
y y
x
x x x NTPP
x y EQ
y y
y NTPPi
y y x
DC EC PO x TPPi

((DC(x, y ) ↔ ¬C(x, y ))). (1)

((P(x, y ) ↔ (∀z(C(z, x) → C(z, y ))))). (2)
((PP(x, y ) ↔ (P(x, y ) ∧ ¬P(y , x)))). (3)
((EQ(x, y ) ↔ (P(x, y ) ∧ P(y , x)))). (4)
((O(x, y ) ↔ (∃z(P(z, x) ∧ P(z, y ))))). (5)
((PO(x, y ) ↔ (O(x, y ) ∧ ¬P(x, y ) ∧ ¬P(y , x)))). (6)
((DR(x, y ) ↔ −O(x, y ))). (7)
((EC(x, y ) ↔ (C(x, y ) ∧ ¬O(x, y )))). (8)
((TPP(x, y ) ↔ (PP(x, y ) ∧ (∃z(EC(z, x) ∧ EC(z, y )))))). (9)
((NTPP(x, y ) ↔ (PP(x, y ) ∧ ¬(∃z(EC(z, x) ∧ EC(z, y )))))). (10)
((Pi(x, y ) ↔ P(y , x))). (11)
((PPi(x, y ) ↔ PP(y , x))). (12)
((TPPi(x, y ) ↔ TPP(y , x))). (13)
RCC: Transitivity table
DC EC PO TPP ...
DC no info. DR,PO, DR,PO, DR,PO,
PP PP PP ...
EC DR,PO, DR,PO, DR,PO, EC,PO, ...
PPi TPP PP PP
PO DR,PO, DR,PO, no info. PO,PP ...
PPi PPi
TPP DC DR DR,PO, PP ...
PP
NTPP DC DC DR,PO, NTPP ...
PP
TPPi DR,PO, EC,PO, PO,PPi PO,TPP ...
PPi PPi
NTPPi DR,PO, PO,PPi PO,PPi PO,PPi ...
PPi
EQ DC EC PO TPP ...

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Represent the relative positions between pairs of (non-overlapping)

convex bodies
Figure: Randell et a. From Images to Bodies: Modelling and Exploiting Spatial

Occlusion and Motion Parallax. IJCAI 2001

B A C
B A JC
JH PH JF JHI PHI JFI

B B A A B B
A A B B A A
B A B A A B A B
A B B A
H EH F HI EHI FI

14 relations:
I C(x, y ), is clear from;
I JC(x, y ), is just clear from;
I PH(x, y ), partially hides;
I PHI(x, y ), is partially hidden by;
I JH(x, y ), just hides;
I JHI(x, y ), is just hidden;
I H(x, y ), hides;
I HI(x, y ), is hidden by;
I EH(x, y ), exactly hides;
I EHI(x, y ), is exactly hidden;
I F (x, y ), is in front of;
I FI(x, y ), has y in front of it;
I JF (x, y ), is just in front of;
I JFI(x, y ), has y just in front of it;

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Region Occlusion Calculus (ROC)
Region Occlusion Calculus (ROC) is an extension of RCC to

represent the various possibilities of interposition (occlusion)
between arbitrary shaped bodies.
Two functions: region and image.
I region: maps a physical body to is occupancy region.
I image maps a physical body to its bi-dimensional projection from a
particular viewpoint.

Region Occlusion Calculus (ROC)
ROC primitive relations: C/2 and TotallyOccludes(x, y , ν) (“body x

totally ocludes body y from the viewpoint ν”).
∀x∀ν¬TotallyOccludes(x, x, ν)
∀x∀y ∀z∀νTotallyOccludes(x, y , ν) ∧ TotallyOccludes(y , z, ν)) →
TotallyOccludes(x, z, ν)

The following axioms introduce RCC8 in ROC:.
((TotallyOccludes(x, y , ν) ∧ P(region(z), region(y ))) →

TotallyOccludes(x, z, ν))
(TotallyOccludes(x, y , ν) → ∀z(P(region(z), region(y ))) →
¬TotallyOccludes(z, x, ν))
(TotallyOccludes(x, y , ν) → ∀z∀u(P(region(z), region(x))∧
P(region(u), region(y ))) → ¬TotallyOccludes(u, z, ν))
∃y ∃z(P(region(y ), region(x)) ∧ P(region(z), region(x))∧
TotallyOccludes(y , z, ν))
(TotallyOccludes(x, y , ν) → P(image(y , ν), image(x, ν)))

We can introduce a weaker notion of occlusion Occludes/3:
Occludes(x, y , ν) ↔ ∃z∃u(P(region(z), region(x))

∧ P(region(u), region(y ))∧
TotallyOccludes(z, u, ν))

Non occlusion, partial occlusion, mutual occlusion:
PartiallyOccludes(x, y , ν) ↔ Occludes(x, y , ν)∧

¬TotallyOccludes(x, y , ν) ∧ ¬Occludes(y , x, ν)
MutuallyOccludes(x, y , ν) ↔ Occludes(x, y , ν) ∧ Occludes(y , x, ν)
NonOccludes(x, y , ν) ↔ ¬Occludes(x, y , ν) ∧ ¬Occludes(y , x, ν)
NonOccludes(x, y , ν) → DR(image(x, ν), image(y , ν))
PartiallyOccludes(x, y , ν) →
(PO(image(x, ν), image(y , ν)) ∨ PP(image(x, ν), image(y , ν)))
MutuallyOccludes(x, y , ν) →
(PO(image(x, ν), image(y , ν)) ∨ P(image(x, ν), image(y , ν))∨
PI(image(x, ν), image(y , ν)))

NonOccludesDC MutuallyOccludesNTPP
NonOccludesEC MutuallyOccludesTPP
PartiallyOccludesPO MutuallyOccludesEQ
MutuallyOccludesPO PartiallyOccludesNTPP
PartiallyOccludesTPP TotallyOccludesEQ
PartiallyOccludesPO !1 MutuallyOccludesTPP !1
TotallyOccludesTPPI PartiallyOccludesTPP !1
!1
TotallyOccludesNTPPI PartiallyOccludesNTPP
!1
TotallyOccludesTPPI !1 MutuallyOccludesNTPP
TotallyOccludesEQ !1
TotallyOccludesNTPPI !1

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Cardinal Direction Calculus (CDC) is a formalism for reasoning

about the directions between spatial objects
9 base relations: north, south, east, west, northeast, northwest,
southeast, southwest e EQ (EQ(x, y ) means “x is at the same
direction as y”).
Main goal of CDC is to infer facts about the relative direction of
two objects A and B from the known directions between A and C
(A 6= C and B 6= C).
I E.g., from north(A, B) and northeast(B, C), the task is to calculate
the possible directions between A e C.

Cardinal Direction Calculus (CDC) is a formalism for reasoning

about the directions between spatial objects
9 base relations: north, south, east, west, northeast, northwest,
southeast, southwest e EQ (EQ(x, y ) means “x is at the same
direction as y”).
Main goal of CDC is to infer facts about the relative direction of
two objects A and B from the known directions between A and C
(A 6= C and B 6= C).
I E.g., from north(A, B) and northeast(B, C), the task is to calculate
the possible directions between A e C.

Figure: Adapted from SparQ User Manual v0.7

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

is a calculus that defines the direction of a point with respect to a

directed line segment
15 ternary relations on points
represents every distinct relation between the directions left-right
and front-back (e.g.left-front, left-back, left-line, left -perpendicular,
straight-front, ...)
motivation: qualitative description of paths







Figure: Adapted from SparQ User Manual v0.7


Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Other Calculi
size and distance: are defined either in absolute scale (whereas

the relations < or >) are introduced in the usual way), or in
relative terms, where the relative connection between three
objects is used to define the relations of proximity (near than, far
than) and equidistance . Size and distance calculi are usually
coupled with other calculi to extend their expressivity,
shapes: one of the least understood areas of QSR. In general,
shape is defined by means of a number of primitives such as
interior or boundary.

Other Calculi
default: not much has been done wrt default theories about
space; Shanahan formalises a pre-condition about spatial
occupancy of objects assuming that space is empty by default.
spatial change: base to the development of spatial change is the
work of Galton, where both time instants and intervals are
included. Two predicates are used to this end: HoldsT represents
a spatial state that is true at a time instant, whereas HoldsI
represents true stated during a time interval. From this, 8 distinct
kinds of transitions between pairs of states are defined in order to
represent the relation of two states in time.
Qualitative Trajectory Calculi, Line Segments, Dipole Calculi, and
many others!

SparQ
toolbox for QSR in applications

I reference implementations of QSR
I typical QSR procedures
I uniform interface
http://www.sfbtr8.uni-bremen.de/project/r3/sparq/

Applications of QSR
qualitative simulation of physical systems

syntax and semantics of visual programming languages
databases integration
GIS
real time event recognition
robotics

Applications of QSR

GIS
robotics

Applications of QSR

GIS
robotics

Applications of QSR

GIS
robotics

Applications of QSR

GIS
robotics

Applications of QSR

GIS
robotics

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

efficiency is inversely proportional to expressivity

first order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)
I look for tractable subsets









Tractable subsets of QSR formalism is the subject of a number of

papers. In particular (Renz e Cohn 2008) presents the following
ingredients for finding tractable subsets of such formalism:
a method to prove that a given subset if tractable
a method to suggest possible tractable subsets
in order to prove that a given set of relations is tractable, it is
sufficient to prove that the inclusion of any new relation makes the
set intractable
J. Renz, Qualitative Spatial and Temporal Reasoning: Efficient
Algorithms for Everyone, in: Proc (IJCAI-07)

Tractable subsets of QSR formalism is the subject of a number of

papers. In particular (Renz e Cohn 2008) presents the following
ingredients for finding tractable subsets of such formalism:
a method to prove that a given subset if tractable
a method to suggest possible tractable subsets
in order to prove that a given set of relations is tractable, it is
sufficient to prove that the inclusion of any new relation makes the
set intractable
J. Renz, Qualitative Spatial and Temporal Reasoning: Efficient
Algorithms for Everyone, in: Proc (IJCAI-07)

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Our knowledge of the external world
(...) what is actually given in sense is much less than most

people would naturally suppose, and (...) much of what at first
sight seems to be given is really inferred. This applies
especially in regard to our space-perceptions. For instance,
we unconsciously infer the “real” size and shape of a visible
object from its apparent size and shape, according to its
distance and our point of view. (...). Thus, the first step in the
analyses of data, namely, the discovery of what is really given
in sense, is full of difficulty.
[B. Russell (1914), Our Knowledge of
the External World, pp.75-76]
Although these ideas are at the foundations of QSR, their application

in scene analysis is still work in progress.

Our knowledge of the external world
(...) what is actually given in sense is much less than most

people would naturally suppose, and (...) much of what at first
sight seems to be given is really inferred. This applies
especially in regard to our space-perceptions. For instance,
we unconsciously infer the “real” size and shape of a visible
object from its apparent size and shape, according to its
distance and our point of view. (...). Thus, the first step in the
analyses of data, namely, the discovery of what is really given
in sense, is full of difficulty.
[B. Russell (1914), Our Knowledge of
the External World, pp.75-76]
Although these ideas are at the foundations of QSR, their application

in scene analysis is still work in progress.

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

What is CogVis?
from ECvision network
A Cognitive Vision System can achieve the four levels of generic

visual functionality: Detection, Localisation, Recognition,
Understanding (role, context, purpose)
and exhibits purposive goal-directed behaviour, is adaptive to
unforeseen changes, and can anticipate the occurrence of
objects and events. This is achieved through:
I Learning semantic knowledge (form, function and behaviours)
I Retention of knowledge (about the cognitive system, its
environment, and the relationship with the environment)
I Deliberation about objects and events, including the cognitive
system itself.

Some approaches for CogVis
VITRA system- Visual Translator: Integration of computer vision

and natural language processing
ALVEN system - textual description of all heart dynamics via
X-Ray image sequences.
Brand’s visual understanding through causal analysis
Siskind’s systems - Event classification from camera input using
force dynamics
Leeds traffic interaction - Modelling traffic interaction using learnt
qualitative spatio-temporal relations and variable length Markov
models
Leeds Protocol learning - Combining continuous and symbolic
models to learn games from observation
FEI contributions - Spatial reasoning image analysis

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Nagel’s Image Sequence Evaluation
Requirements and purpose:

exhaustive internal representation for all tasks and experimental
conditions it is expected to handle
situation: suitable intermediate representation during the
evaluation of image sequences
derive scene-specific conceptual descriptions from image
sequences, based on general assumptions about motion as the
cause of observable changes
Image sequence evaluation is based on the idea that the flow of image
sequences reflects (to some extent) the coherency of the conceptual
world.


world.


world.


world.

Intermediate levels of description:

change: any deviation of sensor signal which significantly differs
from noise
event: any change which has been defined a priori as a primitive
for the construction of more complex descriptions
verb: describe activities
history: extended sequence of activities

Some early systems
Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987),

CityTour (Herzog 1986)
usually describing traffic scenes/situations from an static viewpoint
(using motion verbs such as overtake, approach, move, beside,
recede
but also to describe a walking person (Hogg 1983), heart motion
(Tsotos 1980) and so on

Some early systems

recede

Some early systems

recede

Project Vitra: Visual Translator
long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot


descriptions
general goals:
recognition
relations
specific goals:


descriptions
general goals:
recognition
relations
specific goals:


descriptions
general goals:
recognition
relations
specific goals:


descriptions
general goals:
recognition
relations
specific goals:


descriptions
general goals:
recognition
relations
specific goals:


descriptions
general goals:
recognition
relations
specific goals:


descriptions
general goals:
recognition
relations
specific goals:


descriptions
general goals:
recognition
relations
specific goals:

Overview of the system:

from raw data, image analysis generates a geometrical
representation of the scene, with object’s locations through time
the geometric description is interpreted by the cognitive level (or
high-level scene analysis)
this high-level analysis extracts:
I spatial relations
I interesting motion events
I presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions into
utterances


I spatial relations
agents
utterances


I spatial relations
agents
utterances


I spatial relations
agents
utterances


I spatial relations
agents
utterances

Alven (1984)
natural language description of ventricular wall motion

data extracted from X-ray image sequences
knowledge organisation: ontology
Analyse pre-operative and post-operative marker films to evaluate
the efficacy of surgery
analysis using both quantitative and qualitative representations
major issues:
I understanding visual motion from image tokens over time
I reasoning about spatio-temporal relationships

Alven (1984)

major issues:

Alven (1984)

major issues:

Alven (1984)

major issues:

Alven (1984)

major issues:

Alven (1984)

major issues:

Alven

Alven’s knowledge base
Set of classes organised using relations about

IS_A: generalisation/specialisations (taxonomy)
PART _OF : part/whole (partonomy)
temporal precedence
Should satisfy:
1 motion classes should be sufficient to express the domain
2 image tokens are connected to general knowledge in the leaves of
a PART _OF hierarchy of motion concepts

Alven’s knowledge base
Set of classes organised using relations about

IS_A: generalisation/specialisations (taxonomy)
PART _OF : part/whole (partonomy)
temporal precedence
Should satisfy:
1 motion classes should be sufficient to express the domain
2 image tokens are connected to general knowledge in the leaves of
a PART _OF hierarchy of motion concepts

Alven’s control structure
extract tokens from the input signal, instantiating then in the

leaves of the PART _OF hierarchy
follows this hierarchy to activated hypotheses that are aggregates
of the input tokens
this set of hypotheses is specialised by going down one level in
the IS_A hierarchy
each hypothesis is matched with other data instances: matching
leads to further specialisations, failure leads to the selection of
other hypotheses
the best hypotheses generate a set of predictions
predictions are mapped back to the image level and are used as a
guidance to the token extraction procedure


of the input tokens
the IS_A hierarchy
other hypotheses


of the input tokens
the IS_A hierarchy
other hypotheses


of the input tokens
the IS_A hierarchy
other hypotheses


of the input tokens
the IS_A hierarchy
other hypotheses


of the input tokens
the IS_A hierarchy
other hypotheses

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection

scenes
penetration
purpose
I Analyse
I diagnosis
I prediction
I inspection

scenes
penetration
purpose
I Analyse
I diagnosis
I prediction
I inspection

scenes
penetration
purpose
I Analyse
I diagnosis
I prediction
I inspection

scenes
penetration
purpose
I Analyse
I diagnosis
I prediction
I inspection

scenes
penetration
purpose
I Analyse
I diagnosis
I prediction
I inspection

scenes
penetration
purpose
I Analyse
I diagnosis
I prediction
I inspection
Brand’s SPROCKET (1997)

Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion

Siskind’s Event classification from camera input using
force dynamics
similar to Brand’s but on image sequences

force dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN,
CONTACT
more recently: supervised learning of visual event definitions from
video

force dynamics

CONTACT
video

force dynamics

CONTACT
video

Siskind’s Leonard System

Leeds Traffic Interaction
apply QSR to dynamic scene analysis

learning events from the observation of traffic scenes
recognising these events and generate predictions
a scene is interpreted from the comparison of its temporal
development with a (previously learned) transition diagram
representing change in the domain.

CND of relative positions
Leeds Traffic Interaction
Protocol learning
A system for learning protocol behaviour from computer vision

data using ILP
Unsupervised continuous learning of perceptual categories and
unsupervised symbolic learning of protocols (as sets of Horn
clauses);
Protocol Learning
Off-the-shelf vision system

I Attention: identifies key frames with no motion preceded by a
number of frames with notion;
I Statistical classifier: assigns different classes to clusters of features
Off-the-shelf ILP system: Progol
I Generalises a set of positive only examples according to user
defined mode declarations
I Mode declarations restrict the possible form of the proposed
generalisations
Protocol Learning
Protocol Learning
Inducing axioms of ordering and equivalence, without knowing

about numbers
Building equivalence classes to cope with over clustering
sound and completeness of the agent actually playing the game
[AIJ 05]
Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion
Example of QSR from Vision: Depth Profile Calculus
How much knowledge about a robot’s environment can be from

vision alone?
How can we construct the knowledge about objects in the world
from sensor data of the robot?
Construction of a qualitative spatial reasoning (QSR) system
based on sensor data
Use abduction for sensor data assimilation and deduction for
predictions

vision alone?
predictions

vision alone?
predictions

vision alone?
predictions
Simplified environment
Spatial Reasoning about Robot Sensor Data
Attributes:
I Distance, disparity, size;
I Changes in the sensor data;
Representation:
I Depth profiles and time points;
I Displacement between regions;
I Mapping function between images and objects
Assimilating changes
Axioms of the system:
< Dynamic spatial rel > ← < desc. sensor transition >
< Dynamicspatialrel. > ← < obj − obs relation >
Depth Profiles
The model
Extract one horizontal depth profile of each scene from the visual
data;
Objects in the scenes are represented as peaks;
Axiomatise relations on the depth and size of these profiles as
well as displacements;
A theory about displacement, size and depth;

27 base relations;
Large and complex conceptual neighbourhood diagrams and
composition tables;
DPC example
Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion
making explicit the knowledge contained in cast shadows

use it to reason about the robot environment
Computer vision however has largely been filtering out cast
shadows as noise

shadows as noise

shadows as noise
Illusory Motion from Shadows
Perception of shadows
“no luminous body ever sees the shadows that it generates” [da
Vinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]
from the light source viewpoint shadows are occluded by their
casters
We model observer-caster-shadow within qualitative spatial
reasoning: ROC + an axiom about shadow :
Shadow(s, o, Scr , L) ↔ PO(r (s), r (Scr ))∧

TotallyOccludes(o, s, L)∧
¬∃o TotallyOccludes(o0 , o, L).
0
“a shadow is totally occluded by its caster from the lightsource

viewpoint”
casters

0

viewpoint”
casters

0

viewpoint”
casters

0

viewpoint”
Qualitative regions for self-localisation
L
*
4 4
3 3
2 2
o
S
1 1
In practice
Qualitative robot self-localisation

relative depth from the observation of shadows
threshold finding from qualitative regions
In practice
L
*
4 4
3 3
2 2
o
S
1 1
located(Region 1, ν, o, s) ← Is_a_Shadow(s, o) ∧
NonOccludesDC(o, s, v ) ∧ v 6= o;
NonOccludesEC(o, s, v ) ∧ v 6= o;
PartiallyOccludesPO(o, s, v ) ∧ v 6= o;
TotallyOccludesTPPI(o, s, v ) ∧ v 6= o;
Outline I
1 Preface
Where/what is FEI?
Other calculi
4 Coffee Break
20 min ?
Outline II
Foundations
Early systems

7 Conclusion
Probabilistic Logic Encoding of Spatial Domains
incorporate incomplete sensor data and domain knowledge in a

probabilistic logic setting
explore inferences about space
Traffic Scenario
Sensor model
P(OnTwoWayRoad|SensedTwoWayRoad) = 0.99
P(OnOneWayRoad|SensedOneWayRoad) = 0.99
P(DashedDivider |SensedDashedDivider ) = 0.93
P(SolidDivider |SensedDashedDivider ) = 0.07
P(DashedDivider |SensedSolidDivider ) = 0.20
P(SolidDivider |SensedSolidDivider ) = 0.80
Taxonomy of concepts
Lane = GoingUp ∪ GoingDown ,

Divider = DashedDivider SolidDivider ,
Vehicle = OnOneWayRoad ∪ OnTwoWayRoad ,
disjoint(Vehicle, Divider, Lane) ,
disjoint(GoingUp, GoingDown) ,
disjoint(DashedDivider, SolidDivider) ,
disjoint(OnOneWayRoad, OnTwoWayRoad) ,
Hard Constraints
GoingUp ⊂ ∀ri.(GoingUp ∪ ¬Lane),

GoingDown ⊂ ∀le.(GoingDown ∪ ¬Lane),
DashedDivider ⊂ ∃ri.Lane u ∃le.Lane,
SolidDivider ⊂
(¬∃ri.Lane ∪ ¬∃le.Lane) ∪ (∃cdc.GoingUp u ∃cdc.GoingDown)
OnTwoWayRoad ⊂ ∃cdc.OneWayNorth u ∃cdc.OneWaySouth
Generate a Bayesian Net out of it
Answer queries such as
On which lane are we? Argmaxli P(v : Onli) : li is the lane with
maximum probability of being the vehicle.
Which driving directions does each lane permit?
∀i : P(li : GoingDown): for each lane li, the probability of being a
GoingUp lane
Conclusion
We’ve presented a brief overview of QSR and CogVis

Discussed some early work on CogVis systems
presented some new work based mainly at UoL and FEI
Much work remains to be done!
Conclusion

Conclusion

Conclusion

Guide lines
A. Cohn, Slides of the tutorial: Knowledge Representation and

Reasoning for Computer Vision: Qualitative Spatio-temporal
reasoning
A. Cohn and J. Renz, Qualitative Spatial Representation and
Reasoning, Handbook of Knowledge Representation, 2008
B. Bennett, Logical representations for automated reasoning
about spatial relationships, PhD thesis, School of Computing,
University of Leeds, UK.
P. Santos, Raciocínio e Percepção espacial: uma abordagem
lógica. Working notes of CBA 2010 tutorial. Tutoriais do XVIII
Congresso Brasileiro de Automática, 2010.
Thanks!
This author was partially supported by:

FAPESP LogProb project:2008/03995-5
CNPq bolsa PQ

Slides MC5

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Slides MC5

Uploaded by

Copyright:

Available Formats

Raciocínio e Percepção Espacial: Uma abordagem

FEI - São Paulo

September 10, 2010

Paulo Santos ( FEI - São Paulo ) September 10, 2010 1 / 136

6 Part IV: QSR in CogVis

Paulo Santos ( FEI - São Paulo ) September 10, 2010 3 / 136

6 Part IV: QSR in CogVis

Paulo Santos ( FEI - São Paulo ) September 10, 2010 5 / 136

6 Part IV: QSR in CogVis

Paulo Santos ( FEI - São Paulo ) September 10, 2010 8 / 136

reasoning about space is ubiquitous: it is necessary in situations

Paulo Santos ( FEI - São Paulo ) September 10, 2010 9 / 136

reasoning about space is ubiquitous: it is necessary in situations

Paulo Santos ( FEI - São Paulo ) September 10, 2010 9 / 136

We acquire knowledge about spatial relationships mainly in two ways:

Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136

We acquire knowledge about spatial relationships mainly in two ways:

Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136

We acquire knowledge about spatial relationships mainly in two ways:

Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136

We follow the believe expressed in Takeo Kanade Keynote

Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136

We follow the believe expressed in Takeo Kanade Keynote

Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136

We follow the believe expressed in Takeo Kanade Keynote

Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136

6 Part IV: QSR in CogVis

Paulo Santos ( FEI - São Paulo ) September 10, 2010 13 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Every man is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Every man is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Every man is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Every man is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 16 / 136

Paulo Santos ( FEI - São Paulo ) September 10, 2010 16 / 136

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Deduction: Inferring logical truths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Deduction: Inferring logical truths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Deduction: Inferring logical truths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Deduction: Inferring logical truths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Deduction: Inferring logical truths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Deduction: Inferring logical truths