1

PerIormance Evaluation
oI InIormation Retrieval Systems
Many slides in this section are adapted
Irom ProI. Joydeep Ghosh (UT ECE) who
in turn adapted them Irom ProI. Dik Lee
(Univ. oI Science and Tech, Hong Kong)
2
hy System Evaluation?
· There are many retrieval models/ algorithms/
systems, which one is the best?
· hat is the best component Ior:
Ranking Iunction (dot-product, cosine, .)
Term selection (stopword removal, stemming.)
Term weighting (TF, TF-IDF,.)
· How Iar down the ranked list will a user need
to look to Iind some/all relevant documents?
3
DiIIiculties in Evaluating IR Systems
· EIIectiveness is related to the relevancv oI retrieved
items.
· Relevancy is not typically binary but continuous.
· Even iI relevancy is binary, it can be a diIIicult
iudgment to make.
· Relevancy, Irom a human standpoint, is:
Subiective: Depends upon a speciIic user's iudgment.
Situational: Relates to user's current needs.
Cognitive: Depends on human perception and behavior.
Dynamic: Changes over time.
1
Human Labeled Corpora
(Gold Standard)
· Start with a corpus oI documents.
· Collect a set oI queries Ior this corpus.
· Have one or more human experts
exhaustively label the relevant documents
Ior each query.
· Typically assumes binary relevance
iudgments.
· Requires considerable human eIIort Ior
large document/query corpora.
5
documents relevant of number Total
retrieved documents relevant of Number
recall
retrieved documents of number Total
retrieved documents relevant of Number
precision
Relevant
documents
Retrieved
documents
Entire document
collection
retrieved &
relevant
not retrieved but
relevant
retrieved &
irrelevant
Not retrieved &
irrelevant
retrieved not retrieved
r
e
l
e
v
a
n
t
i
r
r
e
l
e
v
a
n
t
Precision and Recall
ê
Precision and Recall
· Precision
The ability to retrieve top-ranked documents
that are mostly relevant.
· Recall
The ability oI the search to Iind all oI the
relevant items in the corpus.
Z
Determining Recall is DiIIicult
· Total number oI relevant items is
sometimes not available:
Sample across the database and perIorm
relevance iudgment on these items.
Apply diIIerent retrieval algorithms to the same
database Ior the same query. The aggregate oI
relevant items is taken as the total relevant set.
8
Trade-oII between Recall and Precision
1
0
1
Recall
P
r
e
c
i
s
i
o
n
The ideal
Returns relevant documents but
misses many useIul ones too
Returns most relevant
documents but includes
lots oI iunk
9
Computing Recall/Precision Points
· For a given query, produce the ranked list oI
retrievals.
· Adiusting a threshold on this ranked list produces
diIIerent sets oI retrieved documents, and
thereIore diIIerent recall/precision measures.
· Mark each document in the ranked list that is
relevant according to the gold standard.
· Compute a recall/precision pair Ior each position
in the ranked list that contains a relevant
document.
10
R÷3/6÷0.5; P÷3/4÷0.75
Computing Recall/Precision Points:
Example 1
n doc # reIevant
1 588 x
2 589 x
3 576
4 590 x
5 986
6 592 x
7 984
8 988
9 578
10 985
11 103
12 591
13 772 x
14 990
Let total # oI relevant docs ÷ 6
Check each new recall point:
R÷1/6÷0.167; P÷1/1÷1
R÷2/6÷0.333; P÷2/2÷1
R÷5/6÷0.833; p÷5/13÷0.38
R÷4/6÷0.667; P÷4/6÷0.667
Missing one
relevant document.
Never reach
100° recall
11
R÷3/6÷0.5; P÷3/5÷0.6
Computing Recall/Precision Points:
Example 2
n doc # reIevant
1 588 x
2 576
3 589 x
4 342
5 590 x
6 717
7 984
8 772 x
9 321 x
10 498
11 113
12 628
13 772
14 592 x
Let total # oI relevant docs ÷ 6
Check each new recall point:
R÷1/6÷0.167; P÷1/1÷1
R÷2/6÷0.333; P÷2/3÷0.667
R÷6/6÷1.0; p÷6/14÷0.429
R÷4/6÷0.667; P÷4/8÷0.5
R÷5/6÷0.833; P÷5/9÷0.556
12
Interpolating a Recall/Precision Curve
· Interpolate a precision value Ior each standard
recall level:
r
i
Z0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}
r
0
÷ 0.0, r
1
÷ 0.1, ., r
10
÷1.0
· The interpolated precision at the -th standard
recall level is the maximum known precision at
any recall level between the -th and ( ¹ 1)-th
level:
) ( max ) (
1
r P r P

r r r

A A

13
Interpolating a Recall/Precision Curve:
Example 1
0.4 0.8
1.0
0.8
0.6
0.4
0.2
0.2
1.0
0.6
Recall
P
r
e
c
i
s
i
o
n
11
Interpolating a Recall/Precision Curve:
Example 2
0.4 0.8
1.0
0.8
0.6
0.4
0.2
0.2
1.0
0.6
Recall
P
r
e
c
i
s
i
o
n
15
Average Recall/Precision Curve
· Typically average perIormance over a large
8et oI queries.
· Compute average precision at each standard
recall level across all queries.
· Plot average precision/recall curves to
evaluate overall system perIormance on a
document/query corpus.

Compare Two or More Systems
· The curve closest to the upper right-hand
corner oI the graph indicates the best
perIormance
0
0.2
0.4
0.6
0.8
1
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
#0.,
!
7
0
.

8

4
3
NoStem Stem
1Z
Sample RP Curve Ior CF Corpus
18
R- Precision
· Precision at the R-th position in the ranking
oI results Ior a query that has R relevant
documents.
n doc # reIevant
1 588 x
2 589 x
3 576
4 590 x
5 986
6 592 x
7 984
8 988
9 578
10 985
11 103
12 591
13 772 x
14 990
R ÷ # oI relevant docs ÷ 6
R-Precision ÷ 4/6 ÷ 0.67
19
F-Measure
· One measure oI perIormance that takes into
account both recall and precision.
· Harmonic mean oI recall and precision:
· Compared to arithmetic mean, both need to
be high Ior harmonic mean to be high.
P R
R P
PR
F
1 1
2 2

20
E Measure (parameterized F Measure)
· A variant oI F measure that allows weighting
emphasis on precision over recall:
· Value oI . controls trade-oII:
. ÷ 1: Equally weight precision and recall (E÷F).
. ~ 1: eight recall more.
. · 1: eight precision more.
P R
R P
PR
E
1
2
2
2
2
) 1 ( ) 1 (

.
.
.
.
Mean Average Precision
(MAP)
· verage Precision: Average oI the precision
values at the points at which each relevant
document is retrieved.
Ex1: (1 ¹ 1 ¹ 0.75 ¹ 0.667 ¹ 0.38 ¹ 0)/6 ÷ 0.633
Ex2: (1 ¹ 0.667 ¹ 0.6 ¹ 0.5 ¹ 0.556 ¹ 0.429)/6 ÷ 0.625
· Mean verage Precision: Average oI the
average precision value Ior a set oI queries.
21
22
Fallout Rate
· Problems with both precision and recall:
Number oI irrelevant documents in the
collection is not taken into account.
Recall is undeIined when there is no
relevant document in the collection.
Precision is undeIined when no document is
retrieved.
collection the in items t nonrelevan of no. total
retrieved items t nonrelevan of no.
Fallout
23
Issues with Relevance
· arginal Relevance Do later documents in the
ranking add new inIormation beyond what is
already given in higher documents.
Choice oI retrieved set should encourage diversity and
novelty.
· 4verage Rati4: The proportion oI relevant items
retrieved out oI the total relevant documents
n4n to a user prior to the search.
Relevant when the user wants to locate documents
which they have seen beIore (e.g., the budget report Ior
Year 2000).
21
Other Factors to Consider
· &ser effort: ork required Irom the user in
Iormulating queries, conducting the search, and
screening the output.
· Response time: Time interval between receipt oI a
user query and the presentation oI system responses.
· Form of presentation: InIluence oI search output
Iormat on the user`s ability to utilize the retrieved
materials.
· Collection coverage: Extent to which any/all
relevant items are included in the document corpus.
A/B Testing in a Deployed System
· Can exploit an existing user base to provide
useIul Ieedback.
· Randomly send a small Iraction (110°) oI
incoming users to a variant oI the system
that includes a single change.
· Judge eIIectiveness by measuring change in
clicthr4:gh: The percentage oI users that
click on the top result (or any result on the
Iirst page).
25

Experimental Setup Ior Benchmarking
· nalvtical perIormance evaluation is diIIicult Ior
document retrieval systems because many
characteristics such as relevance, distribution oI
words, etc., are diIIicult to describe with
mathematical precision.
· PerIormance is measured by benchmaring. That
is, the retrieval eIIectiveness oI a system is
evaluated on a given set of documents, queries, and
relevance udgments.
· PerIormance data is valid only Ior the environment
under which the system is evaluated.
2Z
Benchmarks
· A benchmark collection contains:
A set oI standard documents and queries/topics.
A list oI relevant documents Ior each query.
· Standard collections Ior traditional IR:
Smart collection: Itp://Itp.cs.cornell.edu/pub/smart
TREC: http://trec.nist.gov/
Standard
document
collection
Standard
queries
lgorithm
under test
Evaluation
Standard
result
Retrieved
result
Precision
and recall
28
Benchmarking The Problems
· PerIormance data is valid only Ior a
particular benchmark.
· Building a benchmark corpus is a diIIicult
task.
· Benchmark web corpora are iust starting to
be developed.
· Benchmark Ioreign-language corpora are
iust starting to be developed.
29
· Previous experiments were based on the SMART
collection which is Iairly small.
(Itp://Itp.cs.cornell.edu/pub/smart)
Collection Number OI Number OI Raw Size
Name Documents Queries (Mbytes)
CACM 3,204 64 1.5
CISI 1,460 112 1.3
CRAN 1,400 225 1.6
MED 1,033 30 1.1
TIME 425 83 1.5
· DiIIerent researchers used diIIerent test collections
and evaluation techniques.
Early Test Collections
30
The TREC Benchmark
· TREC: Text REtrieval ConIerence (http://trec.nist.gov/)
Originated Irom the TIPSTER program sponsored by
DeIense Advanced Research Proiects Agency (DARPA).
· Became an annual conIerence in 1992, co-sponsored by the
National Institute oI Standards and Technology (NIST) and
DARPA.
· Participants are given parts oI a standard set oI documents
and TOPICS (Irom which queries have to be derived) in
diIIerent stages Ior training and testing.
· Participants submit the P/R values Ior the Iinal document
and query corpus and present their results at the conIerence.
31
The TREC Obiectives
· Provide a common ground Ior comparing diIIerent IR
techniques.
Same set oI documents and queries, and same evaluation method.
· Sharing oI resources and experiences in developing the
benchmark.
ith maior sponsorship Irom government to develop large
benchmark collections.
· Encourage participation Irom industry and academia.
· Development oI new evaluation techniques, particularly Ior
new applications.
Retrieval, routing/Iiltering, non-English collection, web-based
collection, question answering.
32
TREC Advantages
· Large scale (compared to a Iew MB in the SMART
Collection).
· Relevance iudgments provided.
· Under continuous development with support Irom
the U.S. Government.
· ide participation:
TREC 1: 28 papers 360 pages.
TREC 4: 37 papers 560 pages.
TREC 7: 61 papers 600 pages.
TREC 8: 74 papers.
33
TREC Tasks
· Ad hoc: New questions are being asked on a static
set oI data.
· Routing: Same questions are being asked, but new
inIormation is being searched. (news clipping,
library proIiling).
· New tasks added aIter TREC 5 - Interactive,
multilingual, natural language, multiple database
merging, Iiltering, very large corpus (20 GB, 7.5
million documents), question answering.
31
Characteristics oI the TREC Collection
· Both long and short documents (Irom a Iew
hundred to over one thousand unique terms in a
document).
· Test documents consist oI:
SJ all Street Journal articles (1986-1992) 550 M
AP Associate Press Newswire (1989) 514 M
ZIFF Computer Select Disks (ZiII-Davis Publishing) 493 M
FR Federal Register 469 M
DOE Abstracts Irom Department oI Energy reports 190 M
35
Sample Document (with SGML)
·DOC~
·DOCNO~ SJ870324-0001 ·/DOCNO~
·HL~ John Blair Is Near Accord To Sell Unit, Sources Say ·/HL~
·DD~ 03/24/87·/DD~
·SO~ ALL STREET JOURNAL (J) ·/SO~
·IN~ REL TENDER OFFERS, MERGERS, ACQUISITIONS (TNM)
MARKETING, ADVERTISING (MKT) TELECOMMUNICATIONS,
BROADCASTING, TELEPHONE, TELEGRAPH (TEL) ·/IN~
·DATELINE~ NE YORK ·/DATELINE~
·TEXT~
John Blair & Co. is close to an agreement to sell its TV station
advertising representation operation and program production unit to an
investor group led by James H. RosenIield, a Iormer CBS Inc. executive,
industry sources said. Industry sources put the value oI the proposed
acquisition at more than $100 million. ...
·/TEXT~
·/DOC~

Sample Query (with SGML)
·top~
·head~ Tipster Topic Description
·num~ Number: 066
·dom~ Domain: Science and Technology
·title~ Topic: Natural Language Processing
·desc~ Description: Document will identiIy a type oI natural language
processing technology which is being developed or marketed in the U.S.
·narr~ Narrative: A relevant document will identiIy a company or institution
developing or marketing a natural language processing technology,
identiIy the technology, and identiIy one oI more Ieatures oI the
company's product.
·con~ Concept(s): 1. natural language processing ;2. translation, language,
dictionary
·Iac~ Factor(s):
·nat~ Nationality: U.S.·/nat~
·/Iac~
·deI~ DeIinitions(s):
·/top~
3Z
TREC Properties
· Both documents and queries contain many
diIIerent kinds oI inIormation (Iields).
· Generation oI the Iormal queries (Boolean,
Vector Space, etc.) is the responsibility oI the
system.
A system may be very good at querying and
ranking, but iI it generates poor queries Irom the
topic, its Iinal P/R would be poor.
38
Evaluation
· Summary table statistics: Number oI topics, number
oI documents retrieved, number oI relevant
documents.
· Recall-precision average: Average precision at 11
recall levels (0 to 1 at 0.1 increments).
· Document level average: Average precision when 5,
10, .., 100, . 1000 documents are retrieved.
· Average precision histogram: DiIIerence oI the R-
precision Ior each topic and the average R-precision
oI all systems Ior that topic.
39
GOV2 eb Corpus
· Recent web-based gold-standard corpus
assembled by NIST.
· 25 million web pages in the .gov domain
High proportion oI .gov pages in 2004
· Total oI 426 GB oI text
· Set oI 50 relevance-iudged queries
10
11
Cystic Fibrosis (CF) Collection
· 1,239 abstracts oI medical iournal articles
on CF.
· 100 inIormation requests (queries) in the
Iorm oI complete English questions.
· Relevant documents determined and rated
by 4 separate medical experts on 0-2 scale:
0: Not relevant.
1: Marginally relevant.
2: Highly relevant.
12
CF Document Fields
· MEDLINE access number
· Author
· Title
· Source
· Maior subiects
· Minor subiects
· Abstract (or extract)
· ReIerences to other documents
· Citations to this document
13
Sample CF Document
AN 74154352
AU Burnell-R-H. Robertson-E-F.
TI Cystic Iibrosis in a patient with Kartagener syndrome.
SO Am-J-Dis-Child. 1974 May. 127(5). P 746-7.
MJ CYSTIC-FIBROSIS: co. KARTAGENER-TRIAD: co.
MN CASE-REPORT. CHLORIDES: an. HUMAN. INFANT. LUNG: ra. MALE.
SITUS-INVERSUS: co, ra. SODIUM: an. SEAT: an.
AB A patient exhibited the Ieatures oI both Kartagener syndrome and
cystic Iibrosis. At most, to the authors' knowledge, this
represents the third such report oI the combination. Cystic
Iibrosis should be excluded beIore a diagnosis oI Kartagener
syndrome is made.
RF 001 KARTAGENER M BEITR KLIN TUBERK 83 489 933
002 SCHARZ V ARCH DIS CHILD 43 695 968
003 MACE J CLIN PEDIATR 10 285 971
.
CT 1 BOCHKOVA DN GENETIKA (SOVIET GENETICS) 11 154 975
2 OOD RE AM REV RESPIR DIS 113 833 976
3 MOSSBERG B MT SINAI J MED 44 837 977
.
11
Sample CF Queries
QN 00002
QU Can one distinguish between the eIIects oI mucus hypersecretion and
inIection on the submucosal glands oI the respiratory tract in CF?
NR 00007
RD 169 1000 434 1001 454 0100 498 1000 499 1000 592 0002 875 1011
QN 00004
QU hat is the lipid composition oI CF respiratory secretions?
NR 00009
RD 503 0001 538 0100 539 0100 540 0100 553 0001 604 2222 669 1010
711 2122 876 2222
NR: Number oI Relevant documents
RD: Relevant Documents
Ratings code: Four 0-2 ratings, one Irom each expert
15
Preprocessing Ior VSR Experiments
· Separate Iile Ior each document with iust:
Author
Title
Maior and Minor Topics
Abstract (Extract)
· Relevance iudgment made binary by
assuming that all documents rated 1 or 2 by
anv expert were relevant.

$8902;,:,943
W %070,702,370970;,24/08

,47928

 889028 .430890-089 W ,9890-089.42543039147
#,331:3.943 /49
574/:.9 .4830  %072800.943 894547/7024;, 890223 %072093 % % 

W 41,7/43907,30/89,:807300/ 94449413/8420

:20398  .700..39/4..

2.:77039300/8 439.3507.:807 8:/2039 $9:.-3..7 9.9.3.47 3..90/9490700...05943.#0.3/5439 8 $:-0.03088870. 4170970.:9083.389.00503/8:543.:.3-0..3.834995.:9 :/2039942.3.7-:9.3084.943. 1742.0/ 9028 W #00../11.00503/843:2..:2.9.1.3.031700..07920  ..11.0 W #00.90894:807 8.8-3.93#$89028 W 110.3/-0.4393:4:8 W .850.

4/$9..3050798 0.0.475:8 W ..70/4..9..39/4.:89.475:841/4.0 :/20398 W #06:708.7/ W $9.3011479147 .:2.799.-090700.6:07 W %5.7700.0430472470:2..3/.:20398 1470.3..-0:2..:20398 W 40.:2039.-00/47547.809416:070814798.88:208-3.3.438/07.

47547.6:07.  .

.39 70970.0/-:9 700.943 /4.39/4.:20398 70970.:2039870970.:2039870970.0/ 570..40.3:2-0741 700.39 #0970.0/ 70.39 700.:20398 700.39/4.39 7700.3/#0.!70..39 4970970. %49. 3970/4.39 70970.:2039870970.0/  7700.:2039 #00.843.39 ..0/ 34970970. 3:2-0741 /4.0/ :2-0741 700.0/  34970970..0/  7700..39/4.:20398 :2-0741 700.0/  .....843 %49...0/ /4.

9413/.-9419080. W !70.7.:20398 9.9.39 W #0. 4190 700.3/#0..-99470970.30//4..399028390.843 %0..843.702489700.475:8  . %0.0 945 7..!70.

9.:9 W %49.3/5071472 700.811.39809  .80.890949.250..3.-......70.4792894908..9041 700.39902889.3:2-0741700.-.748890/.3990288 84209208349.0907233#0.80147908..20 /.0:/20394390809028 55/110703970970..206:07 %0.700...03.9.-0 $.

  #09:7382489700.39 /4.  !70.:/08 49841:3  .%7../0 411-09003#0.843 #0.843 #09:738700...:20398-:93..3:801:4308944 %0/0..3/!70.:20398-:9 288082.39/4.

..425:93#0.

.08 /110703980984170970..843!4398 W 47.:20398 .30/89574/:.97084/43987.0//4..3/ 90701470/110703970.30/8941 70970.036:07 574/:.0907.8 W /:893..!70.

7/ W 425:90..47/394904/89....:20393907.8:708 W .70.3/.570.84320./4..39.70.98 700.30/899..

8435.548943 3907.:2039  ..39 /4.38.570.439.30/899.9.700..71470.

425:93#0...

41700..843!4398 .39/4.3070.0. 700.8 0.!70..39                                   09949..5439 #.250 3 /4..

  !.

 #.

  !.

 #.

 !.

  #.

 !.

:2039 0.. #.  883430 700.0770.39/4..

  5.

  .   70..

425:93#0...

8 0..3070...39                                   09949.0.39/4.250 3 /4.!70.5439 #.843!4398 .41700.. 700.

  !.

 #.

  !.

  #.

 !.

  #.

 !.

  #.

 !.

  #.

  5.

   .

..390754.#0.93.

.9 .2:2343570.0 7 Z                    < 7  7    7  W %0390754.90/570.570.90.990 989.843.843.3/.843..0-0900390 9.7/ 70..!70.08902.3/.89. ! 7 7 A7 A7   .0.0 W 390754.0.3/   9 0.370..:01470.843:7.7/ 70..0 ! 7  2.0.

#0...93.390754.

  .250 !70.0 .!70.843:7..843           #0.

.93.#0..390754.

0 .250 !70.!70.843:7..  .843           #0.

07...0#0..

843.3.0570.70 809 416:0708 W 425:90..0....7/ 70..843:7..0570.04.!70.90.7488.07.89.05071472..0 W %5..3/.07.07.07.0..6:0708 W !49.843.

70. /4.904.:.:7.889025071472.043.:2039..0894 0..07.3..

6:07.475:8  .

843             #0.70%447470$89028 W %0.480899490:550779 .3..4730741907.  4$902 $902  ..:7.0.53/.3/ .0   !70.90890-089 5071472.425.

250#!:7.0147475:8  .$.

6:079..:20398 3 /4..33 41708:98147.843. 700.# !70.9.843..39 /4.8#700.843 W !70.990# 95489433907.8 # !70.39/4.39                                   #41700.

   .

3/570.34170.08394 .3/570.8:70 W 3020.843  !#      ! # # ! W 425.4:39-4970.79209.09. 0.394-0  ..20....8:70415071472.843 W .3 -49300/94 -0147.99.20.20.3.70/94..7243..7243.

.8432470  . !#  .48093 025.43974897.09570.    . ! # # ! W '..8:709. 09570.8:70 W .9.7.843.  .8434.0770..2470 .394120.8843570.0.8:70 5.  .  .3/70. .209070/0./0 411   .7.:041. 6:. 0970..

07.39 /4.700..0.0!70..9.0.99054398.07.0/           .:2039870970.07.04190570..3.0!70.:08.843 .843 ! W .843.

              .

.04190 .  W 0.0570.843.0!70.07..809416:0708  .843.3.07.07.:0147.

4:39 #0.943 !70.943  ..03394.39902870970.:20398 70970...39/4.0/ .4:9 949.399028390...40.39/4..90 W !74-0289-49570.34 41 343700.843... :2-07417700.8:3/0130/039070834 700.94383499.:20398390 .8438:3/0130/0334/4.3/70.4:9#.0/ 34 41 343700.:2039390..40.40.

94%057454794341700.3/ 34.0#.88:089#00. #00.04..90/4.90.0 W ....3./.#00.:8075747949080.398944.0789.:20398390 7.3..09  W 4.04170970.33.07.03307/4.//3031472.98 .390390:807.0/.08003-01470 0  90-:/09705479147 0.70.7.4:7..700.:20398 343 94.7  .73.907/4.39/4.399028 70970.943-043/.0/80984:/03..0/4:94190949.:20398 .:20398 4.

7.05941.4. .70033904:95:9 W #0854380920%203907.0903994.-994:909070970.3/ 8.4:95:9 1472.07.. :8076:07.43/:.3.94331:03.939080.9434188902708543808 W 472415708039.7.943..94390:807 8.0/ 2.907.907.04180..8 W 40.-0900370.3/905708039.947894438/07 W &80701147947706:70/174290:8073 1472:.936:0708 .

 700..399028.:/0/390/4.703.:2039.475:8  ..

.

.4390945708:9 47.03088-20.3/42803/.0   .974:%0507.943    41 3.3708:94390 17895..303 .041:80789..0540/$8902 W . W #.30549..830..30 W :/00110..8:73.7..8094574.039.9 .17.9.39419088902 9.423:807894.%08933.82.:/08.30893:807-./0 :801:100/-..93.

9.7.:.9438/11.73 W 3.0380941/4..889028-0..:.9..3.$09:514703..:994/08.:203970970.:9147 /4.8:70/--03.:802.70/11.3.3 ..8700.570.8..3.5072039.843 W !071472.88:. 5071472..3/ 700.0 /897-:94341 47/8 09.7-09 2.902.:20398 6:0708 .0/. .2.0820.7432039 :3/07.00.90/   ...2.908890280.9.73 %./431479003..9..0110.9 8 9070970.0308841.3..:.0:/20398 W !071472.3..90/43.889028 0.90789...

2.3/6:0708.439.7.3/.943.03.2.38 8094189.7//4.78 W -03.40.:20398.

40..:203981470.# $2.8 8941700.40.3/..7/.943814797.945./943.39/4.943195.79.6:07 W $9.

.

4730 0/:.195 .8 .

5:-.

82.79 %#995.

.

389 4.970.

0/ 708:9 .7/ 708:9  .3/.. $9.3/.943 $9.3/. $9.7/ 6:0708 4792 :3/079089 #0970.40.7/ /4.943 !70.:2039 .3/70.:.843 ..

70 :8989...7.2.2.79.0.47547.3.714703 .7-03.79394 -0/0.0450/  .2.73 %0!74-028 W !071472.8.70:8989.-03.7 W :/3.2.03.3:./43147.2. 5..0450/ W 03.70-.475:88.:9 9.0/.:.79394-0/0.8 W 03./11.9.47547.

782.7%08940.4:8050720398070-.943.40.9438 W !70..80/4390$#% .81.  195.

.

4730 0/:.8 .195 .

5:-.

$0 -908            W 11070397080.82.078:80//11070399089.79 40.943 .7.9438 .20  $ #  % :2-07 1 4.3/0.:.40..36:08  .:20398             :2-07 1 ":0708      #.94390.

4310703.%0%#03.7 W %#%09#970.0 995..2.

.

389 4.970.

20.:20398 .3/ #! W !..285438470/- 010380/.035..3988:-2990!.094-0/07.398.6:0708.9803.!740..5.943.0814797.3.79.79.3899:9041$9. 73.3.79841.3/90893 W !.3/.03 .90/174290%!$%#5747.4 85438470/-90 .0/#080. #! W 0.4310703.7/8.3/.0/ 3 /110703989.33:.33.5.7/80941/4..7.3/%0.344 $% .89.70.3/% !$ 1742.

990.4310703.:081479013..0  ./4.3/5708039907708:98.#.3/6:07.:2039 .475:8.

045390 -03.7147 30.79.200./02.79.9.083/0.073203994/0.3/050703...7.94390.40..3/6:0708 .2080941/4.4224374:3/147..3/.70 -03.%0%# -0.:./0.4:7.2.9438 W 3..045.:20398.9432094/ W $.9438 #0970..2.08 W !74.73417084:7.73/1107039# 90.4785438478517424.94317423/:897.08.:.36:08 5.36:08 $..5.425.045203941300.05.55.:.3/8. 74:93. W 0.7 92.

80/ .19073 343 38.40.38073  .943 0- -.40.943 6:08943.

08 W .08  %#5.708.10390$#% 40.0732039 W /05.0 .045203998:554791742 90& $ 4.5078  .08 %#5..%#/.5.943 %#5.4393:4:8/0.3./0/ W &3/07..5078 5.0:/20398574.79.08 %#5.5078 5.70/94..943 W #00.5078 5.39.425.

.//0/.88.-.206:089438. 80941/.475:8     243/4.3:.9438-0380.06:089438.70-03.89.7.553  -7.70.%#%.80/ -:930 31472.9:7.1907%# 3907.0 2:950/.80 2073 19073 .  W #4:93$.9.88 W /4..70-03.0  2:93:.9.07.757413 W 09.80/43. 3.0/  308.38073  .9.:20398 6:08943.9.

84190%#40.943 W 4943.0743094:8.4388941 $ . /4.3/:36:0907283.08    ! 884.:20398 1742..#08907  -897.7.79.10 :3/70/944..988 11 ..3/8479/4..:2039 W %089/4.8!:-83  # 0/07.90!70880870    425:907$00.$970094:73..792039413077054798         .98174205.:20398..90789.

2504.:2039 9$      $  .$.

..    43.47/%4$0&39 $4:7.7.780.08$.

  .

.

.

 $  $%#% &#  .

$   #%# #$ ##$ "&$% $ %  #% '#%$ % % &% $  # $% %!  %#! % .

 %  #.

7002039948098%'89.2574/:.208 #480310/ .3/5747.943 .6:8943.085:990.:041905745480/ .924709.48094.07983705708039.0894774:50/-.254 8.0  3/:89784:7.% %% 43.7 .943:3994.943.088.3 243   .:9../.147207$3.3 3./ 3/:89784:7. 00..3.9434507.

%% .

   .

..344 .0883 /08.:2039/0391.$.9:7.943.03.39/4.9 .9:7084190 .250":07 9$ 945 0.0453472.425.059 8  3.08.77 .344.0 574.950413.8-03/0..947 8  3.3473899:943 /0.088390.43 43.3:.0.7 1.3 8574/:..0574.0  /.943 .77.759434. 08.9:7.:2039/0391.3:./ %58907%45..0574...425.9:7.9 .9:7.943.344  /03919090.3$.0700.3//039143041247010.3:.3.088390.7090/390& $  3.. .38.7093.9..9& $ .3:.344 990 %45.0!74.3:.75943 3:2 :2-07  /42 42.0883 97.3/%0.0450/472.

9 .3.

.1. /01 0139438 8  .

945  .

9813.943 10/8 W 0307.0744/.33 -:9190307.3  '0.96:073.!..%#!74507908 W 49/4.947$5.32.:20398.6:0708 440.3 /11070393/84131472.439.3/ 7.-0.94341901472.3/6:0708. 8907085438-94190 88902 889022.90854476:0708174290 945.0 09.

#4:/-05447  .

0570.07.0# 570.07..0570..943 W $:22.84303       /4.8890281479..0/ 3:2-0741700.0/ W ..07.07.0570.0.989.:20398 W #0.04190# 570.9 70.843.:20390.07.0.843 41.:20398.945.:.8 3:2-07 41/4.7020398 W 4.39 /4..0.2110703.  .9 3..:2039870970.9945.08 94.7070970.0. 570.8:2-0741945.3/90.843..07.8431470...79.-089..8438947.

 .

08390 4.5./42.475:8 . '0-475:8 W #0.4141909 W $0941 700.3 57454794341 4.80/4/ 89.083  W %49..0 :/0/6:0708  .8802-0/-$% W 2430-5.7/.0390- -.3.3/.

.984120/..89.943706:0898 6:0708 390 147241..9020/.3/7..39 700.-897.0 49700.73.08 43 W  31472..39  .700.425090386:089438 W #00..4:73.:20398/0907230/.943 W  ..79.-7488  40..05079843 8.39/4.39 .7..90/ -805.

:20390/8 W W W W W W W W W .478:-0.0 .94389498/4..:20398 9.08944907/4.9 #010703..:2039  .9 47097.98 -897..4..98 3478:-0.0883:2-07 :947 %90 $4:7.

9:70841-49.79.42-3.030783/7420 $ 2  8 / .3 $%./.3/ . 1-748884:/-00.$. $ &.2504.4 7.4 #%# %#.943 89.1-74883.3 5.89.90390-90/9010.0307 83/742082./0 # #%#%#%&# $#'#$ !%#  %  '% $ '%%$   ##'#$!#$  $$#%$  .79.1-7488 92489 9490.5.:2039  &:730 #  #4-079843   %89.:/0/-01470.   !  $% # $$.348841.4 $ #! #%  #$.90399.79.  $%&$ '#$&$.3 & % &7.:9478 340/0 98 70570803989097/8:.7054794190.030783/7420.

:20398 #..:20398 ##00.394.3430/893:8-09003900110.98905/.94343908:-2:.48.$..05079  .94797.938..39/4.4/04:7 7.3/ 310.425489434170857..:850780..250":0708 "  "&.93 #  #         "  "&.938 43017420.70943.709438 #  #              #:2-0741#00.94780.98412:.3/8419070857.

3..9 W #00.7.47.7- . /4.39  .3 05079070700.!70574.90/47- ..:20399:89 :947 %90 .90101470.:203987./4...3/347%45.8 -897.88:239./0-3.9 97.9.0:/20392.0883147'$#50720398 W $05..

Sign up to vote on this title
UsefulNot useful