4.

Recognizing cited facts and principles in legal judgments

Recognizing cited facts and principles in

legal judgments
Yu Chen, 03722670

Concepts
k: Kappa

Summary
In common law jurisdictions, cases that have similar facts should receive similar decisions
with respect to the principles, but it is time intensive for legal professionals to identify such
facts and principles in precedent cases. (Challenges)
NLP technology based on Bayesian classifier can classify which sentences in legal judgments
contain cited facts and principles with an overall k of 0.72, compared with human
annotators' k=0.65 and k=0.95 for inter- and intra-annotator agreement respectively.
(Solutions)
Automated analysis of legal principles and facts within cited cases allows identifying the key
information about the cited case which can be used for many purposes, including creation of
detailed case treatment summaries, improvement of search and retrieval methodology for
the case law and many others. (Significance)

Three positive aspects

Machine annotation with Naive Bayesian Multinomial classifier achieves positive
experimental results evaluated by a range of metrics including: Kappa (0.72), precision and
recall(both between 80% and 90%).
The automatic annotation technology introduced in the paper is able to liberate legal
professionals from time intensive works of identifying facts and principles in precedent case.
Automatic identification of legal principles and facts allows a legal practitioner to associate
legal principles and facts providing deep access to and insight into the development of the
law , and to access the law directly rather than via the edited and structured materials made
available by legal service providers.

Three criticisms
Only one machine annotation method based on Naive Bayesian Multinomial classifier is
compared with human annotation, so we still do not know the state-of-the-art results that
can be achieved by automatic annotation.
The author's reason why the machine learning framework is selected for the automatic
annotation is only that its performance is often comparable to more sophisticated

learning methods (Schneider 2005).

In Error analysis, the author only analyze the error source from the data but not analyze the
classification model itself.

Three questions to the author

If a huge dataset is introduced and used in the experiment for both training and testing, will
the performance of machine annotation be improved?
Is this automatic annotation technology used in a real application of law jurisdictions now?
How to further improve the method in the aspects of data and machine learning framework?