Honorable Garrett E. Brown, U.S.D.J. (Ret.)

Civil Litigation Track
Recent Legal and Technological Developments in E-Discovery:

What Every State and Federal Practitioner Should Know
Co-Sponsored by the Federal Practice Section
Moderator: Honorable Garrett E. Brown, U.S.D.J. (ret.)
Speakers: Honorable Tonianne J. Bongiovanni, U.S.M.J

Jennifer Marino Thibodaux, Esq.
Gibbons, PC (Newark)
Matthew F. Knouff, Esq., General Counsel and E-
Discovery Counsel
Complete Discovery Source
Fernando M. Pinguelo, Esq., Chair, Cyber Security &
Data Collection Group
Scarinci Hollenbeck, LLC (Lyndhurst)
© 2013 New Jersey State Bar Association. All rights reserved. Any copying of material herein,
in whole or in part, and by any means without written permission is prohibited. Requests for such
permission should be sent to the New Jersey State Bar Association, New Jersey Law Center, One
Constitution Square, New Brunswick, New Jersey 08901-1520.
The material contained in these pages is for educational purposes only and not intended as a
substitute for the professional services an attorney would normally provide to a client, including
up to the minute legal research.
“E-Discovery For Federal and State Litigators: Practice Pointers, A View
From the Bench, Recent Case Law, and Computer-Assisted Review”
Federal Practice Section, NJSBA Convention, Atlantic City, May 16, 2013
Presented by:
Honorable Tonianne J. Bongiovanni, U.S.M.J., Matthew Knouff, Esq.,

Fernando M. Pinguelo, Esq., and Jennifer Marino Thibodaux, Esq.
Moderated by:
Honorable Garrett E. Brown, U.S.D.J. (ret.)
General E-Discovery
Tab No. Material
1. 52 eDiscovery Terms That You Should Know, Complete Discovery Source
2. Peace of Mind for Serious eDiscovery, Complete Discovery Source
3. Computer Systems Checklist (Administrator), Complete Discovery Source
4. Fernando M. Pinguelo, Esq., Shannon Capone Kirk, Esq., & Sara Johnson Meyers,
Esq., “The Cloud, Server Consolidation, Ephemeral Data, and Information
Security: (Manageable) E-Discovery Challenges Facing CIOs and InfoSec,”
Bloomberg BNA, Digital Discovery & e-Evidence, December 6, 2012
Legal Hold
Tab No. Material
5. The Sedona Conference® Commentary on Legal Holds: The Trigger & The
Process. Copyright © 2013. The Sedona Conference®. Reprinted With
Permission.
6. Medeva Pharma Suisse A.G., et al. v. Roxane Laboratories, Inc., No. 07-5165
(FLW) (TJB) (D.N.J. Jan. 28, 2011)
#1919176 v2
099998-00047
Meet-and-Confer
Tab No Material
7. Local Civil Rule 26.1
8. The Sedona Conference® “Jumpstart Outline,” March 2011 Version. Copyright

© 2013. The Sedona Conference®. Reprinted With Permission.
9. Meet and Confer Checklist, Complete Discovery Source
Preservation & Production

Tab No Material
10. Computer Systems Checklist (Employee), Complete Discovery Source
11. Chain of Custody Short Form, Complete Discovery Source
12. I-Med Pharma Inc. v. Biomatrix, Inc., et al., No. 03-3677 (DRD) (D.N.J. Dec. 9,
2011)
13. Jennifer Marino Thibodaux, Esq., “Not So Fast: 95 Million Reasons to Carefully
Select and Limit Search Terms,” Gibbons P.C. E-Discovery Law Alert, January
24, 2012
14. Goldmark v. Mellina, No. A-5918-10T2 (N.J. App. Div. Jun. 18, 2012)
15. Kevin W. Weber, Esq., “New Jersey State Courts Enter the E-Discovery Arena in
Earnest; Award Sanctions for Email Spoliation,” Gibbons P.C. E-Discovery Law
Alert, June 29, 2012
16. Bull v. UPS, Inc., 665 F.3d 68 (3d Cir. 2012)
17. D’Onofrio v. Borough of Seaside Park, No. 09-6220 (AET) (TJB) (D.N.J. May 30,
2012)
18. Juster Acquisition Co., LLC v. N. Hudson Sewerage Authority, No. 12-3427
(JLL)(MAH) (D.N.J. Jun. 6, 2012)
#1919176 v2
099998-00047
Discoverability of Social Media
Tab No Material
19. Fernando M. Pinguelo, Esq. & Chris Borchert, “Recent Developments in NJ Law:
Social Media and More,” Dictum NJSBA Young Lawyer Division Newsletter,
March 2013
20. Romano v. Steelcase Inc., 907 N.Y.S.2d 650 (N.Y. Sept. 21, 2010)
21. Melissa C. DeHonney, Esq., “‘Private’ Facebook and MySpace Postings Are
Discoverable,” Gibbons P.C. E-Discovery Law Alert, October 11, 2010
22. McCann v. Harleysville Ins. Co. of N.Y., 78 A.D.3d 1524 (N.Y. Nov. 12, 2010)
23. Robert D. Brown, Jr., “New York’s Appellate Division Finds Facebook Accounts
Off-Limits When Discovery Demands Are Non-Specific,” Gibbons P.C. E-
Discovery Law Alert, January 31, 2011
24. Largent v. Reed, Civ. No. 2009-1823 (Pa. C.P. Franklin Nov. 8, 2011)
25. Hoy v. Holmes, No. S-57-12 (Pa. C.P. Schuylkill Jan. 31, 2013)
26. Mailhoit v. Home Depot, U.S.A., Inc. et al., Civ. No. 11-02892 (C.D. Cal. Sept. 7,
2012)
27. Mara E. Zazzali-Hogan, Esq., “No Fishing Expeditions Allowed When it Comes to
the Discovery of Social Media,” Gibbons P.C. E-Discovery Law Alert, October
12, 2012
Social Media & Sanctions

Tab No Material
28. The Kati Roll Co., Inc. v. Kati Roll Platters, Inc., et al., No. 10-3620 (GEB)
(D.N.J. Aug. 3, 2011)
29. Mara E. Zazzali-Hogan, Esq., “Chief Judge Finds That Alteration of Facebook
Page Can Lead to Spoliation Inference,” Gibbons P.C. E-Discovery Law Alert,
December 16, 2011
30. U.S. v. Suarez, No. 09-00932 (JLL) (D.N.J. Oct. 21, 2010)
31. Mara E. Zazzali-Hogan, Esq. & Jennifer Marino Thibodaux, Esq., “Federal Judge
Rules Government Failed to Preserve Text Messages and Orders Adverse
Inference Instruction in Criminal Case,” Gibbons P.C. E-Discovery Law Alert,
#1919176 v2
099998-00047
November 2, 2010
32. Lester v. Allied Concrete Co., et al., Case No. CL.08-150 and CL 09-273 (Va. Cir.
Ct. May 27, 2011)
33. Lester v. Allied Concrete Co., et al., Case No. CL.08-150 and CL 09-273 (Va. Cir.
Ct. Sept. 23, 2011)
34. Mara E. Zazzali-Hogan, Esq., “How a “Stink Bomb” E-Mail and Its Proof That
Facebook Pictures Were Deleted Might Have Blown Up a $10.6 Million Verdict,”
Gibbons P.C. E-Discovery Law Alert, August 8, 2011
35. Mara E. Zazzali-Hogan, Esq., “Lester v. Allied Part 2: ‘Clean Up’ of

Compromising Social Media Evidence Can Result in Severe Sanctions,” Gibbons
P.C. E-Discovery Law Alert, February 21, 2012
36. Jennifer Marino Thibodaux, Esq., “Ex-Juror Who ‘Friended’ Defendant Faces Jail
for Bragging on Facebook About Dismissal From Jury Duty,” Gibbons P.C. E-
Discovery Law Alert, March 22, 2012
Technology-Assisted Review
Tab No Material
37. Da Silva Moore v. Publicis Group & MSL Group, No. 11 Civ. 1279 (ALC) (AJP)
(S.D.N.Y. Feb. 24, 2012)
38. Phillip J. Duffy, Esq., “Taking the Plunge: Judge Peck Issues First Opinion
Endorsing Computer-Assisted ESI,” Gibbons P.C. E-Discovery Law Alert, March
2, 2012
39. 11 Predictive Coding Lessons from Judge Peck’s Opinion in Da Silva Moore,
40. Dino Medina, “The ‘CAR’ Keys to Defensibility (Computer Assisted Review),”
41. Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-

Discovery Can Be More Effective and More Efficient Than Exhaustive Manual
Review, XVII Rich. J. L. & Tech. 11 (2011). Reprinted with Permission.
42. The Grossman-Cormack Glossary of Technology-Assisted Review, Federal Courts

Law Review (2013). Reprinted with Permission.
43. Technology Assisted Review Workflow Graphic, Complete Discovery Source
#1919176 v2
099998-00047
52 eDiscovery Terms That You Should Know
A Discovery: The process of identifying, securing, reviewing
Admissible: Evidence that is allowable in court information that is potentially relevant to the matter and
producing information that can be utilized as evidence in
Analytics: The term used to refer to the various technologies
the legal process
used to provide multiple views into the data set
Document Family: All parts of a group of documents that are
Archive: Long term repository for the storage of records and
connected to each other for purposes of communication; e.g.
files
an email and its attachments
Attachment: A document or file that is connected to another
document or file either externally, e.g. a document connected E
to an email, or embedded, e.g. an image in a word processing e-Disclosure: The eDiscovery process as it is practiced in the
document. European Union
Attachment Backup: Both the action of and the result of Electronic discovery: ediscovery, e-discovery. The process of
creating a copy of data as a precaution against the loss or identifying, preserving, collection, preparing, reviewing and
damage of the original data. producing ESI in the context of a legal or investigative process
B Electronic evidence: Information that is stored in an electronic
format. This is used to prove or disprove the facts of a legal
Backup tape: Portable media used to store copies of data that
matter.
are created as a precaution against the loss or damage of the
original data Email: An electronic communication sent or received via a data
application designed for that purpose (e.g. MS Outlook, Lotus
Batch Processing: The processing of a large amount of ESI in a
Notes, Google Gmail)
single step
ESI: Electronically stored information
C
F
Chain-of-Custody: The processing, tracking and recording
the movement, handling and location of electronic evidence Filtering: The process of applying specific parameters to
chronologically from collection to production. It is used to verify remove groups of documents that do not fit those parameters,
the authenticity of the ESI. in order to reduce the volume of the data set, e.g. date ranges
and keywords
Child Document: A file that is attached to another
communication file. E.g. the attachment to an email or a Forensics: The handling of ESI including collection,
spreadsheet imbedded in a word processing document examination and analysis, in a manner that ensures its
authenticity, so as to provide for its admission as evidence
Container File: A single file containing multiple documents
in a court of law
and/or files, usually in a compressed format; e.g. zip, rar, pst
FRCP: Federal Rules of Civil Procedure, the rules that govern
Custodian: Most often refers to the individual from whose file
ediscovery and other aspects of the civil legal process.
systems a group of records were extracted. This person is not
necessarily the author of the documents. H
D Hash: An algorithm that generates a unique value for each
document. It is referred to as a digital fingerprint and is used to
Data Extraction: The process of parsing data from electronic
authenticate documents and to identify duplicate documents.
documents to identify their metadata and body contents
Data Mapping: The process of identifying and recording the I
location and types of ESI within an organization’s network, and Image (Drive): To make an identical copy of a drive including its
policies and procedures related to that ESI empty space. “mirror image”
De-duplication: De-duping. The process of comparing the Image (File): To make a picture copy of a document. The most
characteristics of electronic documents to identify and/or common image formats in ediscovery are tiff and pdf.
remove duplicate records to reduce review time and increase
coding consistency L
De-NIST: The process of separating documents generated by a Legacy Data: Data whose format has become obsolete.
computer system from those created by a user. This automated Legal Hold: A communication requesting the preservation of
process utilized a list of file extensions developed by the information that is potentially relevant to current or a reasonably
National Institute of Standards and Technology. anticipated legal matter and the resulting preservation.
Peace of Mind for Serious eDiscovery™

www.cdslegal.com • (855) 813-0700
Load File: A file used to import data into an ediscovery system. R
It defines document parameters for imaged documents and Recall: In search results analysis, recall is the measure of the
often contains metadata for all ESI it relates to. percentage of total number of relevant documents in the corpus
returned in the results set. See Precision
M
Redact: To intentionally conceal, usually via an overlay,
Media: The device used to store electronic information, e.g.
portions of a document considered privileged, proprietary or
hard drives, back up tapes, DVDs
confidential
Metadata: Often referred to as data about data, it is the
information that describes the characteristics of ESI, e.g. S
sender, recipient, author, date. Much of the metadata is not Search: The process of looking within a data set using specific
accessible to non-technical users. criteria (a query). There are several types of search ranging from
simple keyword to concept searches that identify documents
N related to the query even when the query term is not present in
Native Format: A file that is maintained in the format in which it the document.
was created. This format preserves metadata and details about
Slack space: The unused portion of a disk that exists when the
the data that might be lost when the documents are converted
data does not completely fill the space allotted for it. This space
to image format, e.g. pivot tables in spreadsheets.
can be examined for otherwise unavailable data.
Near-duplicate: Two or more files that contain a specified
Spoliation: The destruction or alteration of data that might be
percentage of similarity. Also, the process used to identify those
relevant to a legal matter
nearly-identical files
Structured data: Data stored in a structured format such as a
Normalization: Reformatting data so that it is stored in a
database. Structured data can create challenges in ediscovery.
standardized format
See Unstructured data
O
T
OCR: Optical Character Recognition is the process of converting
TIFF: Tagged Image Format is a common graphic file format. The
images of printed pages into electronic text
file extension related to this format is .tif.
P
U
Parent Document: A document to which other documents/files
Unallocated space: Most often, this is space created on a hard
are attached
drive when a file is marked for deletion. This space is no longer
Predictive Coding: A document categorization process that allocated to a specific file. Until it is overwritten, it still contains
extrapolates the tagging decisions of an expert reviewer across the previous data and can often be retrieved.
a data set. It is an iterative process that increases accuracy with
Unicode: The code standard that provides for uniform
multiple training passes.
representation of character sets for all languages. It is also
Precision: In search results analysis, precision is the measure referred to as double-byte language.
of the level of relevance to the query in the results set of
Unstructured data: Data that is not stored in a structured
documents. See Recall
format such as word processing documents and presentations
Processing: The ediscovery workflow which ingests data,
extracts text and metadata, and normalizes the data. Some
systems include the data indexing and de-duplication in their
processing workflow.
Complete Discovery Source (CDS Legal) is a leading provider
Production: The delivery, to the requesting party, of documents
of eDiscovery services and a trusted advisor to more than
and ESI that meet the criteria of the discovery request
75% of the Am Law 100 and a quarter of the Fortune 500. CDS
uses the most advanced, tested, and defensible technology
to support all stages of the EDRM including planning, early
case assessment, processing and production, review hosting,
managed review, and cross-border global coverage.

eDiscovery is a critical part of your case. From proactive litigation-readiness to

managed review, the process must be fast, efficient, and focused on reducing costs.
Choosing the right partner to ensure a strategic and defensible approach gives you the
first advantage. Having experts manage that course of action puts you one step ahead
from start to finish.
CDS is the first choice www.cdslegal.com
of more than 75% of CDS uses the most advanced, analytic technology (855) 813-0700
Am Law 100 firms. available. Our hands-on experts personally shepherd
your evidence through time-tested processes to provide clear strategies and superior
outcomes. From accessible project managers, who you
know by name, to our SAS 70 Type II audited, secure One quarter of
data centers, everything we do is about providing you the Fortune 500
with unmatched peace of mind. partners with CDS.
“Early strategies are critical to the success and path of work. Then those
strategies need to be executed flawlessly. CDS is our first choice for reliable,
defensible eDiscovery.”
— PARTNER, AM LAW 100 FIRM
Litigation Readiness Processing and Production

eDiscovery requirements are changing fast. eDiscovery that is fast, efficient, and defensible
Companies and law firms must be prepared allows your litigation team to focus on more
for today’s complex discovery requests while manageable volumes of meaningful evidence.
anticipating what’s next. CDS partners with Our project managers and engineers use best-in-
legal teams to establish proactive information breed hardware and software to expertly guide
management through better data retention and your evidence through the process. Size does not
destruction policies. Shrinking data volumes matter. We can scale our services from a single
before a complaint is filed makes you faster CD to terabytes of ESI.
out of the gate, reduces costs, and offers you
Time has direct implications on eDiscovery cost
better command of even the most demanding
and resources. CDS offers unmatched processing
discovery requests.
speeds at competitive prices. Our capacity and
dedication turn most projects around in the time
Early Case Assessment that it takes the competition to get started.
eM@S, CDS’ proprietary data analysis system, • High-speed multi-thread processing

indexes documents at lightning speed, so • Real-time reporting
you can start evaluating data before most • De-duplication
other systems have finished ingesting. Multi- • Full redaction tools
threading and our distributive architecture • Native and image file production
allow your CDS team to analyze multiple data • Concordance or EDRM compliant
sources simultaneously. Detailed reports, de- XML load files
duplication, and extensive filtering tools mean
• Integrated production services
that the dataset can be drastically reduced
prior to processing. eM@S delivers time and
cost savings that are just one part of the ECA continued on back
process.
NEW YORK • WASHINGTON, DC • CHICAGO • SAN FRANCISCO • ATLANTA

Review Hosting and Global Coverage and Trans-

Managed Review border eDiscovery/eDisclosure
If you’re working around the clock, so are we. The world is getting smaller, but the demands of
We support our highly secure SAS 70 Type II data global electronic discovery are growing. Multi-
centers 24/7. And rest assured that we employ national corporations involved in U.S. litigation www.cdslegal.com
the most advanced technologies and protocols can be required to collect data that is subject (855) 813-0700
to protect sensitive data from damage by malice, to EU directives. CDS’ international eDisclosure
accident or error. program merges EU-based eDisclosure specialists
CDS Headquarters
with our proven U.S. based eDiscovery firm,
CDS partners with the industry’s leading review 345 Park Avenue
to seamlessly carry you through the process. New York, NY 10154
tool developers and contract review staff to meet
And CDS’ Safe Harbor certification ensures that (212) 813-7000
your case’s specific needs. For complex projects,
data which reaches the U.S. remains compliant
our experienced PMs and Director of eDiscovery
with EU privacy regulations. From collection
Strategy work with case teams to tailor work- CDS Washington
to production, our solutions are ready for your
flows—from reviewer training to trial preparation. 1050 Connecticut Avenue NW
overseas needs.
Washington, DC 20036
Our Unicode-compliant technology provides (202) 822-5999
CDS’ National Data Center is  multi-lingual capabilities. It automatically
SAS 70 Type II audited. Our three identifies the languages within documents
locations are available 24/7. CDS Chicago
and provides detailed reports. Reviewers can
70 West Madison St. 
search documents in Roman, Asian, and Cyrillic Three First National Plaza 
languages. Suite 1400 
Chicago, IL 60602-4270
(312) 756-7301
“CDS has repeatedly proven its reputation as an industry leader, a partner guaranteed
CDS San Francisco
to tackle and meet any project and deadline in a cost effective manner. [Our] entire
425 Market Street 
project had to be completed within two days time . . . and they completed the project Suite 2200 
early . . . CDS is a highly valuable, respected, and competitive asset in the e-discovery San Francisco, CA 94105
community.” (415) 692-5595
— E-DISCOVERY PROJECT MANAGER, TOP 50 LAW FIRM
“CDS has been a tremendous help and life saver . . . we have used them to migrate
several databases from other platforms because of their stellar support and services . . .
they have lived up to what they promised.”
— LIT SUPPORT MANAGER, AM LAW 100 FIRM
We self-certify compliance with

New York Top Three
Schedule
Best eDiscovery U.S. ★ EU
• BEST IN SERVICE
Law Journal Provider Contract 35F-0480R SAFEHARBOR
• CERTIFIED RESELLER Rankings Complete Discovery Source U . S . D E PA R T M E N T O F C O M M E R C E
COMPUTER SYSTEMS CHECKLIST
(ADMINISTRATOR)

COMPANY NAME
IT ADMINISTRATOR
JOB TITLE
DEPARTMENT
CONTACT NUMBER
EMAIL ADDRESS
AVAILABILITY

GENERAL INFORMATION

1. Where are the physical locations where target devices reside?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

2. Are there any target locations outside the U.S.? [Yes / No] If yes, please list these other locations.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

3. What are the operating systems of the servers to be collected (e.g., Unix, Linux, Windows,
Macintosh OS, etc.)?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

4. How many file servers are on the system (size and locations)?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be
discussed with or distributed to any other parties without prior written consent from CDS.
© 2013 Complete Discovery Source, Inc.

(ADMINISTRATOR)

5. Are there any proprietary systems or business servers? [Yes / No] If yes, please describe.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

6. Does the company use SharePoint or other content management or data storage systems? [Yes /
No] If yes, please describe.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

7. Are there any structured data repositories that need to be collected (e.g., accounting data, human
resources information, etc.)? [Yes / No] If yes, please describe. NOTE: This type of information, as
well as information from other sources, could be subject to HIPAA or other regulations.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

8. How many data storage devices and custodian computers are within the scope?

Device Number Make Model Storage
Desktop
Laptop
Mobile devices
CDs
DVDs
Thumb drives


(ADMINISTRATOR)

9. Does the company use any private or public cloud‐based services for data storage or content
management? [Yes / No] If yes, please describe.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

10. Can you briefly describe the ways in which your company secures company data? Are there any
documented policies regarding access restrictions, device requisitioning and disposal, passwords, or
any other such security and/or use policies? [Yes / No] If yes, please describe. Request copies of
each policy.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

11. Have any computers, servers, or other devices been replaced or decommissioned during the
relevant time period? [Yes / No] If yes, please describe. Request copies of any records.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

12. Are home drives available to employees? [Yes / No] If yes, please describe. Be sure to include
information on how documents are shared internally within the organization.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________


(ADMINISTRATOR)

13. Do users have remote access to the network from outside the office? [Yes / No] If yes, please
describe.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

DOCUMENT RETENTION POLICY

1. Is there an existing document retention policy? [Yes / No] If yes, request a copy.
______________________________________________________________________

2. How is the policy implemented and enforced?
________________________________________________________________________
________________________________________________________________________

3. Has the policy been suspended since this matter began? (i.e. litigation hold procedures) [Yes / No]
If yes, please describe.
________________________________________________________________________
________________________________________________________________________

EMAIL SYSTEM

1. What type of email system(s) is currently deployed (e.g., Microsoft Outlook, IBM Lotus Notes,
QUALCOMM Eudora, etc.)?
________________________________________________________________________
________________________________________________________________________

2. Is the current email system the same system that was used during the relevant time period? [Yes /
No] If no, then please describe.
________________________________________________________________________
________________________________________________________________________


(ADMINISTRATOR)

3. Is the email data center located on‐site or at a remote location?
________________________________________________________________________

4. Is there a size restriction for individual mailbox(s)? [Yes / No] If yes, what action is taken to reduce
mailbox sizes?
________________________________________________________________________

5. How are emails stored? Are there multiple email stores? [Yes / No] If yes, how many and where are
they located?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

6. What system(s) is currently used to back up email?
________________________________________________________________________

7. In what format are emails backed up?
________________________________________________________________________

8. How frequently are the emails backed up (e.g., daily, weekly or monthly)?
________________________________________________________________________

9. Where are the email archives stored?
________________________________________________________________________

10. How long are the archives maintained?
________________________________________________________________________

11. What type of storage media is used for email backups?
________________________________________________________________________

12. Is the backup data accessible? [Yes / No] Whether yes or no, please describe.
________________________________________________________________________


(ADMINISTRATOR)

13. Is the backup data text searchable? [Yes / No] Whether yes or no, please describe.
________________________________________________________________________

14. Is the backup data separable by custodian? [Yes / No] Whether yes or no, please describe.
________________________________________________________________________
________________________________________________________________________

15. Is there a disaster recovery plan in addition to the backup procedure? [Yes / No] If yes, request a
copy.
________________________________________________________________________

16. Does anyone use personal email accounts for business? [Yes / No] If yes, please provide further
details on access restrictions and acceptable use.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

DESKTOPS AND LAPTOPS

1. What is the estimated amount of data per custodian of interest?
________________________________________________________________________

2. What operating system(s) do the workstations generally run on?
________________________________________________________________________

3. Do any custodians use home computers for business purposes? [Yes / No] Is there a record of that
information? [Yes / No] If yes, request a copy.
________________________________________________________________________


(ADMINISTRATOR)

4. What system(s) is currently used for workstation backup?
________________________________________________________________________

5. How are user workstations backed up?
________________________________________________________________________
________________________________________________________________________

6. How frequently are the workstations backed up (e.g., daily, weekly or monthly)?
________________________________________________________________________

7. Where are the workstation archives stored?
________________________________________________________________________

8. Is there an Acceptable Use Policy for personnel use of PCs, Laptops, the Internet, and/or any other
devices? [Yes / No] If yes, request a copy.
________________________________________________________________________

9. Is there any encryption on the desktops and/or laptops? [Yes / No] If yes, please describe. Is there
an Encryption Policy? [Yes / No] If yes, request a copy.
________________________________________________________________________

BACKUP TAPES

1. Please describe your backup schedule(s) and procedures.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________


(ADMINISTRATOR)

2. What backup software packages and versions are currently being used?
________________________________________________________________________

3. Is there a documented backup tape rotation schedule? [Yes / No] Whether yes or no, please
describe.
________________________________________________________________________
________________________________________________________________________
4. What types of tapes are/were used, including manufacturer, model, and size (e.g., QIC, DAT, DLT)?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

5. Do the tapes contain e‐mail or file server data? [Yes / No / Both] Whether yes, no, or both, please
describe.
________________________________________________________________________

6. What are the date ranges for the tapes?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

7. Is there a historical records/master index of the backup tapes contents? [Yes / No] If yes, request a
copy.
________________________________________________________________________

8. Is there an established tape‐labeling protocol? [Yes / No] If yes, please describe.
________________________________________________________________________


(ADMINISTRATOR)

REMOVABLE MEDIA

1. Do custodians store data on other media (e.g., USB drives, Zip drives, DVD, CD, Floppy, removable
hard drive)? [Yes / No] If yes, please describe. Is there a policy regarding removable media? [Yes /
No] If yes, request a copy.
________________________________________________________________________

2. Do custodians backup their own data? [Yes / No] If yes, please describe.
________________________________________________________________________

DATABASES

1. What types of database server systems and client programs are currently in use?
________________________________________________________________________

2. What are the names of database software used?
________________________________________________________________________

3. Are there standard reports prepared from the databases? [Yes / No] If yes, please describe.
________________________________________________________________________

LEGACY SYSTEMS

1. Are there legacy e‐mail, file handling, file storage, voicemail, or database systems? [Yes / No] If yes,
please describe.
________________________________________________________________________

2. How is this data stored? Is this storage server within the scope of the preservation obligation?
________________________________________________________________________


(ADMINISTRATOR)

3. Is the system active or archived?
________________________________________________________________________

VOICE DATA

1. What voicemail system is in use?
________________________________________________________________________

2. Where is the voicemail data stored?
________________________________________________________________________

3. Can users store messages locally? [Yes / No] If yes, for how long?
________________________________________________________________________

4. Is voice data backed up? [Yes / No] If yes, how is this data backed up?
________________________________________________________________________

MOBILE DEVICES (e.g., Blackberry, Palm, iPad, Treo, iPod, Cell Phones, etc.)

1. Does the organization issue or support any mobile devices? [Yes / No] If yes, which types of
devices? Please include device type, make, model, storage sizes, and any other relevant
information.
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________


(ADMINISTRATOR)

2. Is the use of personal mobile devices permitted? [Yes / No] If yes, can they be connected to work
issued devices (e.g., desktops, laptops, etc.)? [Yes / No] Are employees allowed to store data on
mobile devices? [Yes / No] If yes to any of the above, please explain.
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________

3. Is the same data stored on the device stored on shared drives at the organization? [Yes / No] If so,
where are these shared drives located?
______________________________________________________________________

4. Is there an Acceptable Use Policy for personnel use of mobile devices? [Yes / No] If yes, request a
copy.
______________________________________________________________________

SOCIAL MEDIA AND NETWORKING

1. Are employees restricted from using social media or networking sites? [Yes / No] Please describe.
If yes, is there a documented policy? [Yes / No] If yes, request a copy.
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________

2. Are you aware of any employees using social media within the scope of their work‐related duties?
[Yes / No] If yes, please describe.
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________


(ADMINISTRATOR)

3. Does the company itself maintain a presence on any social media or networking sites? [Yes / No] If
yes, which sites? Who maintains these sites? What security precautions are taken with respect to
these sites?
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________

4. Are you aware of any social media, networking sites, or blogs related to the type of work that your
company does or the industry in which your company operates? [Yes / No] If yes, please name
these sites.
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________

ADDITIONAL SOURCES & COMMENTS
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________


Digital Discovery
& e-Evidence ®
Reproduced with permission from Digital Discovery & e-Evidence, 12 DDEE __, 12/06/2012. Copyright 姝 2012 by
The Bureau of National Affairs, Inc. (800-372-1033) http://www.bna.com
BNA INSIGHT
The authors explain why CIOs and CISOs must balance the sometimes competing needs
of cost-effective technological solutions for business needs against legal preservation and
collection requirements.
The Cloud, Server Consolidation, Ephemeral Data, and Information

Security: (Manageable) E-Discovery Challenges Facing CIOs and InfoSec
BY FERNANDO M. PINGUELO, SHANNON CAPONE Why are CIOs and CISOs asked to balance the some-
times competing needs of cost-effective technological
KIRK, AND SARA JOHNSON MEYERS solutions for business needs against legal preservation
and collection requirements (a.k.a. E-Discovery)? Sim-
ply: Because they have to.
I. Issues Facing CIOs and CISOs Cloud Computing. Today’s CIOs are storing more data
That May Impact E-Discovery in more places than ever before. In 2011, 90.9 percent
of data center sites surveyed by the Association for
oday’s CIOs and CISOs face challenges with a wide
T range of technological issues: cloud storage, server

consolidation, retention of ephemeral data, data
security, and privacy laws, to name a few. On top of
Computer Operations Management (AFCOM) used
more storage space than they did three years ago.2 Dur-
ing that same three-year period, 37 percent were able to
that, electronic discovery (E-Discovery)1 invades nearly
every aspect of these issues. ample, in December 2006, FRCP 26(a) was amended to add
‘‘electronically stored information’’ as its own category of dis-
coverable information; FRCP 26(b)(2) sets up a two–tier dis-
1
Electronic discovery, more commonly known as covery process for accessible and inaccessible data, including
E-Discovery, is the component of U.S.-based legal actions procedures for cost shifting on inaccessible data. FRCP
whereby parties (and in some cases, non-parties) are com- 26(b)(5) and FRE 502 both address the increased risk of inad-
pelled to exchange information (in particular electronically vertent production of privileged documents given the volume
stored information (ESI)) relevant to the claims and defenses of ESI to be reviewed, and FRCP 26(f) mandates that parties
asserted in the lawsuit. The Federal Rules of Civil Procedure meet and confer prior to the commencement of discovery in
(FRCP), the Federal Rules of Evidence (FRE), and state and lo- order to agree on review and production protocols.
2
cal procedural rules have been amended to address the unique AFCOM Survey Examines Data Center, Cloud Computing
challenges posed by ESI in the context of a lawsuit. For ex- Use. http://www.tmcnet.com/channels/colocation/articles/
COPYRIGHT 姝 2012 BY THE BUREAU OF NATIONAL AFFAIRS, INC. ISSN 1941-3882

2
reduce their staff, and 29 percent kept their staffing lev- Bask Iyer, CIO of Juniper Networks, added that busi-
els the same.3 According to AFCOM CEO Jill Yaoz, this nesses must be mindful of employee use habits so as to
trend is in large part due to the development and imple- ensure that employees utilize the company’s chosen
mentation of new tools and processes that have allowed storage service providers.7
IT departments and data centers to store massive
amounts of data efficiently and inexpensively. Specifi-
cally, IT departments and data centers are increasingly
relying on cloud-based systems for the storage of big [C]ompared to pre-cloud storage, businesses may
data.
The advent of cloud-based storage systems has had a lose control over the timing and execution of
profound impact on the way businesses collect and
store their information. By offering the ability to store the ‘‘extraction’’ of data.
large amounts of data remotely, cloud storage not only
enables CIOs to drastically reduce their IT budgets, but
also allows IT departments the freedom to concentrate
on matters more central to business growth, such as Simply put, because of the ease associated with stor-
high-value activities and innovation. The number of ing data in the cloud, businesses might assume the
data centers implementing cloud-based storage is only same is true for producing data from the cloud. This as-
expected to increase. According to Yaoz, ‘‘[o]ur predic- sumption, if untrue, might complicate a company’s at-
tion is that 80 percent to 90 percent of all data centers tempt to comply with E-Discovery demands: Data may
will be adopting some form of cloud computing in the be in a foreign location with privacy laws, manpower,
next five years.’’4 and technology that impede the search, storage and ex-
Yet, E-Discovery concerns may require businesses to traction of data in the method or time required. Also,
maintain comprehensive records on what data is stored data for one custodian may not necessarily be grouped
in the cloud and where, precisely, that cloud exists. This together on one server, and a group of custodians might
information may become important in the context of be dispersed amongst many servers.
E-Discovery if data is located in a foreign jurisdiction Moreover, issues can arise with respect to the notifi-
with privacy laws restricting or ‘‘blocking’’ the transfer cation and imposition of a company’s preservation obli-
and/or storage of data.5 gations when information is stored in the cloud as op-
In addition, some cloud service providers employ posed to on a company’s own servers.
subcontractors to store data. This may be problematic Admissibility issues can also be triggered if the chain
for businesses ordered to produce data in a timely man- of custody for the various document sources is not ac-
ner, especially if the requested data is located in more curately tracked and documented. In other words, the
than one place with more than one subcontractor. In data is quite simply not all in one room; it may be in
other words, compared to pre-cloud storage, businesses many shifting rooms, which brings us to our next topic:
may lose control over the timing and execution of the server consolidation.
‘‘extraction’’ of data.
At a recent conference in Palo Alto, Google CIO Ben
Fried addressed the importance of streamlining various Server Consolidation
cloud storage providers tapped by the same company to
At the same time that CIOs are looking to reduce
ensure easy and efficient integration—and to prevent
spending by outsourcing data management and reten-
what he characterized as a widespread lack of good cor-
tion to the cloud, they may also be looking to consoli-
porate practices in the context of remote data storage.6
date internal servers to one location, or if it is a world-
wide company, to one or several ‘‘regions.’’8
171139-afcom-survey-examines-data-center-cloud-computing- The analysis a CIO might consider may typically in-
use.htm. clude the following factors:
3
IT Workloads Up As Data Center Technology Added
Without More Staff. http://www.infoworld.com/d/data-center/ 1. Cost savings with consolidation—will we reap
it-workloads-data-center-technology-added-without-more- them?
staff-186.
4
Id.
5 7
See, e.g., Directive 95/46/EC, of the European Parliament Id.
8
and of the Council of 24 October 1995 on the Protection of In- As noted above, foreign data privacy laws must be taken
dividuals with Regard to the Processing of Personal Data and into consideration when implementing decisions about data
on the Free Movement of Such Data [hereinafter Directive 95/ storage. In the event of U.S. litigation, data stored in a foreign
46], 1995 O.J. (L281) (EC), available at http://eurlex.europa.eu/ country will likely not be as accessible in response to U.S. dis-
LexUriServ/LexUriServ.do? covery demands. Moreover, foreign data privacy laws may re-
uri=CELEX:31995L0046:EN:HTML (prohibiting transfers of strict the retention of certain types of data, such as personal
data to other countries that do not provide adequate levels of information, and may impose restrictions on whether and how
protection), and Working Document 1/2009 on Pre-Trial Dis- long data can be retained, making compliance with preserva-
covery for Cross Border Civil Litigation, at 8 (Feb. 11, 2009) tion obligations under U.S. case law problematic. See, e.g., the
[hereinafter E.U. Working Document], available at http:// discussion of the Zubulake case, infra; Directive 95/46, supra
ec.europa.eu/justice/policies/privacy/docs/wpdocs/2009/ and E.U. Working Document, supra, note 6, at 8. For a detailed
wp158_en.pdf (noting that ‘‘[t]he mere or unsubstantiated pos- analysis of the implications surrounding the imposition of U.S.
sibility that an action may be brought before the U.S. Courts’’ litigation holds in foreign jurisdictions, see Kenneth N. Rash-
may not be sufficient to impose preservation obligations.). baum, Matthew Knouff & Melinda Albert, ‘‘U.S. Legal Holds
6
Top CIO Challenges: Cloud, BYOD, Big Data. http:// Across Borders: A Legal Conundrum?’’, 13 N.C. JOLT 68 (Fall
www.cioinsight.com/c/a/Expert-Voices/Top-CIO-Challenges- 2011) available at http://www.ncjolt.org/sites/default/files/Art_
Cloud-BOYD-Big-Data-490657/. Rashbaum_Knouff_Albert_69_94.pdf.
12-6-12 COPYRIGHT 姝 2012 BY THE BUREAU OF NATIONAL AFFAIRS, INC. DDEE ISSN 1941-3882
3
2. Business continuity—is there a risk of consolidat- Although the opinion does not directly address the is-
ing i.e. if the servers in one location experience an in- sue of server consolidation, the court does refer to the
terruption, how (and how long) will that impact the fact that all of Samsung’s email was consolidated on
company? (Think power loss, storms/natural disaster, servers in Korea. ‘‘[Samsung’s email] contains a ‘gen-
war/civil unrest.) eral guideline [that] calls for all emails to be automati-
cally deleted after the passage of two weeks.’ This func-
3. Data privacy laws—how might they be impacted tionality operates and stores email company-wide in
by consolidation? How can and should I align disparate Korea [and] has no exceptions . . . .’’11 Without elabo-
laws and regulations? rating on its opinion of the server consolidation, the
4. Security—how can I ensure it? Court did take the time to make this further statement:
‘‘[Samsung’s email platform] stores received and sent
It is difficult to answer these questions in a vacuum, employee emails on company-wide servers, as opposed
especially because no two companies’ needs are the to dividing the servers by business unit . . . .’’12
same and because local counsel in differing jurisdic- From these references, some might wonder whether
tions are usually required for full analysis. While no the court intended to imply that a company might bet-
U.S. court is known to have directly weighed in on the ter manage its preservation duties by not consolidating
pros and cons of server consolidation, one court did re- all functionality in one location. In other words, is the
cently refer to server consolidation in the context of a court asking whether Samsung could have lifted the 14-
larger E-Discovery dispute. day deletion policy for certain relevant business units
In the patent infringement action, Apple v. Sam- and thereby complied with its preservation duties and
sung,9 the parties sought adverse inference jury instruc- also not had the burden of storing all emails, company-
tions regarding spoliation of evidence. Responding to wide?13
Apple’s motion, the court considered whether Samsung
took adequate steps to prevent the spoliation of relevant
data after it should have reasonably anticipated litiga-
tion. At issue was Samsung’s default email system, Ephemeral data—such as temporary Internet files,
called mySingle, which automatically deletes all emails
after a two-week period, unless otherwise specified by frequently overwritten or automatically updated
the employee user.
According to Samsung, it relied on the automated de-
metadata . . . , log files . . . , RAM, and fragments
letion feature for several business reasons. First, auto-
mated deletion can help to prevent the misappropria-
of deleted files in unallocated (slack) computer
tion of confidential business information in the event a
computer is lost or stolen. Second, a two-week deletion
space—may be possible sources of responsive
period is less expensive to implement than a 30-day de-
letion period. Third, the shorter window means that
data.
confidential business information is less likely to be dis-
closed to a third-party via email or unauthorized access.
Fourth, a two-week automated deletion period best Ephemeral Data and Data Security. The management of
complies with Korean privacy laws. cloud-based data and server consolidation is not the
Apple argued that Samsung’s continued use of its bi- only consideration for a company in litigation. Ephem-
weekly email destruction policy without any methodol- eral data—such as temporary Internet files, frequently
ogy for verifying whether Samsung employees com- overwritten or automatically updated metadata (such as
plied with Samsung’s litigation hold instructions dem- last-opened files), log files (such as firewall or other se-
onstrated inadequate data retention methods resulting curity related log data), RAM, and fragments of deleted
in spoliation of relevant evidence. The court agreed files in unallocated (slack) computer space—may be
with Apple, and directed that an adverse inference in- possible sources of responsive data.
struction be used at trial against Samsung.10
tical adverse inference instructions were not given at trial be-
9
Apple Inc. v. Samsung Electronics, BL _____ (N.D. Cal. cause ‘‘. . . the parties indicated at the August 20, 2012 hearing
July 25, 2012); Motion for Relief from Judgment granted in that if the Court decided to issue identical adverse inference in-
part by Apple Inc. v. Samsung Electronics, 2012 BL _____ (N.D. structions against both parties, they prefer that neither adverse
Cal. August 21, 2012); (see, 12 DDEE 304, 8/2/12) inference instruction be given.’’ Id.
10 11
Apple Inc. v. Samsung Electronics, 2012 BL _____, supra, Apple Inc. v. Samsung Electronics, 2012 BL _____, supra,
at __. Note however, that this instruction was never given be- at __.
12
cause the presiding trial judge, Federal District Court Judge Id.
13
Lucy Koh, overruled Federal Magistrate Judge Paul S. Grew- Although arguably it may seem to be less burdensome to
al’s Order, and decided instead to give the same instruction implement a hold where, as here, servers are consolidated,
with respect to each of the parties, i.e., that they had each Samsung filed affidavits claiming that to extend ‘‘the retention
failed to preserve relevant evidence after the duty to preserve policy for its employees would have cost an additional
arose and ‘‘[w]hether this fact is important to you in reaching $35,983,193 per year.’’ Id. at __, note 43. However, Samsung
a verdict in this case is for you to decide.’’ http:// ‘‘did not estimate the cost of temporarily moving key custodi-
www.groklaw.net/pdf3/ApplevSamsung-1848.pdf.; Apple Inc. ans’ email accounts to unique servers that do not biweekly de-
v. Samsung Electronics, 2012 BL _____, supra at *__. This was stroy emails, or the cost of temporarily moving . . .’’ these cus-
a significant victory for Samsung which had argued that Ap- todians’ accounts to Microsoft Outlook. Id. In addition to cost,
ple’s duty to preserve must have arisen no later than Sam- data privacy laws may also affect a company’s data storage de-
sung’s. Magistrate Judge Grewal had not reached Samsung’s cisions. As noted above, Samsung averred that their consoli-
equal treatment argument initially because he had denied dated email storage system policy was the one that ‘‘best com-
Samsung’s motion as untimely. Ultimately, however, the iden- plies with Korean privacy law.’’ Id. at note 45.
DIGITAL DISCOVERY & E-EVIDENCE REPORT ISSN 1941-3882 BNA 12-6-12

4
Two recent cases address the issue of ephemeral data

in the context of E-Discovery. In Nacco Materials Han-
dling Group v. Lilly Company,14 a lift truck manufac- Although ephemeral data is certainly not necessary
turer brought an action against a former dealer alleging
that the dealer improperly accessed the manufacturer’s in most cases . . . parties should consider whether
secure website.
The court held that the dealer had an obligation to
ephemeral data is a source of information that
initiate the preservation of relevant ESI—including might be responsive to a given litigation.
ephemeral data such as Internet browsing histories
—immediately upon receiving the manufacturer’s com-
plaint. Because the dealer failed to preserve the relevant
data, the court imposed sanctions, specifically ordering Another nuance in the preservation of ephemeral
the reimbursement of plaintiff’s costs for forensic ex- data is the need to sometimes preserve firewall, anti-
amination, as well as the cost of imaging and analysis virus, intrusion prevention and/or detection, network
of the manufacturer’s hard drives. activity, and other logs of information that might show
the path a hacker might have taken. The preservation,
In Tener v. Cremer,15 the court considered whether a of course, might be for a forensic investigation or litiga-
non-party should be compelled to produce temporary tion needs, but may very well be different from the nor-
user logs, which were deleted through the course of mal course of business.
normal business operations. After finding that defen- Parties faced with this type of preservation request
dant posted defamatory statements using a computer are often asked if they have centralized logging, and if
owned and operated by New York University (NYU), not, how quickly the overwriting of logging can be dis-
plaintiff served the school with a subpoena seeking the abled. Any CISO concerned with being prepared for in-
names of all of the individuals who accessed the Inter- vestigating security incidents might want to consider
how to respond to an urgent request to preserve this
net on the date the statement originated.
type of data.17
Plaintiff argued that NYU could use special software
to retrieve the data, despite NYU’s claim that the data
had been overwritten numerous times and therefore II. What Is the Legal Department Faced With?
could not be retrieved, at least not without undue bur- As noted in the discussion of the Samsung case
den or cost. above, several courts have held that it is counsel’s duty
The court articulated a two-pronged inquiry to deter- to affirm that a party has complied with its applicable
mine whether NYU would be compelled to produce the discovery obligations; several note that counsel’s duty
requested data. First, the court considered whether the does not end with the issuance of a litigation hold.18
data sought is recoverable. Second, the court engaged Courts have also held that counsel cannot rely on a
in a cost-benefit analysis to determine if the needs of client’s assertions that all relevant information has been
preserved and collected. Therefore, counsel may seek
the case warrant the data sought. Here, the court re-
to work directly with IT personnel, many times below
manded the case to determine whether the data sought the rank of CIO, CISO, manager or director, who have
was accessible and whether it could be produced with- direct responsibility for and knowledge of relevant data
out undue burden or cost. sources, both on site and in the cloud.
Although ephemeral data is certainly not necessary For example, in a data security matter, counsel may
in most cases, considering the high costs associated wish to speak directly with the person who manages the
with these forensic discovery methods, parties should firewall logs. Then, Counsel and IT professionals might
consider whether ephemeral data is a source of infor- work in tandem to assist with the implementation of the
litigation hold by identifying the appropriate custodians
mation that might be responsive to a given litigation.16
and other data sources.
14
Nacco Materials Handling Group v. Lilly Company, 278 Case Law. In the seminal case on this subject, Zubu-
F.R.D. 395 (W.D. Tenn. 2011). lake v. UBS Warburg,19 Judge Shira Scheindlin im-
15
Tener v. Cremer, 931 N.Y.S.2d 552 (N.Y. App. Div. 2011); posed sanctions on UBS for failing to guarantee that
(see 12 DDEE 164, 4/26/12). relevant data was both preserved and produced. Ac-
16
If so, consideration should be given to admissibility is- cordingly, she granted plaintiff’s motions for sanctions
sues, particularly with respect to foreign ephemeral data, such in the form of an adverse inference instruction with re-
as whether such data can be authenticated and/or would fall
under the Business Records exception to the hearsay rule, FRE
17
803(6). The Sedona Conference Commentary on ESI Admissi- Again, as noted above, foreign data privacy laws may op-
bility notes, for example, that when a system, instead of an in- erate to restrict the ability to collect and preserve foreign
dividual, is the creator or owner of a document or file, as when ephemeral data, by for example, limiting that which is col-
logs of certain activity are automatically created by an operat- lected to ‘‘that which is relevant and not excessive to a particu-
ing system, it may be difficult to provide foundational evidence lar investigation.’’ Directive 95/46, supra, note 6, at art. 6; E.U.
as to how and where the data was created. The Sedona Con- Working Document, supra, note 6 at 10.
18
ference Commentary on ESI Evidence and Admissibility at 11- Under FRCP 26(g), counsel of record must certify that all
12. (March 2008), available at https:// disclosures are correct and complete and sanctions may be im-
thesedonaconference.org/publication/The%2520Sedona% posed for non-compliance with this requirement.
19
2520Conference%25C2%25AE%2520Commentary%2520on% Zubulake v. UBS Warburg, LLC, 229 F.R.D. 422 (S.D.N.Y
2520ESI%2520Evidence%2520%252526%2520Admissibility. 2004) (‘‘Zubulake V’’).
5
spect to deleted emails and the imposition of certain sibilities. Searching for an answer on Google (or Westlaw
costs. or Lexis) is very different from searching for all responsive
In reaching this conclusion, the Zubulake court dis- documents in the FOIA or e-discovery context.23
cussed counsel’s role in ensuring the identification, Judge Scheindlin concluded that it was unnecessary
preservation, and production of relevant information: for her to evaluate the adequacy of defendants’ search
A party’s discovery obligations do not end with the imple- terms. Instead, she ordered the parties to work coopera-
mentation of a ‘‘litigation hold’’—to the contrary, that’s only tively ‘‘to agree on additional search terms and
the beginning. Counsel must oversee compliance with the protocols—and, if necessary, testing to evaluate and re-
litigation hold, monitoring the party’s efforts to retain and fine those terms. . . . Defendant agencies, in turn, will
produce the relevant documents. Proper communication need to cooperate fully with plaintiffs.’’24
between a party and her lawyer will ensure (1) that all rel- National Day Laborer Org. Network deals with a
evant information (or at least all sources of relevant infor-
number of relevant current E-Discovery issues, includ-
mation) is discovered, (2) that relevant information is re-
tained on a continuing basis, and (3) that relevant non- ing the adequacy of keyword search terms and
privileged material is produced to the opposing party.20 technology-assisted review. Importantly, from the per-
spective of a CIO, it also demonstrates the technologi-
In a recent decision, Nat’l Day Laborer Org. Network cal expertise required to assure the adequacy of a col-
v. United States Immigration & Customs Enforcement lection effort.
Agency,21 Judge Scheindlin strongly admonished A recent case in the Eastern District of Texas is in-
against the practice of ‘‘self collection,’’ whereby par- structive on this point. In Green v. Blitz,25 plaintiff
ties allow individual employees to search for relevant brought a product liability suit against a manufacturer
data and applied that caution against government agen- of gasoline cans, alleging that a gas can had caused her
cies in the context of a Freedom of Information Act husband’s death. Her main theory of liability was pre-
(‘‘FOIA’’) case. Nat’l Day Laborer Org. Network dealt mised on the fact that the can had no flame arrester,
with plaintiffs’ requests for documents under the FOIA which defendants had not incorporated and which they
and their assertions that defendant government entities’ argued did not work anyway.
searches for such information were inadequate. Al- During the jury deliberations, the parties entered into
though this case involves an analysis of the reasonable- a high-low settlement agreement under which the par-
ness of such search efforts under the FOIA, Judge ties agreed that, notwithstanding the verdict, the case
Scheindlin’s opinion is applicable in the broader con- would be dismissed with prejudice and a release ex-
text of E-Discovery. ecuted. The jury ultimately ruled in the plaintiff’s favor,
Plaintiffs charged that the search efforts of the FBI but under the parties’ agreement, the amount awarded
and Department of Homeland Security were unreason- triggered a payment by the defendant on the low end of
able. In the context of the FOIA, the court found that the the scale.
scope of some of the government’s searches were ad- Subsequently, counsel for plaintiff was involved in
equate while some were not, noting that it is ‘‘impos- related litigation during which he learned of ‘‘extremely
sible to evaluate the adequacy of an electronic search relevant and material’’ documents that had not been
for records without knowing what search terms have produced in the Green case. Plaintiff filed a motion to
been used.’’22 Disagreeing with defendants’ argument re-open the case and for sanctions arguing that defen-
that custodians should be trusted to run effective dant’s discovery violations were in bad faith and had
searches of their own (i.e., ‘‘self-collect’’), the court led to an unfair outcome.
stated: The court held that the defendant had committed sev-
There are two answers to defendants’ question. First, cus- eral discovery abuses, including failing to investigate,
todians cannot ‘‘be trusted to run effective searches,’’ with- search for, and produce relevant evidence, and never is-
out providing a detailed description of those searches, be- suing a written litigation hold order. The sanctions were
cause FOIA places a burden on defendants to establish that both steep and creative: The court ordered the defen-
they have conducted adequate searches; FOIA permits dant to pay plaintiff $250,000 in sanctions, to disclose
agencies to do so by submitting affidavits that ‘‘contain rea- the court’s order to plaintiffs ‘‘in every lawsuit proceed-
sonable specificity of detail rather than merely conclusory ing against it,’’ and to disclose the order in any case in
statements.’’ Defendants’ counsel recognize that, for over which defendant is a party for the next five years (a
twenty years, courts have required that these affidavits ‘‘set
[ ] forth the search terms and the type of search per-
$500,000 sanction would be returned upon compli-
formed.’’ But, somehow, DHS, ICE, and the FBI have not ance).
gotten the message. So it bears repetition: the government What was the nature of the defendant’s failings in
will not be able to establish the adequacy of its FOIA Green that warranted these extreme sanctions? Accord-
searches if it does not record and report the search terms ing to the court, in responding to discovery, the em-
that it used, how it combined them, and whether it searched ployee ‘‘solely responsible for searching for and collect-
the full text of documents. ing documents relevant to the litigation,’’26 and who
The second answer to defendants’ question has emerged also ‘‘headed up the research and investigation around
from scholarship and case law only in recent years: most flame arresters,’’27 failed to take several steps neces-
custodians cannot be ‘‘trusted’’ to run effective searches be- sary to discharge a duty to preserve and investigate re-
cause designing legally sufficient electronic searches in the sponsive documents: No litigation hold was issued, key
discovery or FOIA contexts is not part of their daily respon-
23
Id. at *1113.
20 24
Id. at 432. Id. at *1315.
21 25
Nat’l Day Laborer Org. Networkv. United States Immi- Green v. Blitz, 2011 BL 52006 (E.D. Tex, Mar. 1, 2011);
gration and Customs Enforcement Agency, 2012 BL 176257 (11 DDEE 168, 4/14/11).
26
2012 WL 2878130 (S.D.N.Y. July 13, 2012). Id. at *94.
22 27
Id. at *1011. Id.
DIGITAL DISCOVERY & E-EVIDENCE REPORT ISSN 1941-3882 BNA 12-6-12

6
word searches were not run (a search for ‘‘flame ar- tion with IT professionals can be in the context of
rester’’ was not run and would have garnered respon- E-Discovery.
sive information as it was in the subject line of emails),
and the individual did not contact defendant’s IT per- III. E-Discovery Tips for the IT Professional
sonnel to coordinate with the search and extraction of
relevant data. Making matters worse, this employee ad- In conclusion, the following considerations may
mitted that he was ‘‘about as computer literate-illiterate prove useful in managing the E-Discovery issues posed
as they get.’’28 by big data:
In light of the undisclosed documents, the court 1. Consider anticipating E-Discovery issues in ad-
found that defendant’s conduct was willful and plaintiff vance of litigation, taking precautions such as mapping
was prejudiced. The failure to search relevant data was the location and/or filepaths of relevant data (wherever
compounded where no coordination with IT occurred located, e.g., on-site, in the cloud, etc.)
and thus preservation was not ensured. Indeed, accord- 2. Consider reviewing your cloud storage vendor and
ing to the opinion, a lead member in IT had repeatedly subcontractor contracts and ensure that the appropriate
requested that employees delete their emails during contractual obligations are in place with respect to how
‘‘the precise time period’’ that defendant ‘‘was actively and by whom stored data are to be preserved, collected,
defending multiple products liability suits relating to its and produced.
gas cans and had a duty to preserve evidence.’’29
3. Consider the above questions concerning server
The court also addressed defendant’s failure to halt
consolidation with appropriate counsel, along with
the overwriting of back up tapes, concluding that ‘‘it
other issues counsel may raise.
will never be known how much prejudice against the
plaintiff was actually caused by Blitz’s failure to pre- 4. Consider if education among IT staff is necessary
serve documents.’’30 in conjunction with the legal department’s needs.
While obviously an extreme factual scenario, Green 5. Plan early for preservation in countries where rel-
should serve as an example of how valuable coordina- evant data reside, taking into account local laws that
may affect implementation of litigation holds.
28
Id. at *7. 6. Document data collection, review and chain of
29
Id. at *9 custody, particularly for data in the cloud and outside
30
Id. at *10 the U.S., in order to lay a foundation for admissibility.
Fernando M. Pinguelo is the Chair of the Cyber Security & Data Protection Group at Norris McLaughlin
& Marcus; Shannon Capone Kirk is E-Discovery Counsel at Ropes & Gray; Sara Johnson Meyers is a Senior
Discovery Attorney at Ropes & Gray. The views set forth in this article are those of the authors alone and
do not reflect the views of their respective firms or clients. Nothing in this article should be viewed as attor-
ney advice. Any individuals seeking advice on the matters presented herein should seek individual and spe-
cific advice, as facts and application of law change for every matter. The authors may be contacted as fol-
lows: fernando@CyberJurist.com, shannon.kirk@ropesgray.com, sara.meyers@ropesgray.com.
The Sedona Conference® Commentary
on Legal Holds: The Trigger & The Process

Information is the lifeblood of the modern world, a fact that is at the core of our litigation discovery system. The law has
developed rules regarding the manner in which information is to be treated in connection with litigation. One of the
principal rules is that whenever litigation is reasonably anticipated or pending against an organization – or an organization
contemplates initiating litigation – that organization has a duty to take reasonable steps to preserve relevant information.
The duty to preserve information includes an obligation to identify, locate, and maintain information that is relevant to
specific, predictable, and identifiable litigation. When preservation of electronically stored information (“ESI”) is required,
the duty to preserve supersedes records management policies that would otherwise result in the destruction of ESI. A “legal
hold” program defines the processes by which information is identified, preserved, and maintained when it has been
determined that a duty to preserve has arisen.
The basic principle that an organization has a duty to preserve relevant information in anticipation of litigation is easy to
articulate. However, the precise application of that duty can be elusive. Every day, organizations apply the basic principle to
real-world circumstances, confronting the issue of when the obligation is triggered and, once triggered, what is the scope of
the obligation. Recent court decisions, most notably Pension Committee v. Banc of America Securities, LLC, 685 F. Supp. 2d
456 (S.D.N.Y. Jan. 15, 2010) and Rimkus Consulting v. Cammarata, 688 F. Supp. 2d 598 (S.D. Tex. Feb. 19, 2010), illustrate
the importance of understanding preservation obligations on the consequences of ignoring them.
This updated 2010 Commentary is intended to provide guidance on those issues and is divided into two parts: the “trigger”
and the “process.” Part I addresses the trigger issue and provides practical guidelines for making a determination as to when
the duty to preserve relevant information arises. What should be preserved and how the preservation process should be
undertaken including the implementation of legal holds is addressed in Part II. The guidelines are intended to facilitate
reasonable and good faith compliance with preservation obligations. The guidelines are meant to provide the framework an
organization can use to create its own preservation procedures. In addition to the guidelines, suggestions as to best practices
are provided along with several illustrations as to how the guidelines and best practices might be applied under hypothetical
factual situations.
Guideline 1: A reasonable anticipation of litigation arises when an organization is on notice of a credible probability
that it will become involved in litigation, seriously contemplates initiating litigation, or when it takes
specific actions to commence litigation.
Guideline 2: Adopting and consistently following a policy or practice governing an organization’s preservation
obligations are factors that may demonstrate reasonableness and good faith.
Guideline 3: Adopting a process for reporting information relating to a probable threat of litigation to a responsible
decision maker may assist in demonstrating reasonableness and good faith.
Guideline 4: Determining whether litigation is or should be reasonably anticipated should be based on a good faith
and reasonable evaluation of relevant facts and circumstances.
Guideline 5: Evaluating an organization’s preservation decisions should be based on the good faith and reasonableness
of the decisions undertaken (including whether a legal hold is necessary and how it should be executed) at
the time they are made.
Guideline 6: The duty to preserve involves reasonable and good faith efforts, taken as soon as is practicable and applied
proportionately, to identify and, as necessary, notify persons likely to have relevant information to preserve
the information.
p 1
The Sedona Conference® Commentary
on Legal Holds: The Trigger & The Process

Guideline 7: Factors that may be considered in determining the scope of information that should be preserved include
the nature of the issues raised in the matter, the accessibility of the information, the probative value of the
information, and the relative burdens and costs of the preservation effort.
Guideline 8: In circumstances where issuing a legal hold notice is appropriate, such a notice is most effective when the
organization identifies the custodians and data stewards most likely to have relevant information, and
when the notice:
a) Communicates in a manner that assists persons in taking actions that are, in good faith,
intended to be effective
b) Is in an appropriate form, which may be written
c) Provides information on how preservation is to be undertaken
d) Is periodically reviewed and, when necessary, reissued in either its original or an amended form,
and
e) Addresses features of relevant information systems that may prevent retention of potentially
discoverable information.
Guideline 9: An organization should consider documenting the legal hold policy, and, when appropriate, the process of
implementing the hold in a specific case, considering that both the policy and the process may be subject
to scrutiny by opposing parties and review by the court.
Guideline 10: Compliance with a legal hold should be regularly monitored.
Guideline 11: Any legal hold policy, procedure, or practice should include provisions for releasing the hold upon the
termination of the matter at issue so that the organization can adhere to policies for managing
information through its useful lifecycle in the absence of a legal hold.
Copyright © 2010, The Sedona Conference®

Reprinted courtesy of The Sedona Conference®
The full text of this document is available free for personal use from The Sedona Conference®
website at www.thesedonaconference.org.
p 2
Case 3:07-cv-05165-FLW -TJB Document 223 Filed 01/28/11 Page 1 of 36 PageID: 4275
UNITED STATES DISTRICT COURT

DISTRICT OF NEW JERSEY
MEDEVA PHARMA SUISSE A.G., et al., Civil Action No. 07-5165 (FLW)
Plaintiffs,
v. MEMORANDUM OPINION
ROXANE LABORATORIES, INC.
Defendant.
BONGIOVANNI, Magistrate Judge
This matter comes before the Court upon Plaintiffs Medeva Pharma Suisse A.G., Warner
Chilcott Pharmaceuticals Inc. and Warner Chilcott Company, LLC’s (collectively, “Medeva”)
motion to preclude Defendant Roxane Laboratories, Inc. (“Roxane”) from disputing the location
and duration of release of mesalamine from Roxane’s generic product and Asacol in the human
body. The Court has fully reviewed and considered all of the papers submitted in support of and
in opposition to Medeva’s motion as well as the arguments set forth by counsel during the
November 23, 2010 hearing. For the reasons set forth more fully below, Medeva’s motion is
GRANTED in part and DENIED in part.
I. Procedural History and Background
Medeva filed the instant Hatch-Waxman patent infringement case against Roxane on
October 26, 2007 after Roxane filed an Abbreviated New Drug Application (“ANDA”) and
Paragraph IV Certification and Notice letter seeking approval to commercially manufacture and
market a generic version of Medeva’s Asacol product, which is marketed under United States
Patent No. 5,541,170 (the “‘170 patent”) to treat mild to moderate ulcerative colitis. The ‘170
patent is owned by Plaintiff Medeva Pharma Suisse A.G. Prior to October 30, 2009, the ‘170
patent was licensed exclusively to Proctor & Gamble Pharmaceuticals, Inc. (“P&G”), now
known as Warner Chilcott Pharmaceuticals Inc. On October 30, 2009, Warner Chilcott
Company, LLC acquired P&G. Warner Chilcott Company, LLC is now the exclusive licensee of
the ‘170 patent. The ‘170 patent relates to the formulation of the active ingredient mesalamine,
by which a core of mesalamine is specially coated for release to the afflicted gastrointestinal
tissues. The ‘170 patent claims a tablet formulation of mesalamine which includes an acrylic-
based resin coating that releases the entire dose of mesalamine “to the right side of the colon” so
that it acts topically to relieve symptoms. The infringement issue in this matter relates to whether
Roxane’s proposed generic product releases its mesalamine dose in accordance with the claim.
The validity issues concern whether the claimed invention would have been obvious to a person
of ordinary skill in the art.
At issue in the present motion is Medeva’s claim that Roxane is responsible for the
spoliation of evidence. Specifically, Medeva argues that Roxane (1) destroyed highly-relevant
documents concerning Roxane’s product development efforts as well as its testing to determine
where and when Asacol releases mesalamine and (2) manufactured a “secret batch” of its generic
product in order to conduct testing in humans after persuading the Court to preclude Medeva
from conducting the same kind of testing without prior Court approval. Both of these arguments
as well as Roxane’s opposition thereto are discussed in more detail below. As a result of the
alleged spoliation, Medeva requests that sanctions be imposed. Initially, Medeva sought to
preclude Roxane from disputing the location and duration of release of mesalamine from
Roxane’s generic product and Asacol in the human body. Medeva then expanded its request for
2
sanctions to include additional and lesser sanctions such as the spoliation inference, reopening
discovery, waiver of the attorney-client privilege and the imposition of attorney’s fees and costs.
A. The Destruction of Documents
1. Medeva’s Position
Medeva claims that Roxane impermissibly destroyed relevant documents after it
anticipated litigation and was under a duty to preserve same. In this regard, Medeva argues that
Roxane had a duty to preserve documents as early as July 31, 2001, which corresponds to the
date of an entry on Roxane’s privilege log that asserts work product protection; Medeva notes
that the work product doctrine does not apply unless the document at issue was created in
anticipation of litigation. Further, Medeva claims that Roxane certainly had a duty to preserve
documents no later than February 2002 as the evidence establishes that Roxane knew at that time
that it intended to file an ANDA with a Paragraph IV Certification regarding its generic product.
Nevertheless, Medeva claims that Roxane failed to issue a document preservation notice (i.e., a
litigation hold) until July 2007.
Additionally, Medeva claims that Roxane’s normal document retention policies did not
sufficiently protect against spoliation. Indeed, Medeva argues that Roxane’s normal document
retention policies confirm that Roxane breached its duty to preserve documents because they
show that documents were actually destroyed in 2002 and thereafter. Further, Medeva argues
that the testimony obtained in this case also confirms that Roxane had annual file clean-up days
3
during which documents were destroyed.1 Medeva therefore contends that Roxane failed to take
adequate steps to prevent spoliation between July 2001 and July 2007.
Further, Medeva argues that the evidence establishes that relevant documents were
actually destroyed. For example, Medeva claims that Roxane destroyed documents relating to its
first pharmacokinetic study involving Asacol (the “Anapharm study”). Medeva claims that
Roxane failed to produce (1) the protocol for the Anapharm study, which Roxane’s Director of
Regulatory and Medical Affairs testified would have shown Roxane’s objectives and hypotheses
for the study and which Roxane confirmed no longer exists; (2) Dr. Kramer’s (Roxane’s
consultant) final report from the Anapharm study and (3) any subsequent correspondence
regarding the study, despite the fact that Dr. Kramer sent an email to Roxane asking whether any
changes regarding the study should be made. Moreover, Medeva argues that Roxane cannot shift
its blame for failing to preserve documents related to the Anapharm study by arguing that
Medeva could have, but did not subpoena Anapharm for the missing information. Medeva
argues that Roxane had an independent responsibility to preserve the missing information.
Likewise, Medeva claims that it was not in a position to pursue discovery related to the
Anapharm study because it was not listed in Roxane’s Rule 26(a) disclosures, it is only
1
Medeva also argues that Roxane’s reliance on a document retention policy which
established that documents such as laboratory notebooks, technical reports and developmental
and analytical records were preserved for at least 20 years is inadequate because that policy is
dated May 28, 2008. As a result, Medeva contends that Roxane cannot establish that it took
reasonable steps to preserve all relevant information between July 2001 and July 2007 when
Roxane finally issued a litigation hold. Roxane, however, subsequently produced document
retention policies dating back to 1998, which show that it was its normal business practice to
retain technical information for 30 years. Medeva, however, notes that none of the document
retention policies relied on by Roxane specifically reference Roxane’s generic mesalamine
product.
4
referenced in a single email, Roxane’s witnesses did not mention it in their testimony, Roxane
did not produce its research agreement with Anapharm and Roxane would not admit that the
study existed. Indeed, Medeva claims that by the time it learned of the Anapharm study, it was
the fall of 2009 and fact discovery had closed. As such, Medeva claims that it did not have the
opportunity to subpoena Anapharm and that even if it were granted leave to do so out of time, it
did not believe that it would get a response from Anapharm in time to make use of it here.
In addition, Medeva argues that Roxane has only produced several documents in
traditional paper file form and has failed to produce the corresponding electronic versions of
these emails and documents. Such documents relate to analysis of pharmacokinetic studies
involving Roxane’s prototype formulations as well as correspondence regarding the design of
Roxane’s clinical study created to test the alleged clinical effectiveness of Roxane’s generic
product against both Asacol and a placebo. Specifically, Medeva takes issue with Roxane’s
production of electronic documents and emails from several “key custodians.” (Medeva Reply
Br. at 4). For example, Medeva claims that no emails or other electronic documents of any kind
were collected from Eric Spiller, Roxane’s Associate Director of Product Development and the
“project leader” responsible for developing Roxane’s generic product. During oral argument,
Medeva clarified that it is not arguing that Roxane should have produced documents in both
paper and electronic format, but instead, that the lack of electronic documents establishes that
information was destroyed.
Medeva also argues that Roxane failed to produced signed final reports for studies related
to Roxane’s development and testing of its generic product. Medeva contends that it knows that
these signed final reports exist because it obtained them from third parties, even though Roxane
5
failed to produce same. Similarly, Medeva argues that discovery obtained from Quintiles, a
contractor retained by Roxane to conduct a clinical study, supports the fact that Roxane destroyed
documents. In this regard, Medeva points out that Quintiles, but not Roxane, produced a number
of emails sent between Quintiles and Roxane employees between 2005 and 2006 that concerned
Roxane’s evaluation of U.K. Asacol as well as Roxane’s representations to foreign regulatory
authorities concerning the formulation characteristics of its tablets. Medeva claims that because
Roxane did not suspend its routine destruction of documents, there is no way of knowing what
other documents and emails were destroyed before Roxane implemented its litigation hold.
Medeva also argues that what is clear form the sheer number of destroyed documents is that
Roxane is liable for spoliation.
Further, Medeva contends that the destroyed documents are relevant, indeed critical, to
this litigation and that it has been prejudiced by their loss. With respect to relevance, Medeva
argues that the destroyed documents are relevant to (1) Roxane’s efforts to develop its
formulation; (2) Roxane’s claims of non-infringement; and (3) Roxane’s challenge to the validity
of the ‘170 patent. Indeed, Medeva argues that the destroyed documents bear on critical issues
such as whether Roxane’s generic product releases mesalamine to the right side of the colon and
whether Asacol releases mesalamine to the right side of the colon, a fact that Roxane has claimed
is central to its defense of this matter. For example, Medeva contends that the Anapharm study,
which Medeva claims evaluated the release of Asacol in the gastrointestinal tract, is obviously
relevant to Roxane’s allegations that Asacol is not a commercial embodiment of the ‘170 patent
and Roxane’s claims of non-infringement.
6
With respect to prejudice, Medeva claims that the fact that relevant evidence was
destroyed establishes prejudice. Indeed, Medeva argues that all it has to show to establish
prejudice is that its ability to effectively prepare a full and complete trial strategy has been
impeded. Medeva argues that its ability to effectively prepare such a trial strategy has in fact
been impeded because Roxane destroyed documents that concern the location of the release of
mesalamine as well as Roxane’s development of its generic product. Given the destruction,
Medeva does not have access to the missing documentary evidence and has been prevented from
questioning Roxane’s witnesses regarding same. What is more, Medeva claims that the
destroyed documents appear to support the conclusion that Asacol releases at the terminal ileum,
as claimed by the ‘170 patent. Further, Medeva claims that the destruction of the aforementioned
documents is prejudicial because Roxane keeps altering its position regarding the location of
where its generic product releases in the body. Medeva claims that because the cite of release
has been a moving target, it is prejudiced by the destruction of documents that bear on where and
how Roxane designed its product to release in the gastrointestinal tract (i.e., the types of
documents that would shed the most light on Roxane’s design decisions and testing of its product
and Asacol).
In addition, Medeva argues that the prejudice imposed on it has been increased because,
not only were relevant documents destroyed, but Roxane sought to evade discovery through
deposition testimony and interrogatory responses on the subject matter contained in the destroyed
documents. For example, Medeva claims that despite the fact that the FDA was interested in
where Roxane’s generic product releases in the body and despite the fact that Roxane made
statements in documents submitted to the FDA regarding where its product releases mesalamine,
7
Roxane refused to produce discovery regarding where its generic product releases mesalamine.2
Indeed, Medeva argues that no deponent, including Roxane’s Rule 30(b)(6) witness, was able to
testify regarding where Roxane’s generic product releases mesalamine in the gastrointestinal tract
or the basis for Roxane’s characterization of its generic product’s release characteristics that is
found in Roxane’s ANDA, proposed labeling, Paragraph IV Notice letter or patent application.
In addition, Medeva claims that Roxane’s supplemental answers to Medeva’s
interrogatories were insufficient. For example, Medeva’s Interrogatory No. 11 sought details
regarding the release of mesalamine from Roxane’s generic product within the human
gastrointestinal tract. Roxane’s response pointed to its Paragraph IV Notice letter and its
employee Vinay Shukla as sources of responsive information. Medeva contends that Roxane’s
response is insufficient because, as previously stated, Roxane’s witnesses testified that Roxane
had no knowledge of the factual bases for the assertions contained in Roxane’s Paragraph IV
Notice letter and Mr. Shukla specifically testified that he was not the expert in the in vivo area as
to where Roxane’s generic product releases and explicitly refused to answer questions about
Roxane’s Paragraph IV Notice letter that addressed Roxane’s generic product’s release
characteristics.
Given the alleged magnitude of Roxane’s misconduct, Medeva argues that the sanction of
preclusion is justified even if such a sanction is tantamount to dismissal of Roxane’s claim. As
previously stated, Medeva argues that Roxane not only destroyed documents but sought to evade
other efforts to obtain the information contained in the destroyed documents. Moreover, Medeva
2
Such documents include Roxane’s proposed labeling submitted with its ANDA, its
ANDA, its Paragraph IV Notice letter and its revised proposed labeling regarding where its
product releases mesalamine
8
claims that Roxane, whose business model is to challenge the intellectual property rights of
innovative drug companies, knew what it was doing when it failed to issue a timely litigation
hold and destroyed relevant evidence because Roxane’s. Medeva, therefore, asks the Court to
sanction Roxane by precluding Roxane from disputing where its generic product releases and
where Asacol releases in the human body. Medeva also argues that if the Court were to consider
additional and/or lesser remedies, then the following may be be appropriate: (1) imposition of
adverse inferences regarding where Roxane’s generic product and Asacol release; (2) reopening
of fact discovery and requiring Roxane to restore all back-up tapes from 2001 through the
present; (3) finding that Roxane’s conduct constitutes a waiver of work product protection from
2001 through 2007; and (4) imposition of all fees and costs associated with the instant motion to
preclude as well as those incurred by Medeva’s experts in testing and preparing reports on Lot
No. 089226.
2. Roxane’s Position
Roxane opposes Medeva’s motion to preclude as well as its request for additional and/or
lesser remedies arguing that there has been no spoliation of evidence. In the first instance,
Roxane claims that no documents were actually or intentionally destroyed. Roxane argues that
from June 2002 forward it had a formal document retention policy in place that required
laboratory notebooks, technical reports and developmental and analytical records to be retained
for 20 years. Indeed, Roxane argues that from 1998-2004, its document retention policy required
Roxane to preserve technical information for at least 30 years. Further, Roxane claims that its
document retention policy did not permit documents to be purged without safeguards; instead, on
the file clean-up days that occurred under Roxane’s document retention policies, employees paid
9
close attention to the document retention policies to ensure that no technical information,
including all such information related to mesalamine, would have been destroyed. In addition,
Roxane contends that the evidence shows that no documents relating to mesalamine were
destroyed. In this regard, Roxane notes that its Rule 30(b)(6) witness on document retention
issues explicitly testified that any and all documents concerning mesalamine were not destroyed.
Roxane also argues that the litigation hold put in place in 2007 “merely modified the ongoing
document retention policy to suspend any potential destruction of documents.” (Roxane Opp.
Br. at 4). Roxane claims that it would be unreasonable to have required Roxane to institute a
litigation hold in 2001, four years before its first ANDA product was ever manufactured.
Moreover, Roxane claims that such a litigation hold would only have continued the preservation
of documents already being preserved under Roxane’s document retention policies.
With respect to the allegedly destroyed documents identified by Medeva, Roxane argues
that the documents are not critical to this matter and also claims that Medeva has failed to show
that Roxane actually and intentionally destroyed same. For example, with respect to the
Anapharm study, Roxane argues that while witnesses testified that a protocol for the study must
have existed, nothing in the record establishes that Roxane itself had the protocol. Further,
Roxane notes that while a protocol describing how the Anapharm study would be performed was
not produced, Roxane did produce other documents relating to the study, including Dr. Kramer’s
principal and supplemental reports, which detail how the Anapharm study was actually carried
out. Moreover, Roxane claims that there is no evidence that Dr. Kramer prepared other
documents related to the study or that the study results were ever memorialized in any other
format.
10
In addition, Roxane claims that the Anapharm study is not relevant to this litigation, and
certainly not of critical importance, because it does not relate to Roxane’s generic product or a
prototype thereof, but instead relates to Medeva’s Asacol product and Roxane never asked Dr.
Kramer to examine where Asacol releases mesalamine in the gastrointestinal tract. Further,
Roxane argues that Medeva’s conduct establishes that the Anapharm study is not important to
this litigation. In this regard, Roxane notes that Medeva did not pursue discovery on this study
directly from Anapharm. Roxane also notes that in its deposition of Dr. Kramer, Medeva asked
almost nothing about the Anapharm study and did not even inquire into the existence of a
protocol. Further, Roxane claims that despite the fact that Medeva had the results of the
Anapharm study and could have had its experts analyze same, it did not.
Roxane also contends that it was not obligated to make a duplicate production of
documents that were produced in the traditional paper file format. Indeed, Roxane argues that it
was not required to produce both paper and electronic versions of documents and emails.
Consequently, Roxane argues that the fact that certain documents and emails were only produced
in paper format and were not electronically produced does not support the conclusion that these
documents and emails were destroyed. In addition, with respect to Eric Spiller’s files, Roxane
claims that the evidence shows that Mr. Spiller maintained all of his mesalamine related
documents in a specific file and that this file was produced to counsel. Indeed, Roxane argues
that there is no testimony that either Mr. Spiller or any other employee ever deleted or destroyed
a single document concerning mesalamine.
More importantly, Roxane claims that there is no evidence that the allegedly missing
documents are important to this case. For example, Roxane argues that there is no evidence that
11
the allegedly missing product development documents would have contained information about
where either Roxane’s generic product or Asacol releases. Further, Roxane notes that there
could have been no such documents concerning its generic product until April 2005 because
Roxane’s generic product was not manufactured until that time. Similarly, Roxane claims that
the allegedly missing documents concerning Roxane’s prototype formulations are not relevant to
this matter and would not have contained information regarding the release of either its generic
product or Asacol.
Roxane also argues that the record does not support Medeva’s contention that Roxane
ever possessed the allegedly missing final study reports related to Roxane’s development and
testing of its generic product that Medeva obtained from Quintiles. Indeed, Roxane notes that
Elizabeth Ernst, Roxane’s Director of Regulatory Medical, testified that she did not recall ever
receiving the final study reports. In addition, with respect to the remainder of the Quintiles
production, Roxane claims that the “vast majority” of same is made up of raw data generated
from a clinical study that was conducted in a dozen countries and which was never forwarded to
Roxane. (Roxane Opp. Br. at 6). Roxane claims that the fact that a “stray email or two” was
produced by Quintiles, but not Roxane, does not support Medeva’s claim of spoliation. (Id.)
In addition, Roxane argues that it did not violate any discovery orders entered by this
Court, nor did it engage in discovery misconduct. For example, Roxane claims that it did not
evade discovery on where its generic product releases mesalamine; instead, it simply did not
know the answer to Medeva’s questions. Roxane contends that prior to expert discovery it had
no information regarding where its generic product releases mesalamine because it never
conducted any experiment or study that would provide such information as it did not design its
12
product to target a specific site of action. Indeed, Roxane claims that the specific site of release
was not important to its development of its generic product or to the FDA’s evaluation of same.
Instead, Roxane claims that the only matter of importance was whether Roxane’s generic product
showed equivalent efficacy to Asacol. Roxane claims that the fact that certain documents
submitted to the FDA comment on where its generic product releases mesalamine are of no
moment because the statements were not purposefully made. For example, Roxane contends that
the fact that its original draft label submitted to the FDA contained the statement “delays release
of mesalamine until it reaches the terminal ileum and beyond” is an artifact of the fact that
Roxane initially copied Asacol’s label verbatim. Roxane notes that it made amendments to the
label for its generic product as the application process proceeded. Similarly, Roxane argues that
its interrogatory responses regarding where its generic product releases mesalamine are not
deficient because they incorporate Roxane’s expert reports which describe in detail where its
product releases mesalamine. Further, Roxane argues that it cannot be held responsible for any
issues Medeva had in obtaining discovery from Quintiles because Quintiles is a third party and
Roxane was not involved in its production of documents.
Further, Roxane argues that Medeva has not established that it has been prejudiced by
Roxane’s failure to produce the allegedly missing documents. Indeed, Roxane claims that
Medeva does not explain how the missing documents would impede its ability to put on its
infringement and validity case. Roxane argues that that is because Medeva is fully informed on
Roxane’s efforts to develop its generic product and both Roxane’s claims of non-infringement as
well as its challenge to the validity of the ‘170 patent. Roxane notes that previously in this
litigation, Medeva represented that it was ready to go to trial and that that ability has not been
13
impaired. In this regard, Roxane notes that no expert referenced the subject matter of the missing
documents in their opening expert reports and argues that Medeva has made no showing that the
missing documents, including the Anapharm protocol, would have been favorable to its case or
that they would have been any more useful than the information already produced.
As a result, Roxane argues that there is no basis for the Court to enter sanctions against it.
First, Roxane argues that preclusion is an extreme sanction that is only granted in rare
circumstances that do not exist here. Consequently, Roxane claims that it should not be
precluded from disputing the location and duration of release of mesalamine from either its
generic product or Asacol in the human body.
Second, Roxane claims that the adverse inferences sought by Medeva, that Roxane
destroyed documents that established that Roxane’s generic product releases to the right side of
the colon and that Roxane destroyed documents that established that Asacol releases to the right
side of the colon, bear no relation to the content of the allegedly missing documents and are not
warranted. For example, Roxane claims that the missing Anapharm protocol cannot speak to
where Roxane’s generic product releases because the study was conducted on Asacol. Further,
Roxane argues that it would be unfair to use the missing Anapharm protocol to support an
adverse inference regarding Asacol because there is no evidence that Roxane possessed the
protocol or that it intentionally destroyed it. Roxane argues that this is particularly true given the
fact that it produced other information from the Anapharm study, including the test results.
Indeed, Roxane claims that there is no evidence in the record that the allegedly missing
documents might or would have been unfavorable to Roxane. Similarly, Roxane claims that
Medeva has not established that the missing documents have impeded its ability to put on either
14
its infringement or validity case. As a result, Roxane claims that there is no basis to impose
either of the requested adverse inferences on it.
Third, Roxane argues that there is no reason for the Court to reopen discovery or to
permit Medeva to obtain back-up tapes from Roxane. In this regard, Roxane claims that there
has been no spoliation of evidence and consequently Medeva has no need for more discovery.
Roxane also notes that Medeva’s request for additional discovery is insincere because Medeva
has claimed that additional discovery will not cure the alleged spoliation. Further, Roxane
argues that the parties entered a stipulation agreeing that back-up tapes not be produced and
Medeva’s request for the wholesale restoration of same is unfairly burdensome, unnecessary,
duplicative, time-consuming, expensive and would needlessly delay this matter and open the
door for additional discovery motions.
Fourth, Roxane argues that there is no basis for finding a wholesale waiver of Roxane’s
assertion of the attorney-client privilege and work product doctrine. Roxane claims that Medeva
has not challenged a single entry on Roxane’s privilege log and therefore it would be
inappropriate to find that Roxane’s conduct amounts to the broad waiver requested by Medeva.
In fact, Roxane argues that the only basis for a claim of waiver is Roxane’s nondisclosure and
failure to log a claim of privilege regarding the existence of Lot No. 089226. As a result, Roxane
argues that there is no reason to give Medeva “unfettered access” to all of Roxane’s documents
containing attorney work product and attorney-client communications. (Roxane’s Sur-Reply at
9). Moreover, as explained below in more detail, while Roxane denies that it waived work
product protection regarding Lot No. 089226, even if the Court were to find that such a waiver
15
occurred, no case law supports Medeva’s request that such a limited waiver supports the
wholesale production of all of Roxane’s protected information.
Finally, Roxane argues against the imposition of fees and costs. Roxane argues that the
imposition of fees and costs is not warranted because Roxane has not engaged in any improper
behavior. Roxane argues that it properly produced documents and that it properly invoked work
product to protect the existence of the Concealed Lot, which was made at the direction of outside
counsel. As a result, Roxane claims there is no need for the Court to punish Roxane or deter
future misconduct by imposing fees and costs.
B. The Concealment of Lot No. 089226
1. Medeva’s Position
Medeva argues that Roxane impermissibly concealed the existence of a batch of its
generic product, Lot No. 089266 (the “Concealed Lot”), that it manufactured in November 2008.
Roxane used the Concealed Lot to conduct gamma scintigraphy testing on humans and intended
to use the results of that testing as evidence of non-infringement.3 Medeva argues that it sought
discovery regarding all batches of mesalamine tablets manufactured by Roxane, including
samples thereof. Medeva claims that Roxane initially refused to provide Medeva with any
samples of its product and did so only after the Court entered an Order precluding Medeva from
conducting human testing using the samples produced. Despite Medeva’s request for all
samples, Medeva argues that Roxane only produced tablets from three lots, Lot No. 059013,
3
After this motion was fully briefed, Roxane learned that there was something wrong with
the Concealed Lot. The Concealed Lot was not manufactured properly and, as such, the tablets
did not work as Roxane intended. Consequently, while initially intending to rely on its experts’
testing of the Concealed Lot, Roxane is no longer doing so.
16
which pertained to expired tablets, and Lot Nos. 089050 and 079138, which pertained to
unexpired tablets; Roxane did not provide any tablets from the Concealed Lot.
Further, Medeva claims that despite having requested information regarding all samples
of Roxane’s mesalamine product, Roxane neither referenced the Concealed Lot in its discovery
responses nor explicitly claimed that information concerning same was being withheld based on
the attorney-client privilege or work product doctrine. For example, Medeva notes that
Interrogatory No. 5 directed Roxane to:
Identify by batch number, lot number, date, location, and number

any quantities of any Delayed Release Dosage Form containing
Mesalamine, including without limitation the Generic Product
made by or for You or on Your behalf, or any quantities of any
tablets, capsules, or other dosage form containing a coating
component or other agent intended for use with any Delayed
Release Dosage Form containing Mesalamine.
(Decl. of Paul A. Ainsworth in Support of Medeva’s Motion (“Ainsworth Decl.”), Ex. 32 at 3).
Medeva argues that neither Roxane’s response nor supplemental response to this interrogatory,
which follow, disclosed the existence of the Concealed Lot or claimed that any information
regarding same was being withheld based on the attorney-client privilege or work product
doctrine:
RESPONSE TO INTERROGATORY NO. 5
Roxane objects to this interrogatory as vague. To the extent

understood, Roxane also objects to this Interrogatory as overly
broad, unduly burdensome and not calculated to lead to the
discovery of admissible evidence in that it seeks information
concerning products and formulations other than the product
proposed in ANDA No. 79-073.
17
Subject to the foregoing objections and its General Objections,

Roxane states that information responsive to this interrogatory may
be ascertained from documents that will be produced pursuant to
Federal Rule of Civil Procedure 33(d).
SUPPLEMENTAL RESPONSE TO INTERROGATORY NO.

5
Roxane incorporates its original response and further responds as

follows: Information responsive to this Interrogatory may be
ascertained from documents already produced . . . . Roxane
reserves the right to supplement its response further at a future
time.
(Id. at 4) Medeva argues that the Concealed Lot represents Roxane’s ANDA product because
Roxane’s documents establish that these tablets were for “product development use” and
“manufactured to support in-vivo studies.” (Medeva Reply Br. at 25 (emphasis in original)). As
such, Medeva argues that Roxane should have disclosed the Concealed Lot.
In addition, Medeva argues that to the extent Roxane claims that the Concealed Lot and
information related thereto is protected by the work product doctrine, Roxane waived that
protection when it failed to include such a claim on its privilege log and when it did not assert
same in response to Medeva’s numerous direct questions regarding the existence of Roxane’s
tablets. Medeva also argues that, even if properly logged, the Concealed Lot and certain
information pertaining thereto would not be entitled to work product protection.
For example, Medeva claims that the mere existence of the Concealed Lot is not
protected by the work product doctrine and contends that the tablets themselves do not disclose
any attorney’s mental impressions. Medeva also argues that the fact that none of the documents
relating to the Concealed Lot refer to litigation and the fact that Roxane did not disclose the
existence of the Concealed Lot on its privilege log weigh against Roxane establishing that the
18
work product doctrine applies. In this regard, Medeva notes that in its documents concerning the
Concealed Lot: (1) Roxane described the testing of the Concealed Lot as being for “product
development purposes”; (2) Roxane never informed the Institutional Review Board/Independent
Ethics Committee (“IRB/IEC”), the entity that approved Roxane’s testing of the Concealed Lot
in humans, or the prospective study participants that the study would be used for litigation; and,
(3) when informing the IRB/IEC as well as prospective study participants about who may receive
the results of same, Roxane never identified this Court, Medeva, or Medeva’s experts as possible
recipients, despite having identified various other people and entities, including the FDA.4
Further, Medeva argues that the case law relied upon by Roxane to support their non-
disclosure of the Concealed Lot and information related thereto is either inapposite or supports a
finding in Medeva’s favor. First, Medeva claims that only two of the cases relied upon by
Roxane (Novartis Pharm. Corp. v. Abbott Lab., 203 F.R.D. 159 (D. Del. 2001) and Vardon Gold
Co. v. BBMG Golf Ltd., 156 F.R.D. 641 (N.D. Ill. 1994)) discuss product testing and in those
cases the issue was the disclosure of the testing methods and test results, not the actual product.
Moreover, Medeva argues that in six of the seven cases cited by Roxane, the non-disclosing party
raised timely privilege and work product objections to the disclosure of the concealed
information (see Medeva Reply Br. at 14, n.20), and in the remaining case, the court determined
that a waiver had occurred due to the party’s failure to timely raise a claim of privilege. (See Id.
at 14-15).
4
Medeva does not argue that Roxane should have disclosed its testing protocol or test
results related to the Concealed Lot prior to its experts’ reliance thereon. Medeva does, however,
claim that Roxane was obligated to disclose, at a minimum, the existence of the tablets.
19
Medeva also claims that the parties’ stipulation regarding expert discovery does not
justify Roxane’s nondisclosure of the Concealed Lot. The parties’ joint stipulation regarding
expert discovery provides that:
4. Pursuant to this agreement, the following categories

of data, information, or documents need not be disclosed by any
party, and are outside the scope of permissible discovery, except to
the extent required by ¶ 5, and except to the extent required by
FED .R.CIV .P. 26(a)(2):
***
b. draft reports, draft studies, or draft
workpapers; preliminary calculations, computations, or data runs;
or other preliminary or draft materials prepared by, for, or at the
direction of an expert witness[.]
(Decl. of David H. Silverstein in Support of Roxane’s Opp. to Medeva’s Motion (the “Silverstein
Decl.”), Ex. EE ¶ 4). Medeva argues that there was nothing “draft” or “preliminary” about the
final tablets manufactured by Roxane that make up the Concealed Lot and, as such, the parties’
stipulation did not protect their disclosure. In addition, Medeva claims that Roxane cannot shield
otherwise discoverable information by forwarding it to an expert witness. Further, to the extent
Roxane attempts to use the fact that Medeva drafted the parties’ joint stipulation to suggest that
Medeva should have more precisely drafted same to make clear that information like the
Concealed Lot was not protected by the stipulation, Medeva notes that Roxane also had an active
hand in drafting the stipulation and argues that Roxane, being the party who knew it would be
manufacturing the Concealed Lot, could have sought an express stipulation excluding such
tablets from the scope of permissible discovery.
Additionally, Medeva takes issue with the fact that Roxane conducted human testing
using the Concealed Lot. Medeva argues that based on ethical concerns regarding the
20
uncontrolled administration of an unapproved drug product, Roxane convinced the Court to
prohibit Medeva from conducting human testing using expired samples of Roxane’s generic
product produced in discovery. Medeva further claims that Roxane then insisted that Medeva
also agree not to conduct human testing with unexpired samples of its generic drug product
before producing same. Medeva argues that Roxane obtained the aforementioned prohibition
knowing full well that it intended to conduct human testing using the Concealed Lot. Medeva
contends that given the Court Order, Roxane’s testing in humans was inappropriate because the
same ethical considerations applied to Roxane as Medeva.
Finally, Medeva argues that it has been prejudiced by Roxane’s impermissible
withholding of the Concealed Lot. While Medeva acknowledges that some of the harm caused
by Roxane’s nondisclosure of the Concealed Lot has been alleviated, given Roxane’s admission
that the Concealed Lot is not representative of its ANDA product as well as Roxane’s decision
not to rely on its testing of same in this matter, Medeva also argues that it nevertheless has been
prejudiced by Roxane’s failure to timely disclose the existence of the Concelaed Lot. In this
regard, Medeva argues that Roxane’s late disclosure of the Concealed Lot resulted in Medeva
being forced to seek an extension of expert discovery and conduct expedited testing on the
Concealed Lot over Christmas and New Years at considerable expense in order to establish that
the Concealed Lot was not representative of Roxane’s generic ANDA product. Further, Medeva
argues that but for this costly expedited testing, this case would have been tried with Roxane
relying on the Concealed Lot as being representative of its ANDA product. As a result, Medeva
requests that the Court sanction Roxane. In addition to seeking reimbursement of its expert fees
and costs incurred in testing the Concealed Lot, Medeva also seeks the same sanctions described
21
above with respect to Roxane’s alleged destruction of documents. Medeva claims that such
sanctions are appropriate especially in light of the fact that Roxane’s nondisclosure of the
Concealed Lot further establishes Roxane’s distortion of discovery.
2. Roxane’s Position
Roxane argues that the Concealed Lot was protected from disclosure by the work product
doctrine and was not discoverable until its experts relied on the results of the testing conducted
using the Concealed Lot. In support of this argument, Roxane claims that the Concealed Lot was
developed at the request of outside counsel for use by outside counsel’s retained experts in
conducting an experimental study for use at trial in this litigation. Specifically, Roxane claims
that the Concealed Lot, which differs from Roxane’s ANDA product because it contains a “very
small amount of samarium oxide,” was developed to permit Roxane’s experts to use gamma
scintigraphy to examine where Roxane’s generic product releases in the gastrointestinal tract.
(Roxane Opp. Br. at 19).
Roxane contends that the Concealed Lot and Roxane’s testing thereof were protected by
the work product doctrine because the Concealed Lot was prepared for use at trial by Roxane.
Roxane argues that regardless of whether the actual tablets that make up the Concealed Lot
disclose Roxane’s attorneys’ mental impressions, they are still protected attorney work product
because the work product doctrine is not limited to the protection of counsel’s mental
impressions. Moreover, Roxane claims that the tablets, themselves, do contain its attorneys’
mental impressions because the only purpose of adding samarium oxide to the tablets is to enable
a scientist to conduct gamma scintigraphy testing. For this reason, Roxane argues that even
disclosing the existence of the Concealed Lot would have disclosed Roxane’s attorneys’
22
selection of the specific test for which the Concealed Lot was created. Indeed, Roxane claims
that the facts surrounding the creation, manufacture and existence of the Concealed Lot are facts
created during the pendency of this litigation at the direction of counsel for use at trial and were
thus protected from disclosure.
In addition, Roxane claims that the fact that certain labeling uses the phrase “product
development” neither changes the fact that the Concealed Lot was made at the direction of
counsel, nor indicates that the Concealed Lot was not made in anticipation of litigation. Roxane
argues that it is clear that the Concealed Lot was not made in the ordinary course of Roxane’s
business both because the FDA never requested that Roxane perform gamma scintigraphy testing
and because Roxane has not indicated that it will submit the gamma scintigraphy test results to
the FDA. Further, Roxane argues that even Medeva’s experts recognized that the Concealed Lot
was not created in Roxane’s ordinary course of business as they described the Concealed Lot as
being created in anticipation of litigation in their expert reports. As such, Roxane claims that it is
clear that the Concealed Lot and related testing are protected by the work product doctrine.
Moreover, Roxane contends that it did not waive work product protection by failing to
include the Concealed Lot and related documents on its privilege log or by failing to assert a
privilege regarding same during its employees’ depositions. In this regard, Roxane argues that it
was not obligated to log the existence of the Concealed Lot or any documents concerning same
on its privilege log because the Concealed Lot was created after the Complaint in this matter was
filed and there is no rule requiring an ongoing privilege log. Roxane also claims that it was not
bound to assert work product protection during its employees’ depositions because the questions
asked by Medeva involved Roxane’s prototype formulations and generic ANDA product, not
23
litigation tablets that contained samarium oxide. Similarly, Roxane claims that it did not need to
assert that any documents were being withheld based on the work product doctrine in its response
to Medeva’s Interrogatory No. 5 because Roxane only agreed to produce “proposed ANDA
batches” meaning that the Concealed Lot fell outside the scope of discovery that Roxane agreed
to provide. (Roxane Opp. Br. at 23, n.16).
Further, Roxane argues that it was not obligated to include the Concealed Lot or any
related documents on its privilege log because the parties’ joint stipulation regarding expert
discovery put same outside the scope of permissible discovery until Roxane’s experts relied on
Roxane’s testing using the Concealed Lot. In this regard, Roxane claims that the tablets that
made up the Concealed Lot as well as its testing represented “preliminary [] materials” and “draft
studies” within the meaning of the parties’ joint stipulation and thus were outside of the scope of
permissible discovery and not required to be logged.
Roxane also argues that Medeva’s conduct shows that no privilege log entry was
required. Specifically, Roxane claims that Medeva’s privilege log does not contain entries
regarding expert witnesses. Further, Roxane claims that Medeva’s previous complaints about
Roxane’s privilege log did not involve Roxane’s failure to identify documents related to its
expert witnesses.
In addition, Roxane argues that it did not violate any Court Orders when it conducted
human testing using the Concealed Lot. Roxane notes that the Court entered an Order precluding
Medeva from conducting human testing using Roxane’s expired generic product without prior
Court approval, approval which Medeva never sought. Roxane argues that the Court did not
limit Roxane’s ability to test its own product and that Roxane was entitled to assume any risks
24
inherent in its own testing of that product. As such, Roxane argues that it did not engage in any
misconduct when it tested a “modified version” of its generic product in humans under stringent
study conditions and the supervision of a medical monitor, and with prior approval of a medial
board.
Furthermore, Roxane claims that even if it should have disclosed the Concealed Lot and
related documents to Medeva, either in total or on its privilege log, Medeva has not been
prejudiced. In this regard, Roxane claims that because the Court extended expert discovery,
Medeva has not been denied the opportunity to conduct additional testing. Moreover, Roxane
claims that this issue is now moot since Roxane is no longer relying on its testing of the
Concealed Lot. Additionally, Roxane argues that Medeva could have conducted human testing
using Roxane’s expired and unexpired generic product if it had sought the Court’s permission to
do so, but Medeva never did. Roxane contends that Medeva’s failure to request permission to
conduct human testing had nothing to do with Roxane’s obstructing Medeva’s ability to conduct
same. As a result, Roxane claims that the imposition of any sanctions is unwarranted.
II. Analysis
A. Standard of Review
Spoliation is “the destruction or significant alteration of evidence, or the failure to
preserve property for another’s use as evidence in pending or reasonably foreseeable litigation.”
Zubulake v. UBS Warburg LLC, 229 F.R.D. 422, 430 (S.D.N.Y. 2004) (internal quotation marks
and citations omitted). It occurs “when a party has intentionally or negligently breached its duty
to preserve potentially discoverable evidence[.]” Kounelis v. Sherrer, 529 F.Supp.2d 503, 518-
519 (D.N.J. 2008). The duty to preserve evidence arises when a party reasonably believes that
25
litigation is foreseeable and, as such, may arise “many years before litigation commences[.]”
Micron Technology, Inc. v. Rambus Inc., 255 F.R.D. 135, 148 (D.Del. 2009).
Evidence of spoliation may give rise to sanctions, which include: dismissal of a claim or
granting judgment in favor of a prejudiced party; suppression of evidence; an adverse inference
(i.e., the spoliation inference); fines; and attorney’s fees.5 Mosaid Tech., Inc. v. Samsung Elec.
Co., Ltd., 348 F.Supp.2d 332, 335 (D.N.J. 2004). The imposition of sanctions is warranted
“when there is evidence that a party’s spoliation of evidence threatens the integrity of th[e]
Court.” Id. Spoliation sanctions serve the following three functions: remedial, punitive and
deterrent. They level the playing field so that the prejudiced party is restored to the position it
would have been in absent the spoliation. They punish the spoliator for its misconduct and they
warn other potential litigants that spoliation of evidence will not be tolerated and will be dealt
with by the Court if necessary. See Id.
The Court has authority to impose spoliation sanctions pursuant to both the Federal Rules
of Civil Procedure and its inherent authority, and the choice of which sanction should be imposed
rests in the sound discretion of the Court. Id. Of the sanctions that can be imposed, dismissal
and suppression of evidence are considered the two most drastic and should only be levied in the
most extraordinary of circumstances. Id. In determining whether such drastic sanctions are
warranted, the Court considers: “(1) the degree of fault of the party who altered or destroyed the
evidence; (2) the degree of prejudice suffered by the opposing party; and (3) whether there is a
lesser sanction that will avoid substantial unfairness to the opposing party and, where the
5
The imposition of sanctions in patent cases is controlled by regional circuit law.
Monsanto Co. v. Ralph, 382 F.3d 1374, 1380 (Fed. Cir. 2004). Therefore the Court relies on
Third Circuit law in analyzing what, if any, sanctions should be imposed on Roxane.
26
offending party is seriously at fault, will serve to deter such conduct by others in the future.”
Schmid v. Milwaukee Elec. Tool Corp., 13 F.3d 76, 79 (3d Cir. 1994). In order to establish
prejudice, the party seeking spoliation sanctions must “come forward with plausible, concrete
suggestions as to what [the missing] evidence might have been (Id at 80), and must show that its
“ability to prepare effectively a full and complete trial strategy” has been impeded. Ware v.
Rodale Press, Inc., 322 F.3d 218, 222 (3d Cir. 2003). Generally, however, absent a showing of
bad faith and substantial prejudice, dispositive sanctions should not be imposed. See Schmid, 13
F.3d at 80.
Of the lesser sanctions often considered by courts faced with spoliation is the spoliation
inference. “The spoliation inference is an adverse inference that permits a jury to infer that
‘destroyed evidence might or would have been unfavorable to the position of the offending
party.’” Mosaid, 348 F.Supp.2d at 336 (quoting Scott, 196 F.R.D. at 248). In order for the
spoliation inference to apply, the party seeking the adverse inference must establish that the
following four factors have been satisfied: “First, it is essential that the evidence in question be
within the party’s control. Second, it must appear that there has been actual suppression or
withholding of the evidence. Third, the evidence destroyed or withheld was relevant to claims or
defenses. And fourth, it was reasonably foreseeable that the evidence would later be
discoverable.” Mosaid, 348 F.Supp.2d at 336 (internal quotation marks and citations omitted);
see also Aurelio v. Bd. of Educ. of the Borough of Carteret, Civil Action No. 06-3146 (JLL),
2009 WL 1794800, *9 (D.N.J. June 23, 2009) (indicating that burden fell on Plaintiff, party
seeking adverse inference, to show that all four factors were met)).
27
With respect to the second factor (i.e. “actual suppression”), “negligent destruction of
relevant evidence can be sufficient to give rise to the spoliation inference.” Id. at 338. A party
need not show that its adversary “intentionally or knowingly destroyed or withheld” evidence. Id.
at 337. This is true because “[i]f a party has notice that evidence is relevant to an action, and
either proceeds to destroy that evidence or allows it to be destroyed by failing to take reasonable
precautions, common sense dictates that the party is more likely to have been threatened by the
evidence” and, regardless of the “offending party’s culpability[,] . . . it cannot be denied that the
opposing party has been prejudiced.” Id. at 338.6
In addition, with respect to the third factor, that the destroyed or withheld evidence is
“relevant,” while not explicitly addressed by the Third Circuit or the District of New Jersey,
typically, where the culpability of the offending party is negligence, gross negligence or even
recklessness (as opposed to willful or knowing) the party seeking the spoliation inference must
establish not only that the destroyed or withheld evidence is probative under Fed.R.Evid. 401,
but also “must adduce sufficient evidence from which a reasonable trier of fact could infer that
the destroyed or unavailable evidence would have been of the nature alleged by the party affected
by its destruction.” Zubulake, 229 F.R.D. at 431, n. 67 (internal quotation marks and citation
omitted). In other words, the party seeking the spoliation inference must make a sufficient
showing from which a fact finder could reasonably determine that the destroyed or withheld
evidence would have been favorable to the movant.
6
As described above, where one of the more drastic sanctions such as dismissal or
suppression of evidence is sought, a greater showing of culpability may be required. Schmid, 13
F.3d at 80; see Mosaid, 348 F.Supp.2d at 338, n. 11.
28
B. The Destruction of Documents
The Court finds that spoliation sanctions are not warranted for Roxane’s alleged
destruction of documents. While the court cannot fathom why Roxane failed to institute a
litigation hold prior to July 2007 when it clearly anticipated litigation and had a duty to preserve
information at least as early as February 2002, when it knew it would be filing a Paragraph IV
Certification, and perhaps as far back as July 31, 2001 if its privilege log claiming work product
protection over a document with that date is to be credited, the Court finds that Roxane
adequately maintained its mesalamine files pursuant to its normal document retention policies.
These policies required Roxane to retain technical information such as laboratory notebooks,
technical reports and developmental and analytical records for a minimum of twenty years.
Further, while those policies did not specifically reference Roxane’s ANDA product, the record
establishes that documents concerning same were in fact retained. This is true despite the fact
that Roxane held file clean-up days prior to instituting a litigation hold and after it had a duty to
preserve evidence. Indeed, the quantity and more importantly quality of documents produced by
Roxane in this litigation establish that Roxane diligently preserved mesalamine related
information. As such, this matter is easily distinguishable from Sanofi-Aventis Deutschland
GmbH v. Glenmark Pharm. Inc., USA, Civil Action No. 07-CV-5855 (DMC-JAD), *9 (D.N.J.
July 1, 2010), where the Court imposed an adverse inference against the defendants after
determining that they practiced systematic document destruction when no litigation hold was in
place.
While Medeva points to several categories of documents that are allegedly missing,
Medeva simply has not established that Roxane intentionally or negligently destroyed the vast
29
majority of these documents. Indeed, of all of the documents identified by Medeva as having
been destroyed by Roxane, the Court finds that the record at best supports the conclusion that
Roxane at one point possessed but failed to produce the Anapharm contract and protocol and two
emails produced by Quintiles. Moreover, the Court finds that Roxane’s failure to produce these
documents does not warrant the imposition of spoliation sanctions. In this regard, the Court
finds that Medeva has not been prejudiced by Roxane’s failure to produce the Anapharm contract
and protocol and the two emails produced by Quintiles.
With respect to the Anapharm documents, the Court notes that the Anapharm study did
not deal with Roxane’s ANDA product or any prototypes thereof, but instead examined
Medeva’s product. As such, the Anapharm study was not submitted to the FDA and was not
important to Roxane’s ANDA. Further, while the Anapharm study addressed Medeva’s product,
Medeva’s experts do not appear to have relied upon or even considered the study in rendering
their opinions. More importantly, while Roxane failed to produce its contract with Anapharm or
the Anapharm protocol, which would have outlined what the study was going to entail, Roxane
did produce the principal and supplemental reports related to the Anapharm study which set forth
how the study actually proceeded and the results of the study, including a full data analysis with
tables, graphs and explanatory text. Given Roxane’s production of these reports coupled with the
fact that neither Medeva nor Roxane’s experts rely on the Anapharm study, the Court simply
cannot see how Medeva’s “ability to prepare effectively a full and complete trial strategy” has
30
been impeded by Roxane’s failure to produce its contract with Anapharm or the Anapharm
protocol. Ware, 322 F.3d 218, 222 (3d Cir. 2003).7
The Court likewise does not believe that Medeva’s ability to effectively prepare a full and
complete trial strategy has been impeded by Roxane’s failure to produce the two emails that it
once possessed and were produced by Quintiles. In this regard, the Court notes that while
Roxane did not produce these emails, Quintiles did. As such, Medeva has obtained the
information missing from Roxane’s production. Moreover, while the fact that these emails were
missing from Roxane’s production may, under certain circumstances, raise a serious concern
over what other information is missing from Roxane’s production, such a concern is simply not
present here where the breadth and quality of Roxane’s document production establishes that
Roxane met its duty to preserve information. Further, the substance of the two missing emails
establishes that they are not of critical importance to this matter: one discussed whether the
Russian equivalent to the FDA wanted to receive information in writing or via telephone and the
second discussed where Asacol could be purchased in Europe.
Under these circumstances, the Court finds that spoliation sanctions are not warranted.
Indeed, the record does not support the conclusion that Roxane’s destruction/non-production of
documents has threatened the integrity of this Court. See Mosaid, 348 F.Supp.2d at 335. The
Court finds that Medeva’s ability to fairly participate in this matter and effectively prepare a full
and complete trial strategy has not been impeded by Roxane’s alleged destruction of documents.
See Ware, 322 F.3d at 222. As such, the Court finds no reason to punish Roxane for its conduct
7
While not necessary for the Court to reach this conclusion, the fact that Medeva never
sought to obtain the missing Anapharm documents directly from Anapharm, reinforces the
Court’s determination that Medeva has not been prejudiced by their nonproduction.
31
or to deter other potential litigants from engaging in same. See Mosaid, 348 F.Supp.2d at 335
(describing three functions served by imposing spoliation sanctions).
C. The Concealment of Lot No. 089226
Unlike with Roxane’s alleged destruction of documents, the Court finds that sanctions are
warranted for Roxane’s concealment of Lot No. 089226. In coming to this conclusion, the Court
does not reach the issue of whether the Concealed Lot was entitled to attorney-client privilege
and/or work product protection, but, instead, assumes for purposes of this motion that it was. As
such, the Court focuses its inquiry on whether the existence of the Concealed Lot should have
been disclosed by Roxane on its privilege log and whether Roxane’s failure to log same warrants
the imposition of sanctions. The Court finds that both questions should be answered in the
affirmative.
With respect to Roxane’s decision not to include the existence of the Concelaed Lot on its
privilege log, the Court finds Roxane’s reasons for failing to do so completely unavailing. In this
regard, the Court finds that Roxane has not provided any legal support that stands for the
proposition that it was not required to include the existence of the Concealed Lot on its privilege
log. To the extent, Roxane relies on Grider v. Keystone Health Plan Cent., Inc., 580 F.3d 119
(3d Cir. 2009) to argue that it was not required to do so because there is no requirement to
maintain an ongoing privilege log, the Court finds Roxane’s reliance on the Grider decision to be
misplaced. First, Grider does not hold that litigants are not required to maintain ongoing
privilege logs. Instead, in a footnote, the Court in Grider merely acknowledges that “a privilege
log may not be required for communications with counsel that take place after the filing of a law
suit.” Id. at 140, n.22. Second, what is clear from the context of the footnote, the Grider case
32
did not contemplate the requirement or lack thereof to log the existence of a product, as opposed
to a communication, created at the request of counsel. Id.
Further, none of the other cases relied upon by Roxane supports its contention that it had
no duty to include the existence of the Concealed Lot on its privilege log. In this regard, the
Court notes that Roxane did not cite to a single case in which the Court affirmed a company’s
decision to refrain from including a product manufactured at its facilities on the advice of counsel
on its privilege log. The closest cases cited by Roxane are Novartis, 203 F.R.D. 159 and
Vardon,156 F.R.D. 641, which address claims of privilege raised over the methods used and
results obtained from product testing conducted for use in pending litigation; they did not address
the nondisclosure of the product being tested. More importantly, in both cases claims of
privilege were timely made by the party from whom discovery concerning the testing was sought.
Indeed, in the only case cited by Roxane in which a party withheld information it deemed to be
protected by the work product privilege without logging the privilege, the court determined that a
waiver had occurred given the responding party’s failure to include an index of the withheld
documents on its privilege log. Feacher v. Intercontinental Hotels Group, Civ. Action No. 3:06-
cv-877 (TJB/DEP), 2007 WL 3104329, *5-6 (N.D.N.Y. Oct. 22, 2007).
Similarly, neither Roxane’s Response to Medeva’s Interrogatory No. 5 nor the parties’
joint stipulation regarding expert discovery exempted it from having to include a notation
regarding the existence of the Concealed Lot on its privilege log. With respect to Medeva’s
Interrogatory No. 5, Roxane’s statement that it objects to this interrogatory “as overly broad,
unduly burdensome and not calculated to lead to the discovery of admissible evidence in that it
seeks information concerning products and formulations other than the product proposed in
33
ANDA No. 79-073[,]” is insufficient. Information regarding the existence of the Concealed Lot
fell within the scope of Medeva’s Interrogatory No. 5 and clearly is relevant to this litigation.
While the composition of the Concealed Lot differs slightly from Roxane’s ANDA product in
that it contains a “very small amount of samarium oxide,” a material that can be made
radioactive, which would permit Roxane to do testing to determine where its product releases in
the gastrointestinal tract, Roxane intended the Concealed Lot to be functionally representative of
its ANDA product. (Roxane Opp. Br. at 19). As such, if Roxane intended to withhold
information regarding the existence of the Concealed Lot based on the work product doctrine, it
had an obligation to specifically notify Medeva that it was doing so by including an entry in its
privilege log regarding same; the aforementioned objection was simply inadequate.
Likewise, the parties’ joint stipulation regarding expert discovery, which put “outside the
scope of permissible discovery . . . draft reports, draft studies, or draft workpapers; preliminary
calculations, computations, or data runs; or other preliminary or draft materials prepared by, for,
or at the direction of an expert witness” does not excuse Roxane’s failure to make note of the
existence of the Concealed Lot on its privilege log. (Silverstein Decl., Ex. EE ¶ 4). First, the
Court agrees with Medeva that there is nothing “preliminary” or “draft” about the final tablets
manufactured by Roxane that make up the Concealed Lot. Second, according to the Court’s
reading of the parties’ joint stipulation regarding exert discovery, it does not appear that a lot of
tablets falls within the types of materials intended to be covered by the stipulation. If Roxane
had such an intent, then it should have made sure that the joint stipulation was written more
clearly to disclose same.
Thus, the Court finds no justification for Roxane’s failure to include a notation on its
34
privilege log regarding the Concealed Lot. To be clear, the Court does not find that Roxane had
to disclose the purpose behind the Concealed Lot. Indeed, Roxane had no obligation to disclose
that the Concealed Lot contained samarium oxide or that Roxane intended to use the Concealed
Lot to conduct gamma scintigraphy testing in order to determine where Roxane’s ANDA product
releases in the gastrointestinal tract. Instead, Roxane simply had to disclose that it was
withholding samples of and identifying information regarding a batch of tablets created at the
direction of counsel. Such a notation would have put Medeva on notice that information was
being withheld and would have allowed Medeva to have filed a timely petition with the Court to
determine whether Roxane’s privilege claim was appropriate.
Roxane’s failure to appropriately log its claim of work product protection regarding the
Concealed Lot effectively precluded Medeva from challenging Roxane’s claim of privilege and
forced Medeva, perhaps unnecessarily, to engage in costly, expedited expert testing on the
Concealed Lot when Roxane finally disclosed the existence of same at the end of expert
discovery. Further, Roxane’s failure precipitated the filing of the instant motion. As such, it not
only harmed Medeva, but also thwarted the Court’s efforts to resolve this case in an efficient and
economic fashion by preventing the Court from addressing Roxane’s claim of privilege in the
first instance and necessitating the Court’s decision on this otherwise unnecessary motion. As
such, the Court finds that the imposition of sanctions is warranted for Roxane’s unjustified
failure to include a notation on its privilege log regarding the existence of the Concealed Lot.
In this regard, the Court finds that the imposition of Medeva’s reasonable expert fees and
costs associated with its expedited testing of the Concealed Lot along with its reasonable
attorneys’ fees and costs incurred in briefing and arguing the portion of the instant motion related
35
to Roxane’s nondisclosure of the Concealed Lot are an appropriate sanction. The Court believes
that the imposition of the aforementioned costs and fees will fairly compensate Medeva for the
harm it suffered from Roxane’s nondisclosure of the existence of the Concealed Lot and will also
effectively deter Roxane and future litigants from further engaging in this type of conduct. The
Court notes that under other circumstances in lieu of imposing all of the fees and costs noted
above, or perhaps even in addition to same, it may have deemed Roxane to have waived privilege
with respect to the Concealed Lot. The Court, however, finds that such a sanction would be
ineffectual here given the fact that Roxane no longer intends to rely on its testing of the
Concealed Lot in this litigation.
III. Conclusion
For the reasons stated above, Medeva’s motion for the imposition of spoliation sanctions
is GRANTED in part and DENIED in part. An appropriate Order follows.
Dated January 28, 2011
s/Tonianne J. Bongiovanni
HONORABLE TONIANNE J. BONGIOVANNI
UNITED STATES MAGISTRATE JUDGE
36
LOCAL CIVIL AND CRIMINAL RULES
OF THE
FOR THE
With Revisions as of June 22, 2012

TABLE OF CONTENTS
INTRODUCTION - 1984 Revision
FOREWORD - 1997 Revision
LOCAL CIVIL RULES PAGE
1.1 Rules of Procedure; Scope of these Rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
4.1 Service of Process.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
5.1 Service and Filing of Pleadings and Other Papers. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
5.2 Electronic Service and Filing Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
5.3 Protective Orders and Public Access Under CM/ECF. . . . . . . . . . . . . . . . . . . . . . . . . 13
6.1 Extensions of Time and Continuances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
7.1 Application and Motion Practice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
7.2 Affidavits and Briefs.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
8.1 Pleading Damages.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
9.1 Special Matters - Review of Social Security Matters.. . . . . . . . . . . . . . . . . . . . . . . . . 22
9.2 Special Matters - Admiralty and Maritime Rules for the
United States District Court for the District of New Jersey. . . . . . . . . . . . . . . . . . . . . 26
9.3 Local Patent Rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
10.1 Form of Pleadings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
11.1 Signing of Pleadings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
11.2 Verification of Petitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
11.3 Applications for Fed. R. Civ. P. 11 Sanctions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
16.1 Pretrial Conferences; Scheduling; Case Management. . . . . . . . . . . . . . . . . . . . . . . . . 46
24.1 Notice of Claim of Unconstitutionality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
24.2 Statutory Court. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
26.1 Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
27.1 Depositions for Use in a Foreign Country. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
28.1 Letters Rogatory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
33.1 Interrogatories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
34.1 Requests for Production of Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
36.1 Requests for Admission. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
37.1 Discovery Motions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
37.2 Applications for Fed. R. Civ. P. 37 Sanctions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
38.1 Jury Demand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
40.1 Allocation and Assignment of Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
41.1 Dismissal of Inactive Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
42.1 Consolidation of Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
44.1 Seal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
47.1 Petit Jurors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
47.2 Assessment of Jury Costs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
48.1 Civil Jury.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
48.2 Taking of Civil Verdict.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
-ii-
52.1 Oral Opinions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
54.1 Costs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
54.2 Compensation for Services Rendered and Reimbursement of Expenses. . . . . . . . . . . 60
54.3 Prepayment of Clerk's and Marshal's Fees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
56.1 Summary Judgment Motions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
58.1 Entry of Judgments and Orders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
65.1 Applications for Emergency Relief. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
65.1.1 Security and Sureties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
66.1 Receiverships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
67.1 Deposit in Court. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
69.1 Marshal's Vouchers.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
72.1 United States Magistrate Judges.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
73.1 Civil Trials by Consent before United States Magistrate Judges. . . . . . . . . . . . . . . . . 70
77.1 Court Session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
78.1 Motion Days. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
79.1 Custody of Original Papers, Records and Exhibits. . . . . . . . . . . . . . . . . . . . . . . . . . . 71
79.2 Briefs Part of Public Record. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
79.3 Entry of Satisfaction of Judgments and Decrees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
79.4 Filing of Mandate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
79.5 Clerk to Maintain List of Official Newspapers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
80.1 Transcripts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
81.1 Naturalization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
81.2 Habeas Corpus and Motions under 28 U.S.C. §2255 In Non Death Penalty Cases. . 74
81.3 Habeas Corpus and Motions under 28 U.S.C. §2255 In Death Penalty Cases. . . . . . 74
83.1 Adoption and Amendment of Local Rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
83.2 Relaxation or Modification of Local Rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
83.3 Procedure in The Absence of Rule or Statutory Provision.. . . . . . . . . . . . . . . . . . . . . 78
85.1 Title.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
101.1 Admission of Attorneys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
102.1 Withdrawal of Appearance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
103.1 Judicial Ethics and Professional Responsibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
104.1 Discipline of Attorneys.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
105.1 Extrajudicial Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
201.1 Arbitration.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
301.1 Mediation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
401.1 Media Coverage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
501.1 Possession and use of Electronic Equipment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
LOCAL CRIMINAL RULES
1.1 Scope and Applicability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.1 United States Magistrate Judges.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
7.1 Grand Jurors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
12.1 Motions under Fed. R. Crim. P. 12. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
18.1 Assignment of Criminal Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
24.1 Selection and Impanelment of Trial Jurors.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
32.1 Probation.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
41.1 Motions under Fed. R. Crim. P. 41. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
-iii-
44.1 Formal Written Appearance - Criminal Matters.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
46.1 Release from Custody. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
53.1 Conduct in the Courtroom. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
55.1 Record of Proceedings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
55.2 Custody and Disposition of Exhibits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
58.1 Proceedings in Misdemeanor and Petty Offense Cases. . . . . . . . . . . . . . . . . . . . . . . . 115
60.1 Title.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
101.1 Extrajudicial Statements in Criminal Proceedings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
CONVERSION TABLES
General Rule to Local Civil/Criminal Rule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Local Civil Rule to General Rule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Local Criminal Rule to General Rule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
COURT'S APPENDICES
A1. Client's & Supervising Attorney's Authorizations for Appearance by Law Student
A2. Form for Designating Compliance with the Student Practice Rule
B. Criminal Case Appearance Form
C. Affidavit by Owner of Cash Security
D. Order Granting Motion to Deposit Sum of Money with the Court
E. List of Petty Offenses and Minimum Fines Applicable Thereto Pursuant to
L.Cr.R. 58.1(c)
F. Transcript Rates
G. Official Newspapers
H. Appointment of Attorneys in Pro Se Civil Actions
I. Plan of Implementation of the United States District Court for the District of
New Jersey Pursuant to the Criminal Justice Act of 1964, as Amended
J. Plan for Prompt Disposition of Criminal Cases
K. Schedule of Fees
L. Application for Extension of Time to Reply
M. Guidelines for Arbitration
N. RESERVED
O. Optional RICO Case Order
P. In Forma Pauperis Affidavit and Order
Q. Guidelines for Mediation
R. Guidelines for Litigation Conduct
S. Discovery Confidentiality Order
T. Procedures for Patent Pilot Project Cases
-iv-
is unconstitutional. If memoranda have been served discussing the constitutional question, two
copies of each memorandum shall be forwarded with the notification.
(b) If, at any time prior to the trial of an action in which neither the State of New Jersey nor any
officer, agency or employee thereof is a party, a party to the action questions the constitutionality
of any State statute, such party (to enable the Court to comply with 28 U.S.C. §2403(b)) shall
forthwith, upon the filing of any pleading which raises the question, notify the Judge to whom the
action is assigned, in writing, of the existence of said question identifying: (1) the title and docket
number of the action; (2) the statute challenged; and (3) why it is claimed that the statute is
unconstitutional. If memoranda have been served discussing the constitutional question, two
copies of each memorandum shall be forwarded with the notification.
Source: G.R. 32.
Civ. RULE 24.2 STATUTORY COURT
Where, pursuant to law, an action must be heard by a District Court composed of three Judges,
two from this Court and one from the Third Circuit, the procedure to be followed by counsel in
filing pleadings and submitting briefs will be as follows:
(a) All pleadings are to be filed with the Clerk in quadruplicate, the original becoming part of the
Clerk's file, the three copies to be distributed by the Clerk to the members of the Statutory Court.
(b) Six copies of briefs are to be submitted. Unless otherwise directed by the Court, they are to be
delivered to the Clerk for distribution to the members of the Statutory Court.
Source: G.R. 33.

Civ. RULE 26.1 DISCOVERY
(a) Discovery - Generally
All parties shall conduct discovery expeditiously and diligently.
(b) Meeting of Parties, Discovery Plans, and Initial Disclosures
(1) The requirements currently codified in Fed. R. Civ. P. 26(a) and (f) pertaining to required
disclosures, meetings of parties, and submission of discovery plans, shall apply to all civil cases
filed after December 1, 1993 and to all civil cases pending on December 1, 1993 that have not
had their initial scheduling conference prior to January 20, 1994; except that these requirements
shall not apply to those civil cases described in L.Civ.R. 72.1(a)(3)(C) in which scheduling
conferences are not normally held, unless the judicial officer otherwise directs. The judicial
officer may modify or suspend these requirements in a case for good cause.
(2) The initial meeting of parties as required in Fed. R. Civ. P. 26(f) shall be convened at least
21 days before the initial scheduling conference, and the proposed discovery plan under Fed. R.
Civ. P. 26(f)(3) shall be generated at that meeting and delivered to the Magistrate Judge within
-49-
14 days after the meeting of parties. The parties shall submit their Fed. R. Civ. P. 26(f) discovery
plan containing the parties' views and proposals regarding the following:
(a) Any changes in timing, form, or requirements of mandatory disclosures under Fed. R. Civ. P.
26(a);
(b) The date on which mandatory disclosures were or will be made;
(c) The anticipated substantive scope of discovery, including both discovery relevant to the
claims and defenses and discovery relevant to the subject matter of the dispute;
(d) Whether any party will likely request or produce computer-based or other digital
information, and if so, the parties' discussions of the issues listed under the Duty to Meet and
Confer in L. Civ. R. 26.1(d)(3) below;
(e) The date by which discovery should be completed;
(f) Any needed changes in limitations imposed by the Federal Rules of Civil Procedure, local
rule, or standing order;
(g) Any orders, such as data preservation orders, protective orders, etc., which should be entered;
(h) Proposed deadline for joining other parties and amending the pleadings;
(i) Proposed dates for filing motions and for trial;
(j) Whether the case is one which might be resolved in whole or in part by voluntary arbitration
(pursuant to L. Civ. R. 201.1 or otherwise), mediation (pursuant to L. Civ. R. 301.1 or
otherwise), appointment of a special master or other special procedure.
The parties shall make their initial disclosures under Fed. R. Civ. P. 26(a)(1) within 14 days after
the initial meeting of the parties, unless otherwise stipulated or directed by the Court. Such
discovery plans and disclosures shall not be filed with the Clerk.
(c) Discovery Materials
(1) Initial and expert disclosure materials under Fed.R.Civ.P.26(a)(1) and 26(a)(2), transcripts of
depositions, interrogatories and answers thereto, requests for production of documents or to
permit entry onto land and responses thereto, and requests for admissions and answers thereto
shall not be filed until used in a proceeding or upon order of the Court. However, all such papers
must be served on other counsel or parties entitled thereto under Fed.R.Civ.P.5 and 26(a)(4).
(2) Pretrial disclosure materials under Fed.R.Civ.P.26(a)(3) shall be incorporated by reference

into the order entered after any final pretrial conference under Fed.R.Civ.P.16(d).
(3) In those instances when such discovery materials are properly filed, the Clerk shall place
them in the open case file unless otherwise ordered.
-50-
(4) The party obtaining any material through discovery is responsible for its preservation and
delivery to the Court if needed or ordered. It shall be the duty of the party taking a deposition to
make certain that the officer before whom it was taken has delivered it to that party for
preservation and to the Court as required by Fed. R. Civ. P. 30(f)(1) if needed or so ordered.
(d) Discovery of Digital Information Including Computer-Based Information
(1) Duty to Investigate and Disclose. Prior to a Fed. R. Civ. P. 26(f) conference, counsel shall
review with the client the client’s information management systems including computer-based
and other digital systems, in order to understand how information is stored and how it can be
retrieved. To determine what must be disclosed pursuant to Fed. R. Civ. P. 26(a) (1), counsel
shall further review with the client the client’s information files, including currently maintained
computer files as well as historical, archival, back-up, and legacy computer files, whether in
current or historic media or formats, such as digital evidence which may be used to support
claims or defenses. Counsel shall also identify a person or persons with knowledge about the
client’s information management systems, including computer-based and other digital systems,
with the ability to facilitate, through counsel, reasonably anticipated discovery.
(2) Duty to Notify. A party seeking discovery of computer-based or other digital information
shall notify the opposing party as soon as possible, but no later than the Fed. R. Civ. P. 26(f)
conference, and identify as clearly as possible the categories of information which may be
sought. A party may supplement its request for computer-based and other digital information as
soon as possible upon receipt of new information relating to digital evidence.
(3) Duty to Meet and Confer. During the Fed. R. Civ. P. 26(f) conference, the parties shall
confer and attempt to agree on computer-based and other digital discovery matters, including the
following:
(a) Preservation and production of digital information; procedures to deal with inadvertent
production of privileged information; whether restoration of deleted digital information may be
necessary; whether back up or historic legacy data is within the scope of discovery; and the
media, format, and procedures for producing digital information;
(b) Who will bear the costs of preservation, production, and restoration (if necessary) of any
digital discovery.
Amended: March 14, 2001, October 6, 2003

Source: L.Civ.R. 26.1(a) - G.R. 15.E.1; L.Civ.R. 26.1(b) - G.R. 15.B.1-2; L.Civ.R. 26.1(c) -
G.R. 15.G.
Civ. RULE 27.1 DEPOSITIONS FOR USE IN A FOREIGN COUNTRY
(a) A person desiring to take the deposition of a witness who resides or may be found within the
District for use in a judicial proceeding pending in a foreign country may apply ex parte to the
Court for an appropriate order. If the deposition is to be taken upon written interrogatories, a
copy of the interrogatories shall be annexed to the application. If the court of the foreign country
has appointed a person to take the deposition, the order shall designate that person commissioner
unless there be good cause for withholding such designation. If no such appointment has been
-51-
T H E S E D O N A C O N F E R E N C E ® W O R K I N G G R O U P S E R I E SSM
THE SEDONA CONFERENCE®

“JUMPSTART OUTLINE”:
Questions to Ask Your Client & Your
Adversary to Prepare for Preservation,
Rule 26 Obligations, Court Conferences
& Requests for Production
A Project of The Sedona Conference®

Working Group on Electronic Document
Retention & Production (WG1)
MARCH 2011 VERSION
Copyright © 2011, The Sedona Conference®

THE SEDONA CONFERENCE®
“JUMPSTART OUTLINE”
A Project of The Sedona Conference® Working Group on

Electronic Document Retention & Production (WG1)
Author:
The Sedona Conference®
Editor-in-Chief:
Ariana J. Tadler
March 2011 Version
This outline was initially prepared by Ariana Tadler for The Sedona Conference® Institute’sSM program
entitled “Getting Ahead of the eDiscovery Curve: Strategies to Reduced Costs & Meet Judicial Expectations”
held March 13-14, 2008, at the Westin Horton Plaza San Diego, San Diego, CA, as an example of a
tool to assist counsel in dealing with electronic discovery issues.
It was updated recently.
The opinions expressed in this publication, unless otherwise attributed, represent consensus views of the
members of The Sedona Conference® Working Group on Electronic Document Retention & Production.
They do not necessarily represent the views of any of the individual participants or their employers, clients,
or any other organizations to which any of the participants belong, nor do they necessarily
represent official positions of The Sedona Conference®.
Thanks go to all who participated in the dialogue that led to this Commentary.
In addition, we thank all of our Working Group SeriesSM Sustaining and Annual Sponsors, whose support is essential to our
ability to develop Working Group SeriesSM publications. For a listing of our sponsors just click on the
“Sponsors” Navigation bar on the homepage of our website.
COMMENTS & SUGGESTIONS should be directed to:

Ken Withers at The Sedona Conference®
5150 North 16th Street, Suite A-215
Phoenix, AZ 85016
602-258-2499
kjw@sedonaconference.org
Copyright © 2011 The Sedona Conference®

All Rights Reserved.
REPRINT REQUESTS:
Requests for reprints or reprint information should be directed to Richard Braman,
Board Chair of The Sedona Conference®, at rgb@sedonaconference.org.
The Sedona Conference® Jumpstart Outline March 2011
The Sedona Conference® Jumpstart Outline

Introduction
This outline sets forth, by way of example only, a series of topics and questions to ask your client
and your adversary as you prepare for meeting obligations related to preservation, requests for
production, court conferences, and Fed. R. Civ. P. 26. The answers to these questions will guide
you in (i) instructing your client about its preservation and production obligations and (ii)
understanding your adversary’s systems and preservation efforts to date, and (iii) structuring and
tailoring your discovery requests addressed to your adversary. This is a simplified outline to assist, in
particular, those people who have had only limited experience in dealing with electronic discovery.
As those with extensive experience in this arena know, the process of questioning—and even the
questions themselves—are iterative in scope. With each answer you elicit, inevitably additional
questions must be asked. Hopefully, having an outline like this within easy reach will serve as a
“jumpstart” to encourage transparency and dialogue in the discovery process, as contemplated by
the Rules and The Sedona Conference® Cooperation Proclamation.
1. Document Retention Policy
1.1 Do you have a document retention (or records management) policy? Is it a written policy?
1.1.1. If yes, when was the policy implemented?
1.1.2. If yes, is the policy enforced? By whom? How?
1.1.3. If yes, did the policy change during [insert relevant time period]?
1.1.4. If yes, are you willing to produce the policy/policies?
2. Key Custodians of Potentially Relevant Information
2.1. Given the facts of the case, who are the key custodians of potentially relevant information?
Who is responsible for maintaining/administering the company’s electronic systems?
2.2. To what extent has information in the possession, custody, or control of the key custodians
been preserved? (Discuss what those efforts have been to date and what, if any, additional
efforts are underway.)
2.2.1. If conferring with your client, address efforts to date and further efforts that need to
be made.
2.2.2. If conferring with your adversary, discuss efforts to date and, if insufficient, request
that further efforts be made, if appropriate.
p 1
2.3. Disclosure of identities of key custodians
2.3.1. In representing your client, consider disclosing to your adversary the identities of the
key custodians for whom information has been/will be preserved.
2.3.2. If you are a requesting party, consider identifying those people who you believe are
key custodians to memorialize your request for preservation of their information.
2.4. Are there any third parties that may hold potentially relevant information?
2.4.1. To what extent has information in the possession, custody, or control of third parties
been preserved? (Discuss what those efforts have been to date and what, if any,
additional efforts are underway.)
2.4.2. If conferring with your client, address efforts to date and further efforts that need to
be made with respect to third parties.
2.4.3. If conferring with your adversary, discuss efforts to date and, if insufficient request
that further efforts, be made, if appropriate.
2.4.4. In representing your client, consider disclosing to your adversary the identities of the
third parties for whom information has been/will be preserved.
2.4.5. If you are a requesting party, consider identifying those people who you believe are
third parties that may have relevant data to memorialize your request for
preservation of their information.
NOTE: This is an iterative process. You should plan to confer with your adversary on a recurring
basis so that you can continue to update your adversary on any additional key custodians.
3. Network Servers
The questions below concern current and former database and file servers on any potentially
relevant network that now store or previously stored discoverable electronic data (hereinafter
referred to as “network servers”). These questions should be asked of both your client and your
adversary.
3.1. Do you use, for any purpose, a network-based system? If yes, please describe.
3.2. Do you have a system that serves to back up the information managed and/or stored on the
network(s)?
3.2.1. If yes, do you have at least one computer (i.e., non-incremental) backup of each of
your network servers for each month for the period [insert relevant time period]?
3.2.2. If not, for which months do you/do you not have at least one complete backup?
p2
3.2.4. For those months, if any, for which you do not have a complete backup, do you
have incremental backups or other backups from which a full backup can be created
of all data as of a given date in each such month?
3.2.5. If so, please describe the nature of such incremental or other backups and identify
the months for which you have them.
3.3. Can specific files contained on network backups be selectively restored?
3.3.1. How? By what means?
3.3.2. Have you ever done this before?
3.3.3 In what context? Is the context such that the data restored might be deemed relevant
in the context of the current litigation?
3.4. As a matter of firm policy, do you overwrite, reformat, erase, or otherwise destroy the
content of the backups of your network servers on a periodic basis?
3.4.1. If so, under what circumstances?
3.4.2. If so, what is the rotation period?
3.4.3. If the rotation period has changed since [insert date], please describe the changes.
3.5. Do you maintain a company-wide intranet or other database accessible to any employees
that provides/stores potentially relevant information? [Consider being more specific, e.g.,
“regarding [a particular subject].”]
3.6. Do you maintain network servers at any of the company’s divisions/business

units/locations/offices/subsidiaries that exist separately from or in addition to company-
wide server(s)?
3.6.1. If yes, to what extent do any of those servers store any potentially relevant
information in the context of this litigation?
3.6.2. Ask follow-up questions consistent with the network server-based questions above.
4. Email Servers
The questions below concern the current or former servers on your network (“email servers”) that
now or previously stored discoverable electronic internal or external peer-to-peer messages,
including email, third party email sources, and instant messages (collectively, “email”).
4.1. Identify the systems (client and server-side applications) used for email and the time period
for the use of each such system, including any systems used at any [overseas] facilities.
p3
4.2. Do you maintain email servers at any or all of the company’s divisions/business
units/locations/offices/subsidiaries that exist separately or in addition to the company-wide
server(s)?
4.2.1. Are the systems the same/different from those identified in Question 4.1 above?
Discuss any differences.
4.3. Are end-user emails that appear in any of the following folders stored on (i) the end-user’s
hard drive, (ii) an email server, or (iii) a server of a third party application service provider:
4.3.1. “In Box”?
4.3.2. “Sent Items”?
4.3.3. “Delete” or “trash” folder?
4.3.4. End user stored mail folders?
4.4. If any of your email systems have changed since [insert relevant period], identify any legacy
systems, the current system(s), and the date of the last backup made with each relevant
legacy system.
4.5. Do you have at least one complete (i.e., non-incremental) backup of each of your email
servers for each month [for the period ________ to ________]?
4.5.1. If not, for which months do you not have at least one complete backup?
4.5.2. For those months, if any, for which you do not have a complete backup, do you
have incremental or other backups from which a full backup can be created of all
data as of a given date in each such month?
4.5.3. If so, please describe the nature of such incremental or other backups and identify
the months for which you have them.
4.6. Does each complete email backup contain all messages sent or received since creation of the
immediately prior complete email backup?
4.6.1. Do your email backups contain the messages that are in each employee’s “In Box” as
of the time such backup is made?
4.6.2. Do your email backups contain the messages that are in each employee’s “Sent
Items” folder as of the time such backup is made?
4.6.3. Do your email backups contain the messages that are in each employee’s “delete” or
“trash” folder as of the time such backup is made?
p4
4.6.4. Do your email backups contain the messages that are in each employee's stored mail
folders as of the time such backup is made?
4.6.5. Do your email backups contain the messages that have been stored to each
employee’s hard drive?
4.7. Can specific email boxes contained on email backups be restored selectively?
4.7.1. Does the company have or maintain an index or mapping resource that would serve
as a reference to identify which employees’ email is stored on particular backups?
4.8. As a matter of firm policy, do you overwrite, reformat, erase, or otherwise destroy the
content of the backups of your email servers on a periodic basis?
4.8.1. If so, what is the rotation period?
4.8.2. If the rotation period has changed since [insert date], describe the changes.
4.9. Did you, at any time, have a system that maintained electronic copies of all emails sent or
received by certain of your employees? Do you have such a system now?
4.9.1. If so, describe the system(s) and the date(s) of first use.
4.9.2. If so, does such system(s) contain copies of all emails captured from the date of first
use until the present?
4.9.3. If so, does such system(s) capture a copy of all emails sent and/or received by
employees in [identify relevant departments/groups that might be relevant]?
5. Hard Drives
The questions below concern the current and former local or non-network drives contained in
current or former employees’ laptop and desktop computers or workstations.
5.1. As a matter of firm policy, are employees’ desktop and laptop hard drives backed up in any
way?
5.1.1. If so, under what circumstances?
5.1.2. If so, how long are such backups retained?
5.1.3. Please describe the backup system.
5.2. As a matter of firm policy, are employees permitted to save files, emails, or other
data(excluding system- and application-generated temporary files) to their desktop or laptop
hard drives?
p5
5.3. Since [insert relevant date], has it been technically possible for firm employees to save files,
emails, or other data (excluding system and application generated temporary files) to their
desktop or laptop hard drives?
5.4. Do you implement technical impediments to minimize the opportunity for employees to
save files, emails, or other data (excluding system and application generated temporary files)
to their desktop or laptop hard drives?
5.4.1. Is it possible for employees to override such impediments?
5.5. To what extent has a search been done to determine the extent to which any of the key
custodians in this litigation did, in fact, save files, emails, or other data to their desktop or
laptop hard drives? Flash drives?
5.6. As a matter of firm policy, are employees’ desktop and laptop hard drives erased, “wiped,”
“scrubbed,” or reformatted before such hard drives are, for whatever reason, abandoned,
transferred, or decommissioned?
5.6.1. If so, are, as a matter of firm policy, files, emails, or other data stored on such hard
drives copied to the respective employee’s replacement drive, if any?
5.6.2. If so, as a matter of firm policy, are such files, emails, or other data copied on a “bit-
by-bit” basis?
6. Non-Company Computers
6.1 Does firm policy permit, prohibit, or otherwise address employee use of computers not
owned or controlled by the company to create, receive, store, or send work-related
documents or communications?
6.1.1. If so, what is that policy?
6.2. Is there any technical impediment to employees using computers not owned or controlled by
the firm to create, receive, store, or send work-related documents or communications?
p6
The Sedona Conference® Working Group SeriesSM

& WGSSM Membership Program
The Sedona Conference® Working Group SeriesSM (“WGSSM”) represents the
evolution of The Sedona Conference® from a forum for advanced dialogue to an
open think-tank confronting some of the most challenging issues faced by our
legal system today.
The WGSSM begins with the same high caliber of participants as our regular
“DIALOGUE season conferences. The total, active group, however, is limited to 30-35 instead
of 60. Further, in lieu of finished papers being posted on the website in advance of
the Conference, thought pieces and other ideas are exchanged ahead of time, and
DESIGNED the Working Group meeting becomes the opportunity to create a set of
recommendations, guidelines or other position piece designed to be of immediate
TO MOVE benefit to the bench and bar, and to move the law forward in a reasoned and just
way. Working Group output, when complete, is then put through a peer review
THE LAW process, including where possible critique at one of our regular season conferences,
hopefully resulting in authoritative, meaningful and balanced final papers for
FORWARD IN publication and distribution.
A REASONED The first Working Group was convened in October 2002, and was dedicated to
the development of guidelines for electronic document retention and production.
& JUST WAY. The impact of its first (draft) publication—The Sedona Principles: Best Practices
Recommendations and Principles Addressing Electronic Document Production (March
2003 version)—was immediate and substantial. The Principles was cited in the
” Judicial Conference of the United State Advisory Committee on Civil Rules
Discovery Subcommittee Report on Electronic Discovery less than a month after
the publication of the “public comment” draft, and was cited in a seminal
e-discovery decision of the Southern District of New York less than a month after
that. As noted in the June 2003 issue of Pike & Fischer’s Digital Discovery and
E-Evidence, “The Principles...influence is already becoming evident.”
The WGSSM Membership Program was established to provide a vehicle to allow

any interested jurist, attorney, academic or consultant to participate in Working
Group activities. Membership provides access to advance drafts of Working
Group output with the opportunity for early input, and to a Bulletin Board where
reference materials are posted and current news and other matters of interest can
be discussed. Members may also indicate their willingness to volunteer for special
Project Team assignment, and a Member’s Roster is included in Working Group
publications.
We currently have active Working Groups in the areas of 1) electronic document

retention and production; 2) protective orders, confidentiality, and public access;
3) the role of economics in antitrust; 4) the intersection of the patent and antitrust
laws; (5) Markman hearings and claim construction; (6) international
e-information disclosure and management issues; and (7) e-discovery in
Canadian civil litigation. See the “Working Group SeriesSM” area of our website
www.thesedonaconference.org for further details on our Working Group SeriesSM
and the Membership Program.
p 49
Copyright © 2011,
The Sedona Conference®
All rights reserved.
Visit www.thesedonaconference.org
MEET AND CONFER
CHECKLIST

AT THE OUTSET OF LITIGATION / TRIGGERING OF PRESERVATION DUTY
COMPLETED?
TASK NOTES
YES/NO
Review any document retention policy or policies
currently in place.
Interview relevant personnel (e.g., information

technology professionals, in-house counsel,
individual custodians, etc.) regarding any steps taken
to preserve or delete documents prior to retaining
counsel.
Build a team to oversee the litigation hold process.
Make sure that the roles and responsibilities of the
various team members are clearly defined and
understood.
Assemble a team to assist with and manage the
eDiscovery process (e.g., in-house counsel, litigation
support, outside counsel, service providers, other
experts, etc.). Early decisions can and will impact
the entire process. Proactive, competent team
members are the foundation of any defensible
process.
Build factual support for any electronically stored
information that may have been lost due to the
routine, good-faith operation of an electronic
information system. (See FRCP 37(e)).
Ensure that all retention policies and processes
are appropriately suspended to facilitate
compliance with your duty to preserve relevant
information (e.g., automated archiving/deletion
procedures, backup tape rotation schedules, email
filters, etc.).
Review the entire information infrastructure. How is
information created, transferred, accessed, stored,
and discarded? Account for all physical locations
globally (e.g., headquarters, field offices, home
offices, etc.), cloud and other third-party
environments, backup and disaster recovery
resources, social media, mobile devices, personal
devices and habits (e.g., CFO regularly backs up her
email to DVD, consistently works remotely, and uses
personal email for work purposes), etc.

MEET AND CONFER
CHECKLIST

Inquire about the presence of any structured data
repositories, content management systems, or other
data storage systems.
Review all backup tape rotation, recycling, and
retention procedures.
Understand how data travels through the IT
environment. What are the policies in place that
govern how data moves through the data ecosystem
during the entire information lifecycle (e.g.,
Acceptable Use policies, BYOD policies, encryption,
etc.)?
If possible, chart out the information infrastructure
with a “data map.” Information infrastructures are
extremely dynamic so any attempt at “data mapping”
must leverage an iterative process that develops in
tandem with discovery.
Develop a list of all potential custodians of relevant

data including former employees and third parties.
Interview document custodians and educate them on
their preservation obligations.
Develop a list of all potential sources of relevant

data. Consider grouping data according to specific
categories to promote organization (e.g., HR
databases, sales team emails, etc.).
Determine the number and type of data assets that
have been decommissioned during the relevant time
period and the specific data custodians which would
have had the ability to utilize said assets to generate,
transmit, store, received, discard, or otherwise
manipulate data in any fashion.
Understand the technological systems used for
generating, transmitting, storing, and deleting
information (e.g., What email system is used? Is
email stored locally or remotely? Are there size
restrictions on individual mailboxes?).
Develop a comprehensive protocol to implement the
litigation hold(s). Make sure to have well defined
procedures in place to monitor compliance with and
update the hold(s) periodically according to changes
in scope. It is important to also consider hold
termination to avoid the unnecessary preservation of
data that can be deleted in good faith.

MEET AND CONFER
CHECKLIST

Distribute litigation hold notices.1 Consider
issuing a broad hold immediately upon the triggering
event and then refine as understanding develops.
Understand the types of data generated by and
stored in various locations. What is their format?
Are they proprietary in nature? What type of
metadata do they generate?
Determine if there are any uniquely sensitive data
types maintained within the systems (e.g., ePHI, PII,
trade secrets, technical data under ITAR, data
covered by any foreign privacy regulation, etc.).
Begin to build factual support for data that is
reasonably accessible vs. not reasonably accessible.
(See FRCP 26(b)(2)(B)).
Establish a budget for discovery and get input and
approval from the appropriate parties.
Obtain contact information for all necessary IT
administrators and other personnel. Consider
establishing a liaison to facilitate eDiscovery tasks
between team members and stakeholders.
Evaluate all resources (e.g., technological, in-house

and out-sourced personnel, and processes) in order
to build an effective team and eDiscovery process.
Consider creating a “defensibility binder” whereby all

importance decisions and considerations
surrounding the eDiscovery processes are identified
and tracked.
Understand all applicable rules and regulations,
including any hyper-local rules that a specific court
may impose on eDiscovery related tasks. Make sure
that ethical obligations associated with eDiscovery,
social media, and relevant technology are clearly
understood.
Read and understand FRE 502. Consider

entering into a FRE 502(d) order.

1
This checklist is not meant to be a comprehensive guide to successfully implementing defensible legal holds. However, at the very
least, a hold must inform employees of their duty to identify and preserve any information that may be relevant to a pending, current,
or anticipated litigation or investigation. It should be distributed in writing to key players and those with potentially relevant
information. It should clearly describe the target subject matter, contain specific instructions pertaining to the types of information
that should be preserved, and include contact information for relevant members of the hold implementation team.

MEET AND CONFER
CHECKLIST

Set the stage for good faith cooperation by sending a
brief communication to opposing parties
foreshadowing or highlighting various eDiscovery
issues.


MEET AND CONFER
CHECKLIST

PREPARING FOR A MEET & CONFER
COMPLETED?
TASK NOTES
YES/NO
If data sources outside of the United States are
involved, familiarize yourself with any applicable
privacy regulations and/or blocking statutes.
Consider engaging experts in cross-border
eDiscovery as well as local counsel to assist with in-
country culling & filtering and cross-border data
transfers.
Determine if any foreign language documents are
going to play a role in the matter.
Make a list of key data custodians.
Make a list of all target devices and data sources

(e.g., computers, shared networks, servers, personal
computers, cloud email (e.g., Gmail, Yahoo, Hotmail,
etc.), mobile devices, social media, etc.).
Determine if any of the devices are encrypted.
Determine the operating systems for all servers and

devices (e.g., Unix, Linus, Windows, Macintosh OS,
etc.).
Determine if there are redundant sources of data

within the system (i.e. all emails generated on a
user’s workstation are also backed up onto on a
network server). Consider that data located outside
of the United States may also be located on
domestic storage media.
Estimate collection volume based on the number of

custodians and sizes of the various devices.
Develop a collection timeline.
Determine standard data types that will be collected

(e.g., PST/NSF files, MS Office files, Bloomberg
Messages, audio files, etc.).
Determine if there is any sensitive data that needs to

be specifically identified and isolated (e.g., ePHI, PII,
trade secrets, technical data under ITAR, etc.).

MEET AND CONFER
CHECKLIST

Determine if there are any storage systems that may
present unique eDiscovery challenges (e.g.,
SharePoint, voicemail, audio tapes, etc.).
Determine if any legacy data is within the scope of

the matter and needs to be collected.
Determine if proprietary data needs to be collected

and if licenses are required for processing.
Determine the method of collection and the

personnel best suited to perform the collection (e.g.,
data custodian, forensic collections experts, in-house
IT personnel, etc.).
Determine the specifications for data processing.
Consider a priority custodian list for processing

purposes to streamline review efforts.
Determine the level of deduplication (e.g., custodian

level or global level). Will near deduplication be a
consideration?
Define any other culling filters that may be used to
reduce the scope of data for review (i.e. file
extensions, domains, date ranges, keywords, etc.).
Developing these culling filters is an iterative process
that should involve careful selection, sampling,
testing, cooperation, and possible expert assistance.
Determine what search and retrieval methodologies
will be efficacious for the matter. Note that desired
functionality may require a specific index to be built
(i.e. keyword search, DT Index for proximity
searching in certain platforms, analytics index, etc.).
Determine if any advanced analytics or other

specialized processes will be utilized (e.g., near
deduplication, email threading, conceptual
searching, automatic clustering and categorization,
Technology Assisted Review (TAR), etc.).2

2
Please see the Grossman‐Cormack Glossary of Technology‐Assisted Review for definitions of various terms related
to TAR. Available at http://www.fclr.org/fclr/articles/html/2010/grossman.pdf

MEET AND CONFER
CHECKLIST

If TAR is being considered, build a protocol for
developing “seed sets” or other system-training
intelligence (e.g., random sampling, judgmental
sampling, multi-modal approach, etc.).
Determine if you want to review a statistically valid
sample set or some other threshold.
Determine your confidence level/interval (%).
Consider the use of a statistician to develop metrics

around precision and recall. Consider proportionality
when engaging such an expert as such assistance
may only be beneficial when the TAR process is
challenged.
Define an acceptable overturn rate.
Determine if redaction will be necessary and the

methods used to redact.
Determine resources needed for performing the

relevancy review (i.e. online review platforms and
other technologies, physical review locations, etc.).
Consider if contract attorneys are required and the

difference in hourly rates across various markets in
the U.S. Will reviewers be near-shored or off-
shored?
Determine the timeframe to complete the relevancy

review.
Determine resources needed to complete the

privilege review.
Determine the timeframe to complete the privilege

review.
Decide the format in which you want to receive

productions as well as the format in which you want
to produce.
Determine if there has been any spoliation or lost

data and how best to support/defend any gaps in
data.
Understand the rules in your jurisdiction regarding

spoliation and sanctions.

MEET AND CONFER
CHECKLIST

Assess whether you will request cost-shifting or how
to best defend a cost-shifting request from opposing
counsel.
Assess whether you might move for a protective

order or to compel discovery. How might you best
respond to opposition from opposing counsel?
Decide if you want an IT representative or other

expert to attend the conference(s) with you.
Determine what reports and other metrics you need

to measure and track the process. Evaluate these
against the actual reports and metrics that are
available to you based on your team and technology
and then reconcile any discrepancies.
Decide if you want your IT representative(s) to

communicate directly with the IT representative(s) of
the opposing party.
Decide on the 30(b)(6) witness or witnesses who will

be able to testify about the information infrastructure
as well as document retention policies.


MEET AND CONFER
CHECKLIST

DURING A MEET & CONFER
COMPLETED?
TASK NOTES
YES/NO
Discuss the scope of the matter.
 Data custodians
 Key witnesses
 Geographic scope
 Temporal scope
Discuss the preservation and/or production of

unreasonably accessible data.
Discuss deduplication procedures.
Discuss how to handle document families (e.g.,

parent emails and attachments).
Discuss keyword search terms or conceptually
based search queries.
Discuss the efficacy of advanced analytics and TAR.
If Technology Assisted Review is considered,

discuss the TAR process (e.g., methods for
developing seed sets/training documents, access to
seed sets/training documents and other such work
product, acceptable overturn/error rates, etc.).
Discuss production format (e.g., native, image only,
images with extracted text and load files, hybrid
formats such as searchable images with native
Excels, paper, online access through a platform,
shared online repository, etc.).
Discuss whether metadata will be produced.
Discuss a production timeline and if rolling

productions will be required.

MEET AND CONFER
CHECKLIST

Discuss bates numbering and any endorsing.
Consider protocols for bates numbering native
documents.
Discuss whether cost-shifting/sharing will be
necessary.
Agree on how to handle any sensitive data (e.g.,

ePHI, PII, trade secrets, confidential business
information, technical data under ITAR, etc.).
Decide whether IT representatives from each side

need to talk to one another about the data.
Determine protocols surrounding the clawback of

inadvertently produced privileged material. SEE
FRE 502.
Determine if a quick peek agreement would be
efficacious.


MEET AND CONFER
CHECKLIST

AFTER A MEET & CONFER
COMPLETED?
TASK NOTES
YES/NO
Routinely audit compliance with the litigation hold.
Reissue the litigation hold(s) notice as necessary to

remind custodians of their obligations, include new
custodians or data requests, and update procedures
and other information.
Maintain detailed records of all litigation hold efforts.
Document chain of custody.
Test, analyze, sample, redefine and update keyword

search terms or conceptual categories.
Memorialize the conference and any points of

agreement and disagreement. Consider updating a
“defensibility binder.” Creating a track record of the
various attempts made to cooperate can support
claims that good faith efforts were made to cooperate
when moving for an order to compel discovery under
FRCP 37(a) or a protective order under FRCP 26(c).
Schedule additional conferences as the matter

progresses. Cooperation is a process.

If you have any questions please contact Matthew Knouff, Esq. at:

Complete Discovery Source, Inc. Email: mknouff@cdslegal.com
US Headquarters Direct: 212‐813‐7006
345 Park Avenue Mobile: 646‐660‐4313
New York, NY 10154 Fax: 212‐354‐6626
www.cdslegal.com www.linkedin.com/in/matthewknouff


COMPUTER SYSTEMS
CHECKLIST (EMPLOYEE)

COMPANY NAME
EMPLOYEES NAME
JOB TITLE
DEPARTMENT
CONTACT NUMBER
EMAIL ADDRESS

1. COMPUTERS
Please identify any computer systems in possession or shared by the custodian indicated above
(including home computers and laptops). For each computer system that is used, please answer the
following.
No Computer Device Type Ownership Device Location Operating

Name1 (Company or (Office, home or System
personal) remote)
1
2
3
4

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________

1
Enter a unique designation that distinguishes this system from all others.


COMPUTER SYSTEMS

2. DATA STORAGE
Besides the internal hard drive(s) in the above system(s), please list other portable storage media
related to your position. Note that backups will be treated separately in the next section.
No Type (e.g., flash drive, external Company / Device Location Was this data

drive, CDs, DVDs, fileserver, etc.) personal property (e.g., Office, home transferred to any
or remote) other device?
1
2
3
4

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________

3. BACKUP
If applicable, please enter type, location and schedule of your backups.
No Type Location Backup Schedule

(Local tape, DVD, external drive,
etc.)
1
2
3


COMPUTER SYSTEMS

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
4. MAIL SERVICES
No Email Service Function User Account or Message stored

(e.g., corporate email service, Company use / Email address (e.g., locally
MSN®, Yahoo®, Gmail, etc.) personal use or username@msn.com)
both
1
2
3
4
5

INTERNAL DOMAINS
No. Domain Names (e.g.,
xxxxxxx@cds.com)
1
2
3
4
5

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________


COMPUTER SYSTEMS

5. COLLABORATIVE SYSTEMS
No System (e.g., SharePoint®, Extranet Content Location (e.g., URL, Messages Stored

Management System, TextFlow, etc.) Archives, etc.) Locally? Which
computer(s)?
1
2
3
4

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________

6. DATABASES (IF APPLICABLE)
No Database System Type (e.g., Oracle, Database Name

MS SQL Server, MySQL, SYSBASE, etc.)
1
2
3
4

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________


COMPUTER SYSTEMS

7. MOBILE DEVICES (IF APPLICABLE)
No Device Name2 Device Type Ownership Acceptable Use of SIM Card

(e.g., (Company or Device? Available? /
Blackberry, personal) Device Synched
mobile phone, with any other
camera phone, Device?
tablet devices,
iPOD, etc.)
1
2
3
4

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
8. SOCIAL MEDIA AND NETWORKING (IF APPLICABLE)
No Site Name (e.g., Username Password Acceptable Use of Description of

Facebook, Site (e.g., was the Activity (e.g.,
MySpace, site used for work, frequent user,
YouTube, personal, accessed inactive user,
Twitter, from work, etc.?) etc.)
LinkedIn, Ning,
Blogs
1

2
Enter a unique designation that distinguishes this system from all others.


COMPUTER SYSTEMS

2
3
4

Comments:
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
9. ADDITIONAL SOURCES & COMMENTS
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________


Chain of Custody Short Form
Case/Project Information
Case Name
Matter Name
Client/Matter No.
CDS Contact Information
TITLE NAME OFFICE PHONE NO. CELL PHONE NO. EMAIL
Media
Description
Date Received
Quantity
Chain of Custody
Date & Time
Action Taken
Release By sign & print name
Received By sign & print name
Chain of Custody
Date & Time
Action Taken
Chain of Custody
Date & Time
Action Taken
Chain of Custody
Date & Time
Action Taken
This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be reproduced in any
format nor discussed with any parties without prior written consent of Complete Discovery Source, Inc.
I-MED PHARMA INC. v. Biomatrix, Inc., Dist. Court, D. New Jersey 2011 - Google Sch... Page 1 of 6
I-MED PHARMA INC., Plaintiff,

v.
BIOMATRIX, INC., GENZYME CORPORATION and GENZYME
BIOSURGERY, a Division of Genzyme Corporation. Defendants.
Civ. No. 03-3677 (DRD).
United States District Court, D. New Jersey.
December 9, 2011.
KREISBERG & MAITLAND, LLP, by: Gary Maitland, Esq., New York, New York, Attorneys for Plaintiff.
BINGHAM MCCUTCHEN LLP, by: Jessica S. Boar, Esq., New York, New York, by: Thane D. Scott, Esq., by: Carol E.
Head, Esq., by: Shuan Lue, Esq., Boston, Massachusetts, Attorneys for Defendant.
OPINION
DICKINSON R. DeBEVOISE, District Judge.
This case highlights the dangers of carelessness and inattention in e-discovery. The underlying dispute arises from the
alleged breach of two medical distribution contracts between Plaintiff I-Med Pharma, Inc . ("I-Med") and Defendant
Biomatrix, Inc. [1] Plaintiff alleges that Defendant, in conjunction with its related entities, entered into contracts under
which I-Med would be the exclusive Canadian distributor of several medical products produced by Biomatrix . Plaintiff
further claims that after a merger with Genzyme, Defendant failed to live up to its obligations to produce the licensed
products for distribution.
Presently before the Court is Defendants' Appeal from Magistrate Judge Shipp's September 9, 2011 order. The
September 9 order modified the terms of a prior January 14, 2011 order that required Plaintiff to produce the results of
a forensic examination of its computer system. Under the terms of the new order, Plaintiff would not be required to
produce documents recovered from any unallocated space files found on its system. Defendants claim that Judge
Shipp abused his discretion by refusing to order Plaintiff to conduct a costly privilege review of the 95 million pages of
documents recovered from the unallocated space files. Plaintiff contends that this appeal is a bad-faith effort by
Defendants to force it to spent significant resources on a review of irrelevant data.
For the reasons set forth below, Magistrate Judge Shipp's September 9, 2011 order is AFFIRMED.
I. BACKGROUND
Defendants are biotechnology firms who manufacture a variety of medicines and medical devices. This case concerns
a distribution agreement concerning several medical products manufactured by Defendants. The products in question
http://scholar.google.com/scholar_case?case=8777713665940559114&q=I-Med+Pharma,... 3/27/2013
are the Biomatrix Hylashield ("Hylashield"), the Hylan Surgical Shield ("HSS"), the Hylashield Nite ("Nite"), and the
Hylashield Lite ("Lite"). [2]
On or about August 4, 1994, Plaintiff and Biomatrix entered into an agreement (the "1994 Agreement") whereby
Biomatrix was appointed the exclusive distributor of Hylashield and HSS in Canada, the Bahamas, and the English
speaking Caribbean Island nations. The initial term of the agreement was five years, and the agreement provided that
if it was not terminated in writing by either party at least 90 days prior to its expiration date it would be automatically
renewed for an additional five years. Similarly, on or about October 4, 1995, Plaintiff and Biomatrix entered into an
agreement (the "1995 Agreement") appointing Plaintiff the exclusive distributor in Canada for Nite and Lite. That
agreement also was for an initial term of five years, and it also provided for an automatic five year renewal if neither
party gave notice of termination. Neither party terminated either agreement prior to renewal.
In connection with the Agreements, Plaintiff took steps necessary to obtain certain product registrations and licenses
from Canadian authorities. These included licenses from Health Canada for products distributed under the 1994
Agreement and under the 1995 Agreement, and trademark registrations.
In late 2000, Biomatrix entered into a business combination with Defendant Genzyme. Plaintiff claims that it was
assured on "numerous occasions" that "no changes would be made" that would affect the business conducted under
the Agreements. However, in April 2001, Defendants allegedly closed a facility in Pointe -Claire, Quebec for the
production of the distributed products. Defendants then changed the name of Hylashield Lite to Hylashield CL, and
stopped delivery of the product to Plaintiff under either name.
The Second Amended Complaint alleges that Defendants did not ship sufficient product under the Agreements or use
their best efforts (as mandated by the Agreements) to fulfill Plaintiff's requirements for products. (Doc. No. 44). It also
alleges that after the Genzyme combination Defendants failed or refused to supply products or otherwise perform
under the Agreements. Id. Plaintiff pleads various species of breach of contract and fraud. Id.
The instant dispute concerns data retrieved from a forensic investigation of Plaintiff's computer system. Pursuant to a
May 27, 2010 stipulation between the parties, Defendants hired an expert to conduct a keyword search of I-Med's
computer network, servers, and related storage devices. (Doc. No. 182). The search terms run were as follows: [3]
back order* product* Blephagel expir*
Chalifour and/or allet* Larm loss

Jean-Guy
Grenon revenue* vet* profit*
Domareki and/or agreement Oasis reputation
Wes
contract* credit Bengl and/or refund
Agerup
Hyl* discount Pagano refriger*
CL rebate Balazs and/or Bandi repine*

HS* return Riggs and/or Rofy FDA
HsS pro forma* unidose Health Canada
shield* sample monodosc TPP
Lite Biomatrix visco*
Nite Genzym Laboratories
inventor* I-Drop Unither

manufactur* I-Lid Excelvision
minimum* HA claim
quota* Siccafluid complaint
The search was not limited to targeted document custodians or relevant time periods. Indeed, the search was not even
limited to active files. The expert was instructed to run the search terms across all data on the computer system,
including so-called "unallocated space" — areas of computer memory in which there is no write-protection and in
which deleted and partially deleted files and other temporary data may often be found.
The results should come as no surprise. The broad search terms hit millions of times across the large data set. In the
unallocated space alone, the terms generated 64,382,929 hits. These hits represent an estimated 95 million pages of
data.[4] Plaintiff balked at the prospect of conducting a privilege review of this material and petitioned Magistrate Judge
Shipp for relief from the stipulation. (Doc. No. 219). On September 9, 2011, Judge Shipp conducted a telephone
conference in which the parties explained their positions on the matter. (Doc. No. 236) ("Tr."). After permitting both
sides a full opportunity to be heard, Judge Shipp issued an order permitting Plaintiff to withhold the data found in the
"unallocated space" of its computer systems. (Doc. No. 235). Cogniscent of the costs Defendants had incurred in
extracting and searching this data, the order permitted Defendants to seek reimbursement of those costs from Plaintiff.
Id.
In connection with this order, Judge Shipp made several findings on the record. First, he found that "good cause"
existed to modify his prior order as the "burden on I-Med" would "outweigh any potential benefit that may result." Tr.
32:10, 32:21-24 ("Tr.") (Doc. No. 236). Second, he found that Genzyme had not met its burden of demonstrating the
complete relevancy of the information sought, noting that for all of Defendants' complaints about spoliation, it had not
actually identified any information actually destroyed by Plaintiff. Tr. 32:12-20. Finally, he found that the overbroad
search terms made the likelihood of finding relevant information that would be admissible at trial "minimal." Tr. 33:2-8.
Defendants now appeals from Judge Shipp's order.
II. DISCUSSION
A Magistrate Judge's decision is to be overturned only to the extent that the ruling is "clearly erroneous or contrary to
law." L.Civ.R.72.1(c)(1)(A). The burden of showing that a ruling is "clearly erroneous or contrary to law rests with the
party filing the appeal." Marks v. Struble, 347 F. Supp. 2d 136, 149 (D.N.J. 2004). A finding is clearly erroneous "when
although there is evidence to support it, the reviewing court on the entire evidence is left with the definite and firm
conviction that a mistake has been committed." Dome Petroleum Ltd. v. Employers Mut. Liab. Ins. Co., 131 F.R.D. 63,
65 (D.N.J.1990) (quoting United States v. U.S. Gypsum Co., 333 U.S. 364, 395 (1948)). Moreover, "[a] ruling is
contrary to law if the magistrate judge has misinterpreted or misapplied applicable law." Pharmaceutical Sales and
Consulting Corp. v. J.W.S. Delavau Co., Inc. 106 F.Supp.2d 761, 764 (D.N.J. 2000).
Where an appeal seeks review of a matter within the purview of the Magistrate Judge, such as a discovery dispute, an
even more deferential standard, the "abuse of discretion standard" must be applied. Kounelis v. Sherrer, 529
F.Supp.2d 503, 518 (D.N.J. 2008) "[w]here a magistrate judge is authorized to exercise his or her discretion, the
decision will be reversed only for an abuse of discretion."; see also Holmes v. Pension Plan of Bethlehem Steel Corp.,
213 F.3d 124, 138 (3d Cir. 2000) (discovery orders reviewed for abuse of discretion). An abuse of discretion occurs
"when the judicial action is arbitrary, fanciful or unreasonable, which is another way of saying that discretion is abused
only where no reasonable man would take the view adopted by the trial court. If reasonable men could differ as to the
propriety of the action taken by the trial court, then it cannot be said that the trial court abused its discretion." Lindy
Bros. Builders v. American Radiator & Standard Sanitary Corp., 540 F.2d 102, 115 (3d Cir. 1976).
There is "particularly broad deference given to a magistrate judge's discovery rulings." Farmers & Merchants Nat.
Bank v. San Clemente Financial Group Securities, Inc., 174 F.R.D. 572, 585 (D.N.J. 1997). Federal Rule of Civil
Procedure 26(b)(2)(C) grants the Court considerable authority to limit a party's pursuit of otherwise discoverable
information where the burden of a discovery request is likely to outweigh the benefits. The Court is directed to "limit the
frequency or extent of discovery otherwise allowed by these rules or by local rule if it determines that: (i) the discovery
sought is unreasonably cumulative or duplicative, or can be obtained from some other source that is more convenient,
less burdensome, or less expensive; (ii) the party seeking discovery has had ample opportunity to obtain the
information by discovery in the action; or (iii) the burden or expense of the proposed discovery outweighs its likely
benefit, considering the needs of the case, the amount in controversy, the parties' resources, the importance of the
issues at stake in the action, and the importance of the discovery in resolving the issues." See also Bayer AG v.
Betachem, Inc., 173 F.3d 188, 191 (3d Cir. 1999) ("Although the scope of discovery under the Federal Rules is
unquestionably broad, this right is not unlimited and may be circumscribed . . . The Federal Rules of Civil Procedure
expressly allow a district court to use its discretion and deny discovery requests if the material sought is `unreasonably
cumulative.') (internal citations omitted).
For the reasons described in detail below, this Court finds Judge Shipp's order to be a reasonable exercise of his
considerable discretion in managing the scope of permissible discovery.
B. The Unallocated Space Files

Defendants contend that Judge Shipp abused his discretion by applying the wrong legal standard to determine
whether to permit relief from the stipulation. Defendants claim that "the Magistrate Judge was required to find, that
`exceptional circumstances' would cause I-Med to suffer `manifest injustice' if required to perform its obligations." (Def.
Br. 6). In support of his position, Defendants cite Waldorf v. Shuta, 142 F.3d 601, (3d Cir. 1998) for the proposition that
no discovery stipulation can be modified absent a showing of "manifest injustice." This reading of Waldorf is
overbroad, simplistic, and incorrect.
Waldorf involved litigation over a catastrophic auto accident. The Waldorf plaintiff was a passenger who suffered
debilitating injury after a collision that occurred near a defective traffic light. Waldorf, 142 F. 3d at 606. The plaintiff
sued the drivers, the city that had installed the traffic light, and various city officials. Id. at 607. After an initial $8 million
jury verdict was reversed on appeal, the city defendant proposed a deal. Id. It would stipulate to its liability on the
condition that the trial be bifurcated and the damages trial against all defendants be tried first. Id. The lower court
accepted this stipulation and tried the case as to damages. Id. The result was a $16 million verdict. Id. The city
appealed and obtained a new trial. Id. The city then attempted to vacate the verdict and back out of the stipulation. Id.
The trial court refused to permit it. Id.
On review, the Court of Appeals upheld the trial court's decision to enforce the stipulation, finding that "exceptional
circumstances" justifying relief from the validly executed stipulation did not exist. Id. at 617. In determining whether
"manifest injustice" would result unless the city was relieved from its stipulation the court examined four factors: "(1)
the effect of the stipulation on the party seeking to withdraw the stipulation; (2) the effect on the other parties to the
litigation; (3) the occurrence of intervening events since the parties agreed to the stipulation; and (4) whether evidence
contrary to the stipulation is substantial. . . ." Id. at 617-8 (internal citations omitted). The court concluded that when the
city "made the stipulation prior to the second trial, it had a full understanding of the legal rights it was relinquishing, and
had clear knowledge of the consequences of its stipulation" and that to permit the city to withdraw the stipulation would
be "manifestly unfair to [the plaintiff]. . . ." Id. at 618 -9.
The instant dispute bears little resemblance to the one in Waldorf. The court in Waldof did not speak to the issue of
discovery stipulations and orders. Rather, it addressed the legal standard for vacating a stipulation of liability.[5] While
both might be broadly termed stipulations, they have very different legal significance. A party seeking to modify the
scope of document production before the close of discovery is in a very different position than a party seeking to
withdraw a crucial admission of wrongdoing post-trial. During discovery, the parties are still actively uncovering the
evidence needed to bring a case to trial and have ample opportunity to modify and adjust their litigation strategy to any
important developments. Clearly a court has the power to modify stipulations concerning discovery terms and
deadlines while discovery is still ongoing without the showing of manifest injustice. A court could not effectively
perform its duty to fairly and efficiently manage discovery if every minor change to a stipulated briefing schedule or
deposition date required a showing of "exceptional circumstances" or "substantial and real harm." While courts should
not casually discard agreements between the parties, nor should they abrogate their duty to balance both burden and
the likelihood of uncovering relevant evidence merely because a party made an improvident agreement.
And even if the Court were to apply the Waldorf standard to this case, the factors identified by the Court of Appeals
would still support Judge Shipp's order. The first factor, the effect of the stipulation on the party seeking to modify it,
weighs heavily in favor of modification. A privilege review of 65 million documents is no small undertaking. Even if
junior attorneys are engaged, heavily discounted rates are negotiated, and all parties work diligently and efficiently,
even a cursory review of that many documents will consume large amounts of attorney time and cost millions of
dollars.
At oral argument, Defendants' counsel argued that a less costly privilege review could be done by reviewing only
documents containing the word "privileged" and producing everything else. This argument is not persuasive. [6] Even
when dealing with intact files, potentially privileged information may often be found in emails, memoranda,
presentations, or other documents that are not explicitly flagged as privileged or confidential. And since the data
searched here is likely to contain fragmented or otherwise incomplete documents, it is entirely possible for privileged
material to be found without its original identifying information.
The second factor, the impact on Defendants, does not weigh heavily against modification. Judge Shipp found that
Defendants had failed to demonstrate a likelihood that relevant non-duplicative information would be found in the
unallocated space files. While Defendants claim to have spent thousands of dollars obtaining the data from Plaintiff's
computer system, the September 9 order permits them to seek reimbursement of this expense from Plaintiff. Even if it
had not, this amount pales in comparison to the millions of dollars in additional expenditure that would be required for
Plaintiff to adequately review the material for responsiveness or privilege. Defendants have failed to show that they will
suffer any great prejudice as the result of the modification.
The third factor, the occurrence of intervening events, points weakly in favor of modification. While the precise number
of hits produced was not known in advance and Plaintiff argues that it could not have predicted the volume of material
that the search would uncover, it should have exercised more diligence before stipulating to such broad search terms,
particularly given the scope of the search. In evaluating whether a set of search terms are reasonable, a party should
consider a variety of factors, including: (1) the scope of documents searched and whether the search is restricted to
specific computers, file systems, or document custodians; (2) any date restrictions imposed on the search; (3) whether
the search terms contain proper names, uncommon abbreviations, or other terms unlikely to occur in irrelevant
documents; (4) whether operators such as "and", "not", or "near" are used to restrict the universe of possible results;
(5) whether the number of results obtained could be practically reviewed given the economics of the case and the
amount of money at issue.
Taking all of the factors[7] together, the Court finds that even under the more stringent Waldorf rule, the September 9,
2011 order would be a proper exercise of Judge Shipp's considerable discretion in this case. While Plaintiff should
have known better than to agree to the search terms used here, the interests of justice and basic fairness are little
served by forcing Plaintiff to undertake an enormously expensive privilege review of material that is unlikely to contain
non-duplicative evidence.
And as the routine administration of discovery matters do not even require this heightened level of attention, the
September 9, 2011 order is doubtlessly proper.
III. CONCLUSION
For the foregoing reasons, Magistrate Judge Shipp's September 9, 2011 order is AFFIRMED.
[1] According to the allegations in the Complaint, the other Defendants are entities related to Biomatrix — Defendant Genzyme Corp.
having purchased or merged with Biomatrix in or about December 2000, creating a new entity, Defendant Genzyme Biosurgery.
[2] Hylan, a chemical derivative of hyaluronan, is a proprietary gel manufactured by Biomatrix. It has numerous medical applications
and is found in many of Defendants' products. The products at issue here are varieties of eye drops that contain Hylan fluid.
[3] In addition, many of the terms were also to be run in French.
[4] The Court has no choice but to take the parties at their word concerning these numbers. However it is troubling that the parties
refer to the number of raw hits as though each represented a separate document. Given the volume of hits and search terms used,
this is essentially impossible —statistically speaking terms like "profit," "loss," "revenue," and "profit" frequently occur together, and it
stands to reason that at least some files mentioning product lines would make reference to more than one at the same time.
Consequently, the Court is left to wonder whether the total hit and estimated page numbers are genuinely correct.
[5] Indeed, the lower court was clear that by "stipulation" it meant "an admission which `cannot be disregarded or set aside at will.'"
Waldorf v. Borough of Kenilworth, 878 F.Supp. 686, 691 (D.N.J. 1995) (emphasis added) quoting Wheeler v. John Deere Co., 935
F.2d 1090, 1097 (10th Cir.1991). The stipulation at issue here admits nothing, and neither Waldorf opinion suggests that the holding
was meant to apply to routine discovery orders.
[6] This suggestion was prompted by a question from the Court asking how counsel for the Defendants would conduct a privilege
review of the size and nature of the one that they sought to impose on Plaintiff. In spite of the answer given, it is difficult to believe
that lawyers from Bingham McCutchen regularly disclose large quantities of information from their client's files without examining it.
[7] The fourth factor identified by Waldorf is inapplicable in this case.
Save trees - read court opinions online on Google Scholar.
Not So Fast: 95 Million Reasons to Carefully Select and Limit Search Terms : E-Discover ... Page 1 of 2
E-Discovery Law Alert

Posted at 1:31 PM on January 24, 2012 by Jennifer Marino Thibodaux
Not So Fast: 95 Million Reasons to Carefully Select and Limit Search Terms
It has become commonplace for parties engaged in electronic discovery to discuss and agree upon
“keyword” searches in an effort to limit the overall scope of discovery. A recent decision in the District
of New Jersey, I-Med Pharma, Inc. v. Biomatrix, Civ. No. 03-3677 (DRD), (D.N.J. 2011) , demonstrates
the pitfalls that arise when the parties too eagerly agree to conduct a search for electronically stored
information using an overly broad set of keywords. The case also demonstrates a court ’s willingness to
engage in proportionality analysis to cabin broad discovery.
Biomatrix involved a dispute over two medical distribution contracts, with the plaintiff alleging that the
defendant breached certain exclusivity provisions. During the course of the parties meet and confer
obligations, the plaintiff’s counsel agreed to allow the defendant’s expert to conduct a keyword search of
more than 50 terms on the plaintiff’s computer network, servers, and related storage devices. Counsel
should have known better than to agree to such keywords -- without limits as to time, custodian, or
“active file” status -- that would almost certainly result in millions of hits. In this case, the agreed upon
search yielded more than 64 million hits, approximating 95 million pages of data. Despite agreeing to
conduct such a broad search as part of a previous court order, the plaintiff was forced to seek court
approval to have the prior discovery order modified to further narrow the discovery inquiry. Not
surprisingly, the defendants sought to hold the plaintiff to their initial deal, and also sought costs
associated with the search.
Magistrate Judge Shipp found that (1) “good cause” existed for modification because the plaintiff’s
privilege review of the documents would be unduly burdensome, (2) the defendants did not demonstrate
the relevancy of the documents, and (3) the parties’ overbroad search terms were unlikely to yield
relevant, admissible information. He thus amended the pre-existing discovery order, but held that the
defendants could seek reimbursement of the costs associated with extracting and searching the data on
the plaintiff’s computer system.
District Judge Dickinson R. Debevoise ultimately affirmed Magistrate Judge Michael Shipp’s
modification of the existing discovery order -- finding not only “good cause” to do so, but even
“manifest injustice” if the order was not modified. See Waldorf v. Shuta, 142 F.3d 601, 1998 U.S.
App. (3d Cir. 1998). Judge Debevoise also took the opportunity to provide guidance for parties to keep
in mind when discussing and proposing search terms. In particular, the Court reasoned that the parties
should have been more diligent before agreeing to the broad search terms, and listed the following
factors for parties to consider when evaluating proposed search terms:
1. the scope of the documents searched and whether the search is restricted to specific computers,
http://www.ediscoverylawalert.com/2012/01/articles/litigation-preparedness-strate/not-so-f... 3/27/2013
Not So Fast: 95 Million Reasons to Carefully Select and Limit Search Terms : E-Discover ... Page 2 of 2
file systems, or document custodians;

2. any date restrictions imposed on the search;
3. whether the search terms contain proper names, uncommon abbreviations, or other terms unlikely
to occur in irrelevant documents;
4. whether operators such as ‘and,’ ‘not,’ or ‘near’ are used to restrict the universe of possible
results; and
5. whether the number of results obtained could be practically reviewed given the economics of the
case and the amount of money at issue.
As a practical matter, litigants should pay attention to Judge Debevoise’s guideposts before agreeing to a
broad set of search terms. An eagerness to agree up front to a discovery plan and avoid a fight may only
delay the inevitable if the search terms picked are so broad as to result in an unduly burdensome stack of
material to review. Moreover, courts have become more willing to examine “what’s at stake” in a case
before ordering broad-based discovery; and seeking a more limited discovery order early in the case can
avoid unnecessary expense later on.
Jennifer Marino Thibodaux is an Associate on the Gibbons E-Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2012/01/articles/litigation-preparedness-strate/not-so-f... 3/27/2013
NOT FOR PUBLICATION WITHOUT THE
APPROVAL OF THE APPELLATE DIVISION
SUPERIOR COURT OF NEW JERSEY

APPELLATE DIVISION
DOCKET NO. A-5918-10T2
LLOYD GOLDMARK and ELIZABETH

GOLDMARK,
Plaintiffs-Respondents,
v.
THOMAS MELLINA, BETH MELLINA,

STEVEN NEEDLE and NEEDLE POINT
HOMES, LLC,
Defendants,
and
BRACH EICHLER, L.L.C.,
Appellant.
_____________________________________________________
Submitted 0D\ ² 'HFLGHG -XQH
Before Judges Fisher, Nugent and Carchman.
On appeal from the Superior Court of New

Jersey, Law Division, Union County, Docket
No. L-2053-08.
Brach Eichler, appellant pro se (Bob

Kasolas, of counsel and on the brief).
Wacks & Hartmann, attorneys for respondents

(Christopher E. Hartmann, on the brief).
PER CURIAM
In this appeal, Brach Eichler, L.L.C., contends that the
trial judge abused her discretion in imposing a sanction against
the firm in the amount of $5,502.50, based on the firm's failure
to timely produce certain documents sought in discovery.
Because the record supports the judge's conclusion that the firm
failed to take adequate precautions to prevent the temporary
disappearance of two critical e-mails, we reject Brach Eichler's
arguments and affirm.
In 2007, the parties entered into a contract by which
plaintiffs Lloyd and Elizabeth Goldmark (hereafter "purchasers")
agreed to purchase, for $4,100,000, a home in Westfield to be
constructed by defendants Thomas Mellina, Beth Mellina, Steven
Needle, and Needle Point Homes, LLC (hereafter "sellers");
purchasers provided a $1,000,000 deposit. Disputes arose during
construction. Sellers' counsel at the time, Robert Kraus, Esq.,
wrote to his counterpart on February 6, 2008, suggesting that
purchasers buy the property in its existing stage and complete
the construction with their own suppliers and contractors.
Purchasers rejected this but the parties met to discuss
settlement and settlement documents were later drafted. The
2 A-5918-10T2
alleged settlement, however, was never finalized, and Kraus was
terminated by sellers and replaced by current counsel.1
Purchasers commenced this suit, claiming their construction
disputes had been settled and, pursuant to their settlement
agreement, purchasers were entitled to the return of their
deposit; sellers resisted this contention and claimed purchasers
breached the contract. Prior to trial, the property was sold to
another buyer for a lower price than that which the purchasers
agreed to pay; sellers returned the deposit to the purchasers.
The matter proceeded to a non-jury trial. Among other
things, Judge Frederic Kessler was required to decide: whether
the matter had been settled and whether the settlement should be
enforced; and, if not, whether the contract was breached or
repudiated and, if so, by whom. The trial judge found that a
settlement agreement had not been reached but also held that
sellers had repudiated and breached the contract, thereby
entitling purchasers to retain the deposit that had been
returned prior to trial.
Of relevance to the matter at hand, Judge Kessler relied
upon and quoted extensively two e-mails, between Kraus and the
1
Current counsel was, at the time, affiliated with another law
firm. It is not clear when Brach Eichler assumed the role of
counsel for sellers; however, since Brach Eichler does not claim
that the other law firm is responsible for the imposed
sanctions, the involvement of the other firm is not relevant.
3 A-5918-10T2
sellers, dated March 4 and 31, 2008. Although these documents
did not persuade the judge that the parties' disputes had been
amicably settled, Judge Kessler, as explained in his thorough
written opinion, greatly relied on the two e-mails in finding
that sellers cancelled the contract.
The efforts expended by purchasers in seeking these two e-
mails in discovery lie at the heart of the present controversy.
Early in this suit's history, the purchasers served a subpoena
on Kraus, seeking the non-privileged portions of his file.
Cross-motions resulted in the rescheduling of Kraus's deposition
and an order requiring the production of a privilege log,
pursuant to Rule 4:10-2(e).
Prior to his deposition, Kraus forwarded his file to
sellers' current counsel, who reviewed the file and returned it
to Kraus. Because Kraus was seriously ill, his deposition was
videotaped. The log was provided when the deposition commenced;
it identified, among other documents, the two March 2008 e-mails
exchanged by Kraus and defendant Beth Mellina (hereafter
"Mellina"). Because sellers insisted these e-mails and other
identified documents were privileged, they were not turned over
at the deposition.
4 A-5918-10T2
Subsequent motions led first to the entry of an order
requiring an in camera inspection. 2 In her written decision of
November 4, 2009, Judge Kathryn A. Brock observed that the two
March 2008 e-mails were not in the materials provided for
inspection. Sellers' current counsel responded that he did not
make copies of Kraus's file when previously provided for his
review and the alleged privileged documents were not in his
firm's possession; he also represented to the court that Mellina
did not have possession of copies of the two e-mails. In the
course of these proceedings, sellers' counsel first asserted
that he did not "discuss [with Kraus] what ought to be in the
privilege log." Upon further questioning by the judge, counsel
acknowledged that he was sure he "discussed the log" with Kraus,
and then asserted that what occurred took place a year earlier
and he could not "possibly remember" what he discussed with
Kraus.
In light of the uncertainty about the location of the two
e-mails, purchasers were permitted to amend their complaint to
add a claim of fraudulent concealment, and discovery was
reopened. When re-deposed, Mellina claimed she deleted all sent
and received e-mails and, therefore, no longer had copies of the
e-mails. Purchasers hired a forensic specialist to search the
2
By this time, Kraus had passed away.
5 A-5918-10T2
hard drive on Kraus's computer and the network at Kraus's former
law firm; the e-mails were eventually found on that firm's
server back-up tapes.
The e-mails were then delivered to the judge, who, after an
in camera review, ordered that they be turned over to
purchasers. The judge also granted purchasers' motion to impose
sanctions on Brach Eichler and Mellina because their failure to
preserve the e-mails resulted in additional and unnecessary
efforts by purchasers. In her August 27, 2010 order, Judge
Brock directed Brach Eichler and Mellina to pay the forensic
specialist's $900 fee, as well as reasonable attorneys fees to
be quantified after trial.
The twelve-day non-jury trial before Judge Kessler took
place in October and November 2010. As noted earlier, the two
e-mails greatly influenced the judge's determination of the
issues. The judge quoted extensively from both e-mails.
Kraus's e-mail summarized for sellers "the negotiated
settlement." Mellina's response revealed to Kraus that she had
spoken to her husband and that they were "in accord with your
summary of the negotiated settlement." Judge Kessler also found
Mrs. Mellina's attempt to explain away this

e-mail was preposterous. She essentially
claimed that no agreement was reached, that
everything Mr. Kraus wrote [in his e-mail]
was incorrect, and that when she spoke to
her husband, they were both opposed to the
6 A-5918-10T2
terms contained in [Kraus's e-mail] -- yet
she sent her e-mail because Mr. Kraus was
terminally ill and needed their support.
According to Mrs. Mellina, she told Mr.
Kraus that she accepted the settlement when
in fact she was opposed to it, and she also
lied to him about having obtained her
husband's approval. Not only is this wholly
implausible, but my observations of Mrs.
Mellina's demeanor during this testimony
added to my skepticism. I likewise did not
believe the testimony of Mr. Mellina that he
opposed the settlement and told his wife so.
Rather, I find that the e-mail from

Mrs. Mellina accurately reflected the view
of the Mellinas on March 4 -- that they were
in favor of the settlement and wanted the
agreement with the Goldmarks terminated so
they could find another buyer and "get on to
something else." Whether Mrs. Mellina forgot
about her e-mail or simply did not expect it
to surface (since she had deleted her copy,
and Mr. Kraus'[s] copy had disappeared), it
appears that she did not anticipate that it
would be produced.
After trial, Judge Brock considered purchasers' request for
the quantification of the fees referred to in the August 27,
2010 order; Brach Eichler and Mellina sought reconsideration.
The judge denied reconsideration and awarded purchasers $11,005,
which consisted of $10,075 in attorneys fees, $30 in costs, and
$900 for the forensic specialist. The judge allocated the award
equally, so that Mellina and Brach Eichler were held responsible
to pay to purchasers $5,502.50 each.
7 A-5918-10T2
Mellina has not appealed, but Brach Eichler has, arguing:3
I. [SELLERS'] COUNSEL FULLY COMPLIED WITH R.

4:10-2(e)(1) BY ROBERT KRAUS PREPARING A
PRIVILEGE LOG FOR INSPECTION BY PLAINTIFFS,
WHICH PRECLUDES THE AWARD OF SANCTIONS
BECAUSE IT DISCLOSED THE PRIVILEGED
MATERIALS IN QUESTION, COMPLIED WITH THE
SEPTEMBER 12, 2008 CONSENT ORDER, ALLOWED
THE COURT TO PERFORM AN IN CAMERA REVIEW TO
DETERMINE IF THE DOCUMENTS SHOULD BE
DISCLOSED, AND BECAUSE R. 4:10-2(e) DOES NOT
REQUIRE THE COPYING OR MAINTENANCE OF A
THIRD-PARTY'S FILE.
II. [SELLERS'] COUNSEL FULLY COMPLIED WITH

R. 4:18-1(a) BY KRAUS PRODUCING THE
PRIVILEGED DOCUMENTS IN HIS POSSESSION,
CUSTODY AND/OR CONTROL AS THEY WERE KEPT IN
THE ORDINARY COURSE OF BUSINESS, AND [THEY
WERE] ORGAN[IZ]ED AND LABELLED BY CATEGORIES
IN RESPONSE TO THE SUBPOENA SERVED UPON
ROBERT KRAUS.
III. NOT ONLY DID [SELLERS'] COUNSEL COMPLY

WITH THE COURT RULES, BUT MR. KRAUS
CONFIRMED UNDER OATH HIS ENTIRE FILE WAS IN
HIS POSSESSION AND THAT HE WOULD KEEP THE
FILE SAFEGUARDED IN RESPONSE TO [PUR-
CHASERS'] COUNSEL'S REQUEST TO DO SO.
IV. NO CASE LAW SUPPORTS THE COURT'S AWARD

OF SANCTIONS SINCE [SELLERS'] COUNSEL DID
NOTHING DELIBERATELY IMPROPER TO CONDUCT
DISCOVERY OF THE EMAILS.
V. JUDGE BROCK HERSELF FOUND THAT [SELLERS]

DID NOTHING IMPROPER OR WRONG THAT
RESU[LT]ED IN THE TWO EMAILS MISSING, BUT
STILL ERRO[NE]OUSLY AND WITHOUT LEGAL BASIS
SANCTIONED COUNSEL SIMPLY BECAUSE HE DID NOT
COPY MR. KRAUS'[S] FILE.
3
We have omitted Point I, which solely discusses the standard of
appellate review, and renumbered the remaining arguments.
8 A-5918-10T2
VI. [PURCHASERS] LOST THE ISSUE OF WHETHER A
SETTLEMENT EXISTED AT TRIAL BASED UPON THE
FACE OF THE NON-PRIVILEGED DOCUMENTS IN THE
RECORD, AND ARE THUS NOT ENTITLED TO ANY
SANCTIONS OR FEES DUE TO THEIR WASTEFUL
EFFORT PURSUING PRIVILEGED DOCUMENTS TO
PROVE A SETTLEMENT THAT NEVER EXISTED.
VII. THE TRIAL COURT ERRED IN AWARDING

$5,502.50 TO [PURCHASERS] AS SANCTIONS SINCE
THE ONLY FEES [PURCHASERS] SHOULD HAVE
INCURRED IN SUBPOENAING THE TWO EMA[I]LS
FROM [KRAUS'S LAW FIRM] WAS ONE HOUR OF
THEIR COUNSEL'S TIME TO PREPARE AND SERVE A
SUBPOENA, AND $900 FOR THE COMPUTER FORENSIC
SPECIALIST TO OBTAIN THE EMAILS FROM
[KRAUS'S LAW FIRM'S] HARD DRIVE/BACKUP
TAPES.
We find no merit in these arguments and affirm.
The discovery issue that gave rise to the controversy now
at hand stems from the assertion of an attorney-client privilege
over documents germane to the claims asserted in the trial
court. Purchasers asserted, among other things, that the
parties reached a settlement prior to the commencement of the
suit and that sellers took steps to terminate their contract.
Thus, communications between the parties and their attorneys
prior to suit regarding the alleged settlement and the positions
taken by the parties, through their counsel, were highly
relevant. Sellers, however, also had a colorable claim that the
communications between them and their attorney were insulated by
the attorney-client privilege. Thus, a privilege log was
9 A-5918-10T2
necessary to protect the parties' competing interests in the
disposition of those documents and the judge properly directed
its preparation.
Upon taking the position that relevant material is
privileged or subject to protection from discovery, a litigant
has the obligation of maintaining and eventually disclosing the
material as directed by the court. See, e.g., R. 4:10-2(e)(1).
It would make a mockery of our discovery rules to allow a party
or its counsel -- after identifying privileged information -- to
destroy or carelessly lose or misplace the materials in
question. 4 Here, the judge was not required to find that Brach
Eichler knowingly allowed the disappearance of the e-mails in
order to impose a sanction. The matter was clearly one of
interest to the purchasers, and counsel was obligated to protect
against what either deliberately or accidentally occurred here.
Even if our court rules do not expressly provide a remedy
for these particular circumstances, our judges possess the
inherent authority to require a party or the party's attorney to
compensate a party damaged in this manner. See Lang v. Morgan's
Home Equip. Corp., 6 N.J. 333, 338-39 (1951); Seacoast Builders
4
Sellers' counsel's obligation to preserve those documents
pending the court's further direction arose at that moment, if
not sooner, see RPC 3.4(a), and was particularly enhanced in
this case because one of the parties to those communications was
terminally ill.
10 A-5918-10T2
Corp. v. Rutgers, The State Univ., 358 N.J. Super. 524, 549
(App. Div. 2003); Summit Trust Co. v. Baxt, 333 N.J. Super. 439,
450 (App. Div.), certif. denied, 165 N.J. 678 (2000). As Chief
Justice Vanderbilt wrote for the Court in Lang, supra, 6 N.J. at
338, our liberal discovery procedures are "essential to any
modern judicial system in which the search for truth in aid of
justice is paramount and in which concealment and surprise are
not to be tolerated." Sanctions to enforce conduct that runs
counter to these important principles are "inherent in our
courts" and limited "only to the requirement that they be just
and reasonable in the circumstances." Id. at 338-39. After
careful consideration, we are satisfied that the imposition of
sanctions, as well as their amount, were just, reasonable and
well-deserved.
We find Brach Eichler's other arguments to be without
sufficient merit to warrant discussion in a written opinion. R.
2:11-3(e)(1)(E).
Affirmed.
11 A-5918-10T2
New Jersey State Courts Enter the E-Discovery Arena in Earnest; Award Sanctions for E... Page 1 of 2

Posted at 9:07 AM on June 29, 2012 by Gibbons P.C.
New Jersey State Courts Enter the E-Discovery Arena in Earnest; Award Sanctions
for Email Spoliation
On June 18, 2012, an Appellate Court in New Jersey issued Goldmark v. Mellina, which held that
asserting the attorney-client privilege does not excuse counsel and parties from their obligation to
preserve relevant e-mails or other documents. There, the Court upheld the trial judge’s award of
$5,502.50 in sanctions against a prominent New Jersey law firm because it had failed to timely produce
electronic documents, which had temporarily disappeared, even though the lapse was not knowing.
Because there were virtually no prior opinions (published or unpublished) addressing e-discovery in this
jurisdiction, Goldmark is an important first-step towards providing e-discovery guidance to New Jersey
practitioners.
At issue in this real estate construction litigation was whether the underlying construction dispute was
settled as the purchasers contended or whether the purchasers breached the contract, which was the basis
of the sellers’ claim. Although the Trial Court determined that no settlement agreement had been
consummated, it also held that the sellers had both repudiated and breached the contract. In reaching that
determination, the judge “relied upon and quoted extensively two [March 2008] emails” between the
sellers and their attorney.
Where the sellers and their subsequent attorneys, who were ultimately sanctioned erred, was in
thwarting the purchasers’ efforts to obtain those emails during discovery. Although the sellers’ attorneys
properly produced a privilege log that listed those two e-mails, they failed to provide them to the
judge for an in camera inspection in the context of a discovery motion. As a result, the trial judge
allowed discovery to be re-opened, and the sellers testified that the e-mails had long been deleted.
After the purchasers hired a forensic specialist to inspect the hard drive of the sellers’ former attorney
and the network of his former firm (as he passed away while the litigation was pending), the e-mails
were found on that firm's server back-up tapes. The e-mails were turned over to the judge, examined in
camera and ordered to be turned over to the sellers. The judge awarded sanctions against the sellers and
their counsel “because their failure to preserve the e-mails resulted in additional and unnecessary efforts
by purchasers.” The judge split the sanctions evenly between the sellers and their attorneys. As noted by
the Appellate Division, “[i]t would make a mockery of our discovery rules to allow a party or its counsel
-- after identifying privileged information -- to destroy or carelessly lose or misplace the materials in
question.”
While the court ’s opinion in Goldmark does not give the level of practical guidance that numerous
federal cases have offered such as the seminal case in e-discovery, Zubulake v. UBS Warburg, LLC, it
http://www.ediscoverylawalert.com/2012/06/articles/legal-decisions-court-rules/new-jerse... 3/27/2013
New Jersey State Courts Enter the E-Discovery Arena in Earnest; Award Sanctions for E... Page 2 of 2
does represent a first step by the Appellate Division towards accepting some of the general ESI
preservation principles outlined in Zubulake and its progeny. Hopefully, New Jersey case law and court
rules will soon take the next step and specifically adopt these principles without reservation so that e-
discovery obligations are clear. As we discussed in a prior post, New York state courts appear to be
moving in that direction as evidenced by the decision in U.S. Bank N.A. v GreenPoint Mtge. Funding,
Inc. As the U.S. Bank court held, “We are now persuaded that the courts adopting the Zubulake standard
are moving discovery, in all contexts, in the proper direction.”
Kevin W. Weber is an Associate in the Gibbons Business & Commercial Litigation Department.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2012/06/articles/legal-decisions-court-rules/new-jerse... 3/27/2013
PRECEDENTIAL
UNITED STATES COURT OF APPEALS

FOR THE THIRD CIRCUIT
__________
No. 10-4339
__________
LAUREEN BULL,
Appellant
v.
UNITED PARCEL SERVICE, INC.

__________
On Appeal from the United States District Court

for the District of New Jersey
(D.C. No. 2-07-cv-02291 )

District Judge: The Honorable Dennis M. Cavanaugh
ARGUED SEPTEMBER 15, 2011
BEFORE: SLOVITER, SMITH, and NYGAARD,

Circuit Judges
(Filed: January 4, 2012)

David Zatuchni, Esq. [Argued]
Zatuchni & Associates
287 South Main Street
Lambertville, NJ 08530
Counsel for Appellant
Michael T. Bissinger, Esq.[Argued]

Day Pitney
One Jefferson Road
Parsippany, NJ 07054-2891
Counsel for Appellee
__________
OPINION OF THE COURT

__________
NYGAARD, Circuit Judge
After declaring a mistrial, the District Court dismissed

Laureen Bull’s state-law employment discrimination case as a
sanction for failing to produce originals of certain medical
notes requested by United Parcel Service. Bull maintains that
the District Court abused its discretion by ordering this
sanction. We agree and will reverse and remand for retrial.
I.
A.
Our review is fact-intensive. However, the basic

contours of the case can be summarized as follows. Bull, a
2
part-time employee since 1986, injured her neck and shoulder
on the job in late December 2005. Though she promptly
reported the incident and requested medical attention the
following day, there was a two-week delay before she had
access to a company doctor. She also alleges a number of
instances in which her supervisors ignored, downplayed and
misrepresented her injury to superiors.
The company doctor diagnosed her with contusions

and strains to her shoulder and neck, and restricted her lifting
to twenty-five pounds. She was referred to an orthopedic
specialist who lowered the lifting restriction to twenty
pounds. She was also given physical therapy for
approximately two months. UPS placed her on a temporary
work assignment that did not involve lifting, but at the end of
the 29-day assignment she stopped working and began
receiving workers’ compensation. On March 29, 2006, the
orthopedic specialist opined that—though Bull was only 70
percent recovered—she had reached maximum medical
improvement. The doctor restricted her overhead lifting to 10
pounds, but did not mention other types of lifting.
Bull returned to work and presented the specialist’s

note. She received a new work assignment, but after five
days her new supervisor told her that her medical restrictions
made it impossible for UPS to continue assigning work to her.
UPS advised her to seek permanent disability.
B.
Bull wished to be reinstated and asked her union

representative for help. The representative advised her to get
a second opinion from her own doctor. Through her doctor’s
3
referral, another orthopedic specialist, Dr. Farber, examined
her on June 13, 2006. Farber gave her a note that said among
other things: “patient is capable of lifting 50 pounds or
more.” D.C. No. 2-07-cv-02291, ECF No. 18-14, 10. The
collective bargaining agreement requires that employees be
capable of lifting 70 pounds. Bull faxed the note to the union
representative who, in turn, faxed it to UPS. UPS, however,
found numerous inconsistencies with the note, and told this to
the union representative. 1 Bull’s union representative then
advised Bull to get another note, and to get more information
to satisfy UPS’s issues. Bull called Dr. Farber’s office and
requested another note. She then faxed a second note from
Dr. Farber’s office, dated August 14, 2006. UPS also found
multiple problems with the second note. 2 These two notes
from Dr. Farber’s office have become central to this appeal.
1
UPS found that the Farber note, dated June 13, 2006, listed
conflicting dates and provided contradictory answers on
whether the medical condition was work related. Moreover,
the note ambiguously stated that Bull could lift “50 pounds or
more.” UPS also deemed other portions of the note as
illegible and observed that part of the note was cut off. ECF
No. 18-1, 14-15.
2
UPS asserts that the August 14, 2006 Farber note had
numerous problems, including: inconsistent dates; a
signature differs from the June note; inconsistent answers on
the issue of whether the medical condition was work-related;
and, an ambiguous instruction that “Patient is not able to lift
over 70 pounds.” The note was also cut off at the bottom.
Finally, there were portions of the note that were illegible.
ECF No. 18-1, 16-19.
4
On September 27, 2006, UPS sent a letter to Bull’s
union representative, saying in part:
As you know, we received two

notes from Dr. Farber’s office
regarding Ms. Bull’s ability to
return to work; both notes (dated
June 13, 2006 and August 14,
2006) indicate restricted duty. . . .
The Company also requests that
Ms. Bull produce the original
notes from Dr. Farber’s office due
to the fact that the notes received
to date are blurry and in some
cases illegible.
ECF No. 18-15, 9. The representative contacted Bull, and

requested again a new doctor’s note and more information.
Bull never responded. Instead, she filed a Workers’
Compensation lawsuit and contacted the Equal Opportunity
Employment Commission. She then filed the instant claim in
April 2007. During discovery, Bull turned over new copies
of the Farber notes to UPS in response to their general
discovery requests.
C.
At the March 2010 trial, during Bull’s direct

examination, her counsel sought to introduce copies of the
June 13, 2006 note from Dr. Farber. UPS objected on the
basis of best evidence. During the sidebar that followed, the
District Court asked Bull’s attorney where the original June
13, 2006 note from Dr. Farber was. He responded: “we don’t
5
have - - it doesn’t exist any more. All we have is a copy.”
ECF No. 56-4, 21:24-25. He also pointed out that Dr. Farber
had authenticated the note. UPS responded that they had
“documented letters asking for the originals,” and that, during
this litigation, “we have asked for the originals, and we have
never seen them.” Id. at 23:4-5, 23:18-19. The District Court
ultimately decided to overrule UPS’s objections, concluding
that the argument against admission went to the weight of the
evidence, rather than its authenticity.
Moments later, as Bull’s counsel was about to request

that the note be admitted into evidence, the District Court
interrupted and said to Bull in open court: “Well, before we
do that: Where’s the original of this note?” She answered:
“The original note is in my home. . . .” Id. at 28:6. Surprised
by his client’s response, Bull’s counsel immediately said:
Your honor, I understand what

she just said. I’ve been asking her
for the originals since the very
beginning when Mr. Bissinger
[UPS’s counsel] has been asking
me for the originals. She just kept
telling me that she doesn’t have
them, she’s looked for them but
she doesn’t - - can’t find those
notes anymore, they don’t exist
any more. 3
3
In an affidavit accompanying Bull’s response to UPS’s
motion for sanctions, counsel revisits this statement. He says:
“When queried as to original Farber medical notes, Ms. Bull
indicated that she was unable to locate them and believed
6
Id. at 28:12-17. A few moments later, after a brief sidebar,
Bull’s attorney sought to clarify her statement. The following
exchange occurred in open court.
Q. Laureen, I understood –
have you looked for the original?
Have you actually been able to
find the original note?
A. I have looked for the

original note. Many things going
on in my life with the paperwork,
and the sale—got damage to my
apartment. I can try to find this
note.
they may have been lost due to flooding in her home.” ECF
No. 58-1, 2. He, then, goes on to say: “The Court should be
aware that this is undersigned counsel’s recollection of what
transpired in 2007, but not Ms. Bull’s. Ms. Bull has recently
advised her counsel that she does not recall being asked by
undersigned for the original Farber medical notes and that she
would have conducted a search for these documents back in
2007 is [sic] she had been required to do so.” Id. Moreover,
at oral argument, Bull’s counsel stated emphatically that Bull
did not lie about possessing these documents. Therefore,
without a finding by the District Court, or indeed any
testimony from Bull on this topic, we can neither speculate
upon what Bull understood or intended with regard to the
production of the Farber note originals, nor whether her
responses were credible.
7
Q. Have I asked you to look
for the original?
The Court: Well, wait a minute.

Wait a minute. In response to my
question, you said the original
was at your home. There was no
hesitation.
The Witness: Correct.
The Court: Is the original of

that note at your home?
The Witness: It should be.
The Court: As your attorney

stated, this case has been going
on for years. There were years of
discovery. This note was asked
for. Is there some reason - - have
you made a search for it
previously?
The Witness: No, sir.
Id. at 30:3-21. In the sidebar that followed, the District Court

questioned counsel for Bull and UPS on the appropriate
response to this revelation. UPS advocated excluding the
originals and any copies other than those originally faxed to
them by the union, specifically requesting that the jury be
permitted to see only the document presented to UPS. The
District Court brushed aside UPS’s suggestions and instead
8
decided to declare a mistrial and invited UPS to file a motion
for sanctions.
Bull sent the original June 13 and August 14 notes

from Dr. Farber to the District Court five days after the
mistrial. UPS filed a motion for sanctions, and in October
2010 the District Court ordered the case dismissed with
prejudice to sanction Bull’s conduct. This appeal followed.
II.
A.
The District Court ruled that Bull’s failure to produce

originals of the medical notes was spoliation and it invoked
its inherent authority to order the case dismissed with
prejudice as a sanction. See Schmid v. Milwakee Elec. Tool
Corp., 13 F.3d 76, 78 (3d Cir. 1994). 4 Two questions arise:
first, generally, whether the production of facsimiles and
copies—in place of the originals—can be considered
spoliation; and, second, whether Bull’s specific acts or
omissions in this case provided a reasonable basis to rule that
she spoliated evidence, warranting dismissal with prejudice.
Sanctions for spoliation of evidence are reviewed for an abuse
of discretion. In re Hechinger Inv. Co. of Delaware, Inc., 489
F.3d 568, 574 (3d Cir. 2007).
B.
4
We have jurisdiction over this appeal pursuant to 28 U.S.C.
§ 1291.
9
We first look at whether, generally, failing to produce
original documents can be considered spoliation. Spoliation
is usually referenced in instances where evidence has been
altered or destroyed. See Micron Technology, Inc. v. Rambus
Inc., 645 F.3d 1311, 1320 (Fed. Cir. 2011) (Spoliation occurs
when evidence is destroyed or altered, or when a party fails to
preserve evidence in instances where litigation is pending or
reasonably foreseeable.). We have described it more broadly.
For instance, in a context not involving a mistrial or
dismissal, but while discussing a district court’s decision to
instruct the jury with an adverse inference for spoliation, we
said the following:
When the contents of a document

are relevant to an issue in a case,
the trier of fact generally may
receive the fact of the document's
nonproduction or destruction as
evidence that the party that has
prevented production did so out of
the well-founded fear that the
contents would harm him. Gumbs
v. International Harvester, Inc.,
718 F.2d 88, 96 (3d Cir.1983);
United States v. Cherkasky Meat
Co., 259 F.2d 89 (3d Cir. 1958).
Brewer v. Quaker State Oil Refining Corp., 72 F.3d 326, 334

(3d Cir. 1995) (emphasis added). Indeed, a party’s failure to
produce a document can have the same practical effect as
destroying it and we reaffirm that, under certain
circumstances, nonproduction of evidence is rightfully
characterized as spoliation.
10
Here, though Bull failed to produce the originals, she
did provide UPS with facsimiles and photocopies of the
documents. The District Court concluded that, in spite of
producing copies, Bull’s conduct was spoliation because,
although UPS had not requested a forensic analysis of the
originals, such tests might have yielded information directly
relevant to the question of whether the documents were
authentic. Such an analysis might not be possible on just
copies. We are persuaded that, in some instances, original
documents might yield relevant evidence that is simply not
available from copies. As a result, we conclude with the
District Court that—theoretically—producing copies in
instances where the originals have been requested may
constitute spoliation if it would prevent discovering critical
information. With that said, we reach a very different
conclusion when we turn to the question of whether spoliation
occurred here.
C.
Spoliation occurs where: the evidence was in the

party’s control; the evidence is relevant to the claims or
defenses in the case; there has been actual suppression or
withholding of evidence; and, the duty to preserve the
evidence was reasonably foreseeable to the party. Id. 5 Two
5
The District Court essentially merged its deliberation on
spoliation with its analysis of spoliation sanctions. Though
there is some overlap between the two, there are distinctive
elements of each. We are focusing upon the spoliation
analysis first. However, we note that the sanctions analysis
includes the following factors: “(1) the degree of fault of the
11
of these factors pose little controversy in this case. There is
no question that the documents were in Bull’s control since
she admitted this on the stand and produced the documents
five days after the District Court declared the mistrial. We
also have no doubt that the Farber notes were relevant to both
the claims and defenses of this case since the notes discuss a
central issue in this case: Bull’s capacity to work. The
remaining factors, though, merit closer attention.
D.
With regard to whether Bull intentionally withheld the

original notes from Dr. Farber’s office, the District Court
summarized the facts upon which it made its decision:
Although Plaintiff would have

this Court believe that originals of
the documents in question were
never requested during discovery,
that contention is disingenuous in
light of the record, which
documents that at least as early as
September 27, 2006, Defendant
UPS had requested “that Ms. Bull
produce the original notes from
Dr. Farber’s office . . . .”
party who altered or destroyed the evidence; (2) the degree of

prejudice suffered by the opposing party; and (3) whether
there is a lesser sanction that will avoid substantial unfairness
to the opposing party, and, where the offending party is
seriously at fault, will serve to deter such conduct by others in
the future.” Schmid, 13 F.3d at 79.
12
The District Court also said “despite numerous requests, both
formal and informal, to produce the disputed documents,
Plaintiff never responded.” As a result, the District Court
concluded that:
Common sense and

reasonableness in light of this
record, and the numerous and
repeated references to the fact that
Plaintiff never accounted for the
absence of the originals, point to
nothing less than purposeful
withholding of critical evidence
that can not [sic] be countenanced
or overlooked by this Court.
Although a District Court has discretion to draw inferences

from the record on a party’s intent, it strays beyond the
bounds of its discretion when, as here, there is no factual
basis to do so. The central problem is that the District Court
accepted, without any critical examination,
misrepresentations of the record promulgated by UPS.
We start with the District Court’s foundational

premise: UPS hounded Bull for the originals over the course
of years, but got no response. 6 To the contrary, we count a
6
UPS misstated as fact the following to the District Court:
“We have documented letters asking for the originals.” ECF
No. 56-4, 23:4-5. “I have never been able to get the originals
because they won’t give them to me. . . .” Id. at 24:7-9. “I
sent the trial subpoena. It was very clear I want the originals
13
total of only two requests by UPS for the original documents
in the entire span of this case—dating back to pre-litigation
communications—and neither of these inquiries were
discovery requests. 7
The first communication was UPS’s September 27,

2006, pre-litigation letter requesting the originals. This letter
was addressed to Bull’s union representative, not Bull. UPS
dismisses the distinction because—it says—they have sworn
testimony from the union representative that he told Bull that
UPS wanted the originals. The representative made no such
statement. He only requested that Bull provide new notes,
new information, or maybe better or clearer copies. 8 The
of the notes. Why - - and we have Rule 26, we have all of

these things, and I still don’t have it, and now we’re three
days into trial.” Id. at 37:22-25. “I don’t know what to do
here, Judge. We go through three days of trial. I asked for
the originals notes. I sent a valid subpoena. She apparently
has had them for three years, four years.” Id. at 45:23-46:1.
7
UPS’s request for production of documents in this case
asked for “copies of the documents described herein . . . .” It
did not request originals of these documents.
8
The full text of this portion of the Cherney deposition is as
follows:
Q. Did you ask or attempt to get better

copies of the notes for UPS in response to
[UPS’s request]?
A. Yes, I did.
14
union representative knew that UPS wanted the originals, but
we have no evidence that Bull knew, and we do not have any
basis to impute the union representative’s knowledge of
UPS’s request to Bull.
It does not end there, however. UPS also repeatedly

stresses that, from the beginning of discovery, Bull had the
September 27 letter from UPS. The implication is that Bull
knew about this pre-litigation request for the originals at least
by the early phase of this litigation. Although it may be
Q. Do you recall what you did response to

that?
A. Can you repeat the question?
Q. Sure. I asked you what did you do to get
better copies of the notes or words to that
effect. You said yes you did. I’m asking did
you go to Ms. Bull and ask her to get
something, did you go to Dr. Farber? What
did you do?
A. I had conversations with Laureen in
effect that there was issues with ambiguities
with the note and that to further clarify it, to get
notes that - - a note that would release her back
to full duty.
Q. Did you talk to her about the fact that the
notes in some cases were - - or the company
felt that there was some illegible parts of
the notes or that they were blurry and we
needed clearer copies?
A. Yes.
Robert Cherney Deposition, 57:14-58:12 (emphasis added).
15
technically accurate to say that the letter was in Bull’s
constructive possession, UPS fails to note that it produced the
letter in response to Bull’s general discovery request.
Producing such a letter to Bull in response to her general
requests for documents is not the same thing as presenting
Bull with a specific discovery request for the original notes
from Dr. Farber’s office. Moreover, we would expect that—
had UPS intended this letter to result in Bull turning over the
original notes in question—it would have made at least some
attempt to pursue their production when they did not
materialize. Yet, UPS never raised the nonproduction of the
originals in a motion to compel, or in any other
communication. As a result, any assertion that the pre-
litigation letter of September 27, 2006 to the union
representative put Bull on notice of UPS’s request for
originals fails.
UPS’s second request for the originals came on March

3, 2010—five days before trial commenced—in the form of
an email. Though UPS refers to it as a trial subpoena, the
District Court noted that it did not meet the requirements of
Fed. R. Civ. P. 45. We agree. Indeed, there is no evidence
that the document that UPS calls a subpoena was issued by
the court. As such, given that discovery was closed and that
the email is not a subpoena, we cannot treat this
communication as anything more than a reminder to Bull’s
counsel that, at trial, the best evidence rule would apply to the
notes from Dr. Farber’s office. Moreover, the record does not
contain any evidence that Bull was ever aware of this email.
Therefore, like the September 27, 2006 letter, the email fails
to provide support for an inference that Bull knew of UPS’s
request and still inadvertently or intentionally withheld the
original notes from them.
16
We conclude from this that the District Court was
flatly wrong when it declared that UPS made “multiple”
requests of Bull for the original notes. We must also
conclude that any inference of Bull’s intent to obstinately
withhold the originals also fails for a lack of any factual
foundation that she actually knew UPS wanted them.
The District Court also makes much of UPS’s

challenge to the authenticity of the Farber notes in its motion
for summary judgment, in a pretrial motion, and during its
opening argument. It infers that Bull should have understood
from UPS’s challenge to the authenticity of the notes that
UPS wanted and needed the originals. However, these, alone,
imply no such thing. It is largely counsel’s role to assess
which evidence best serves the case. It was up to UPS’s
counsel to press for the originals if it needed them, and it was
up to Bull’s counsel to diligently investigate all possible
sources for the originals if he needed them to rebut the
argument. If neither counsel pressed Bull for the originals to
make their case, we cannot impute to Bull a sophisticated
knowledge of trial strategy. Indeed, it is obvious from UPS’s
arguments at sidebar that it was content to try the case with
the copies. Moreover, UPS’s challenges to the notes’
authenticity were largely confined to information that was
readily available on the note copies. Bull, and her counsel for
that matter, had no reason to place a high priority on locating
the original notes. 9 Therefore, without anything else, Bull
could not be deemed to be “on notice” that UPS wanted or
needed the originals merely from their authenticity argument.
9
See supra notes 1 and 2.
17
Moving beyond UPS’s requests for originals, the
District Court references a statement made by Bull’s counsel
at trial as a basis to infer her intent. In the instant after Bull
revealed that the original notes from Dr. Farber’s office were
likely in her possession, her counsel said:
Your honor, I understand what

she just said. I’ve been asking her
for the originals since the very
beginning when Mr. Bissinger
[UPS’s counsel] has been asking
me for the originals. She just kept
telling me that she doesn’t have
them, she’s looked for them but
she doesn’t - - can’t find those
notes anymore, they don’t exist
any more [sic].
ECF No. 56-4, 28:12-17. Bull disputes any memory of this

request, and her attorney has moved away from saying that he
made multiple requests of Bull. 10 Yet, even if we accept
counsel’s version of events as told at trial, we do not see how
this exchange between counsel and client can be used to
ground a conclusion that Bull committed, as the District
Court said, “flagrant violations of the most basic code of
judicial propriety and honest dealing.” Absent the backdrop
of UPS’s unsubstantiated portrayal of Bull as an obstinate
stonewaller, this conclusion is dramatically overblown.
10
See supra note 3.
18
This is not to say that we take Bull’s failure to turn
over the originals to her counsel lightly. To be sure, Bull put
her counsel in a terrible position. Her counsel represented in
a side-bar to the District Court—moments before her open-
court revelation—that “we don’t have - - it [the Farber note
original] doesn’t exist any more. All we have is a copy.”
ECF No. 56-4, 21:24-25. He also unwittingly violated the
best evidence rule. Fed. R. Evid. 1002. There is no question
that the late revelation concerning the originals had impacts.
The key issue, however, is whether the discrepancy

between her statements (if in fact there is a discrepancy) was
an intentional misrepresentation or—as her counsel insists—
inadvertence. The record does not answer this question. To
be sure, the District Court was remiss in its failure to examine
this issue more closely and in its failure to make findings of
fact on inadvertence and misrepresentation. Yet, mindful that
it is UPS’s burden to prove Bull’s bad faith conduct, there are
strong reasons favoring a presumption of inadvertence.
Counsel attempted to examine Bull on the record

before the mistrial was declared to get her explanation for her
late revelation, but the District Court inexplicably would not
allow it. 11 There is no evidence that either Bull or her
11
Bull’s counsel said at sidebar:
Bull’s Counsel: I would like to pursue [the reason for Bull

not searching for the Farber notes] with Ms. - - and see ifthere
can be a clarification with her that you can ask her questions
and she can explain. You asked her a question and she gave
you a one-word response, okay? Maybe there’s more to it that
would clarify.”
19
counsel wished to be anything less than candid about this
topic. They were simply prevented from doing so by the
District Court who passed up the opportunity to take
testimony and make findings of fact. Moreover, Bull’s
counsel stated emphatically in oral argument that Bull did not
lie to him about the originals, seemingly ruling out bad faith.
Since the entire case for Bull’s bad faith seems to rest on
counsel’s words, we must take all of his words on the topic
into account, including these. Moreover, even if Bull did
obfuscate on the request from her attorney (a conclusion that
ignores Bull’s counsel’s statements, and the later
representations made by Bull to her counsel), there is still no
evidence that Bull knew UPS wanted the originals, since she
had already produced copies. In light of UPS’s utter failure
to produce any evidence of its own to ground the conclusion
that Bull acted in bad faith, all of this supports abiding by a
presumption of inadvertence.
We must keep in mind that at issue is whether the

District Court was reasonable in ruling that Bull’s late
revelation about the existence of the originals was a “flagrant
violation” and that there is:
[N]o plausible explanation other

than Plaintiff’s misconduct that
explains the withholding of the
original Farber notes. Despite
The Court: No, she gave me a one-word response. Yes, I

have it, it’s at home, and “I’ve never even tried to find it.”
ECF No. 56-4, 42:6-13.
20
numerous requests both formal
and informal, to produce the
disputed documents, Plaintiff
never responded.
On this point, our review of the record has left us with a very
different set of conclusions from the District Court. First, we
do not see any basis for the District Court’s characterization
of UPS as persistently hounding Bull for the originals.
Indeed, it did not. Second, as a result of this, there is not one
instance in which we can verify that Bull actually knew that
UPS wanted the original notes. Third, lacking such evidence
there is no basis to characterize Bull as one who lied or
obfuscated to prevail in her attempt to intentionally withhold
the originals. We conclude from all of this that the District
Court abused its discretion in ruling that, within its spoliation
analysis, Bull intentionally withheld the original documents
from UPS.
E.
We turn, finally, to the issue of whether Bull had a

foreseeable duty to preserve and turn over the originals of the
notes from Dr. Farber’s office. The Court of Appeals for the
Federal Circuit said that the question of reasonable
foreseeability is a “flexible fact-specific standard that allows
a district court to exercise the discretion necessary to confront
the myriad factual situations inherent in the spoliation
inquiry.” Micron Technology, Inc., 645 F.3d at 1320. Here,
the District Court ruled that “it is hard to imagine evidence
that could have been more . . . foreseeable.” We conclude
that the District Court was within its discretion when it
21
determined that there was a foreseeable duty here, but we
have some reservations.
The record is scattered with support for the District

Court’s conclusion. 1.) Bull initiated both an EEOC
proceeding and the instant litigation within a year of UPS’s
employment action. 2.) Bull was aware that she and UPS
fundamentally disagreed on the meaning of the notes from
Dr. Farber’s office. 3.) Though she denies it, her counsel
says that—on at least one occasion—he asked her for the
originals. 4.) Both UPS’s motion for summary judgment and
a pre-trial motion mention the fact that UPS had never seen
the originals. UPS also raised this fact in its opening
argument. 5.) Finally, Bull had a duty under Rule 1002 of the
Federal Rules of Evidence to produce the original documents
before the notes could be introduced into evidence at trial.
With that said, the duty issue is not so clear cut. We

lack evidence that counsel for either party made any
appreciable effort to induce Bull to search for and produce the
original Farber notes. Moreover, it is clear that UPS’s
challenge to the authenticity of the notes rested upon
information that was available on the copies of the
documents. This leads us to wonder whether a lay-person
like Bull, ignorant of the Rules of Evidence, might have
concluded that copies of the notes were sufficient.
Nonetheless, the question before us is not whether a

particular scenario is possible, but rather whether the duty
was objectively foreseeable. Under the circumstances of this
case, we will assume that the District Court acted within its
discretion in determining that the litigation and the future
22
need to provide access to the original notes from Dr. Farber’s
office were foreseeable. 12
F.
12
This highlights a growing concern for us that is not directly
implicated in this case. As electronic document technology
progresses, the concept of an “original” document is
becoming more abstract. Moving from the more easily
distinguishable photocopy or facsimile to documents created,
transmitted and stored in an electronic form means that it will
be increasingly difficult to ascertain where the boundary of an
objectively reasonable duty to preserve such documents lies.
There are—and increasingly will be—circumstances in which
the foreseeability of a duty to preserve the information
contained in a particular document is distinguishable—under
an objective analysis—from the need to preserve that
information in its “original” form or format. Indeed, arriving
at a common understanding of what an “original” is in this
context is challenging enough. Although it does, and always
will rest with the courts to preserve the distinction between an
objectively foreseeable duty and actual knowledge of such a
duty, there is a concomitant obligation that counsel must
assume to clearly and precisely articulate the need for parties
to search for, maintain, and—where necessary—produce
“original” or source documents. This case gives us one more
opportunity to highlight our position that clarity in
communications from counsel that establish a record of a
party’s actual knowledge of this duty will ensure that this
technology-driven issue does not consume an unduly large
portion of the court’s attention in future litigation.
23
In summarizing the Brewer spoliation factors, we
conclude that the District Court was within its discretion in
determining that Bull had the original notes from Dr. Farber’s
office in her possession, that these documents were relevant
to the case’s claims and defenses, and we will assume that she
had a reasonably foreseeable duty to preserve and—when
requested—turn-over these documents. Yet, we conclude that
the District Court abused its discretion in determining that
Bull intentionally withheld these documents from UPS.
In Brewer we discussed the connection between a

finding of sanctionable spoliation and a ruling on bad faith,
stating the following:
For the [spoliation] rule to apply .

. . it must appear that there has
been an actual suppression or
withholding of the evidence. No
unfavorable inference arises when
the circumstances indicate that
the document or article in
question has been lost or
accidentally destroyed, or where
the failure to produce it is
otherwise properly accounted for.
See generally 31A C.J.S.
Evidence § 156(2); 29 Am.Jur.2d
Evidence § 177 (“Such a
presumption or inference arises,
however, only when the spoliation
or destruction [of evidence] was
intentional, and indicates fraud
and a desire to suppress the truth,
24
and it does not arise where the
destruction was a matter of
routine with no fraudulent
intent.”).
Brewer, 72 F.3d at 334 (emphasis added). Therefore, a

finding of bad faith is pivotal to a spoliation determination.
This only makes sense, since spoliation of documents that are
merely withheld, but not destroyed, requires evidence that the
documents are actually withheld, rather than—for instance—
misplaced. Withholding requires intent. 13
As a result, we must be convinced that the District

Court, on sufficient evidence, found that Bull intended to
actually withhold the original documents from UPS before we
can conclude that sanctionable spoliation occurred. There is
no such finding of record here. As we have concluded that
the District Court abused its discretion in ruling that Bull
acted in bad faith, we must rule that the District Court abused
its discretion in determining that Bull committed sanctionable
spoliation. Accordingly, the District Court’s sanction of
dismissal with prejudice, which is grounded in the conclusion
that Bull spoliated evidence, is also an abuse of discretion.
III.
13
We note that the verb “withhold” is defined as “To keep
from doing something, to keep in check or under restraint; to
hold back, restrain.” Oxford English Dictionary.
http://www.oed.com/view/Entry/229665?redirectedFrom=wit
hhold#eid, last viewed December 5, 2011. It inherently
expresses an intentional act.
25
A.
Our decision to reverse the District Court’s ruling on

spoliation eliminates the need for a specific review of the
District Court’s sanction of a dismissal with prejudice, since
it arose from its determination that Bull spoliated evidence.
Yet, the District Court’s rationale for the sanction contained
in its opinion differs markedly from its analysis when it
granted the mistrial and invited a motion for sanctions. When
it granted the mistrial, the District Court based its decision on
a perceived discovery violation that it characterized as
spoliation. In its opinion it retreats from that basis and
instead relies on its inherent power to sanction. The inherent
power of the District Court to sanction parties’ conduct is, of
course, not limited to instances of spoliation. Therefore, in an
abundance of caution, we will review the record to ascertain
whether the circumstances of this case, generally construed,
provide any basis on which the District Court could
substantiate dismissing the case with prejudice.
Generally, “[w]hile we defer to the District Court's

discretion, dismissal with prejudice is only appropriate in
limited circumstances and doubts should be resolved in favor
of reaching a decision on the merits.” Emerson v. Thiel
College, 296 F.3d 184, 190 (3d Cir. 2002). Dismissals with
prejudice are “drastic sanctions.” Poulis v. State Farm Fire
and Cas. Co., 747 F.2d 863, 867 (3d Cir. 1984). Moreover,
district courts ordinarily balance six factors in assessing the
propriety of an involuntary dismissal with prejudice: “(1) the
party's personal responsibility; (2) the prejudice to the
adversary; (3) a history of dilatoriness; (4) willfulness or bad
faith; (5) the availability of alternative sanctions; and (6) the
merit of the claim or defense.” Doe v. Megless, 654 F.3d
26
404, 411 (3d Cir. 2011) (citing Poulis, 747 F.2d at 868). As
the District Court acknowledged, some of the Poulis factors
duplicate the spoliation analysis. Nonetheless, we will review
each one separately.
B.
We begin by looking at Bull’s personal responsibility

for withholding these documents. As we noted earlier, the
District Court concluded that there is:
[N]o plausible explanation other

than Plaintiff’s misconduct that
explains the withholding of the
original Farber notes. Despite
numerous requests, both formal
and informal, to produce the
disputed documents, Plaintiff
never responded.
As we already detailed, however, the District Court’s

inference that Bull intentionally withheld documents is not
grounded in the record. Although Rule 1002 placed
responsibility for producing the originals with Bull and her
counsel, negligence remains a reasonable explanation for
everything that happened. The same can also be said for her
possible misrepresentation of the documents’ whereabouts to
her attorney. The record simply does not have evidence, nor
a factual finding supporting bad faith intent. Moreover, had
the District Court found bad faith conduct on this evidence, it
would have clearly erred. This leads us to conclude that,
though the personal responsibility factor of the Poulis
analysis must weigh against Bull because of the duty she had
27
under Rule 1002, this factor cannot be given great weight
because there is neither evidence of, nor a finding that she
intended to withhold the documents.
C.
The District Court concluded that the next factor,

prejudice, also weighed heavily in favor of dismissal. It said:
Had Defendant been able to prove

that the document was tampered
with, that would obviously have
had significant bearing on the
case. The fact that [UPS] could
not conduct forensic analysis on
the original was certainly
prejudicial, even if the outcome of
such an analysis remains
unknown.
It also ruled that this, combined with the costs of trial

preparation and UPS’s inability to account for the originals in
the development of its trial strategy, resulted in “significant
prejudice.”
Examples of prejudice are “the irretrievable loss of

evidence, the inevitable dimming of witnesses' memories, or
the excessive and possibly irremediable burdens or costs
imposed on the opposing party.” Scarborough v. Eubanks,
747 F.2d 871, 876 (3d Cir. 1984). We have also said:
‘“prejudice’ for the purpose of Poulis analysis does not mean
‘irremediable harm,’ the burden imposed by impeding a
party's ability to prepare effectively a full and complete trial
28
strategy is sufficiently prejudicial.” Ware v. Rodale Press,
Inc., 322 F.3d 218, 222 (3d Cir. 2003).
Certainly, the lack of access, coupled with UPS’s

unreimburseable costs of preparing for trial, and the
disclosure of UPS’s trial strategy all support the District
Court’s use of its discretion in ruling that UPS suffered
prejudice. For this reason, we conclude that the District
Court did not abuse its discretion here.
Again, however, we are dubious about the District

Court’s characterization of the prejudice as “severe” and
“irreparable,” since we do not know the probative value of the
originals (as compared to the copies) and are puzzled by
UPS’s lackadaisical effort in obtaining them—given how
critical they now say they were to their case.
D.
With regard to dilatoriness, the District Court ruled

that it “weighs extremely against Plaintiff.” Yet, as we
addressed above, there is no evidence that Bull was ever
aware that UPS specifically requested the Farber note
originals. Moreover, the first request that UPS made for the
documents during the litigation was literally on the eve of
trial. As a result, we do not find any reasonable basis for the
conclusion that Bull was dilatory as to the production of the
originals to UPS.
With regard to her attorney, we must rely on counsel’s

representations that the delay between his own request for the
original documents and her production of them five days after
mistrial was due to inadvertence or misunderstanding. On
29
this basis, we cannot characterize her conduct as dilatory
because there is no evidence of a willful delay.
E.
The fourth factor, bad faith, already has been discussed

in detail above. Suffice to say, in spite of the District Court’s
strong, negative characterization of Bull on this element, we
conclude that there is not a reasonable basis in the record to
conclude that she willfully withheld the documents or that her
conduct was in bad faith.
F.
With regard to the availability of lesser sanctions, the

District Court said that it had considered them “long and
seriously,” but, ultimately, concluded that anything less than
dismissal with prejudice would “condone flagrant violations
of the most basic code of judicial propriety and honest
dealing.” It went on to say: “[a] sanction other than
dismissal would be insufficient to remedy the prejudice
caused by Plaintiff’s intentional withholding of evidence and
failure to provide appropriate notice to Defendant.” Finally,
it concluded that “there is no way around the fact that
Plaintiff’s conduct has irreparably prejudiced Defendant.”
There are numerous problems with this conclusion.
As we already have said, the record does not support a

ruling that Bull intentionally withheld documents, nor that she
was conducting herself dishonestly in this litigation.
Moreover, any prejudice arising from UPS’s lack of access
was remediable because UPS could have obtained the
originals five days after the mistrial was declared.
30
Second, before the District Court declared a mistrial,
UPS originally argued against a mistrial and strongly
advocated that the District Court remedy the situation by
barring Bull from introducing the originals into evidence. 14
Unlike Ware, where the exclusion of evidence necessitated a
dismissal of the claim, the originals comprised only a portion
of the evidence and UPS was—at least originally—content to
proceed with trial using the copies of the notes that UPS had
always had in its possession.
Finally, though the District Court rejected an adverse

inference instruction as an appropriate remedy, it gave no
solid reason for this. In light of the fact that such an
instruction would have bolstered the very portion of UPS’s
case that it says suffered because it lacked the originals, we
fail to see how an adverse inference—though itself a severe
sanction—would not have been preferable to a dismissal with
prejudice. Given that dismissal with prejudice is a remedy of
last resort, we are puzzled by the District Court’s decision.
For all of these reasons, we conclude that justification

for the dismissal was overblown, and that sanctions other than
14
UPS made the following statements at sidebar, following
the discovery of the Farber originals. “I’m fine with this jury.
I don’t know that we need a mistrial, Your Honor. But using
this note is - - is clearly - - I don’t know what to do.” ECF
No. 56-4, 34:18-20. “I still say, Judge, the appropriate
remedy is to let the jury rely upon what UPS relied upon.” Id.
at 38:1-2. “I think the appropriate remedy again is to allow
the trial to proceed with the faxed versions of the notes.” Id.
at 42:25-43:1.
31
dismissal with prejudice were available to adequately address
any impacts suffered by UPS and the judicial process. The
District Court abused its discretion in concluding that
dismissal with prejudice was the only appropriate remedy.
G.
Finally, in assessing the sixth Poulis factor, the District

Court concluded that the merits of Bull’s claim are “utterly
lacking.” Yet, the District Court appears to have
misunderstood the subject of this portion of the analysis to be
Bull’s defense against sanctions. Poulis directs the District
Court to examine the merits of the underlying claim: which,
in this case, would be Bull’s employment discrimination
claim. Poulis, 747 F.2d at 869-70. Moreover, even if the
discrimination claim was judged to be meritless, Bull also
advanced a second claim of workplace harassment that rested
on a factual foundation that is distinct from the Farber notes.
For all of these reasons, we conclude that the District Court
abused its discretion in concluding that this factor weighed in
favor of dismissal because it did not even address the merits
of Bull’s claim.
H.
To summarize, then, we conclude that two of the

Poulis factors—personal responsibility and prejudice—weigh
in favor of dismissal and four factors—dilatoriness, bad faith,
availability of lesser sanctions, and the merits of the
underlying claim—weigh against a dismissal. We are left
with the question of whether the District Court’s decision to
dismiss the case with prejudice could be regarded as within
its discretion when it is based primarily upon the prejudice
32
UPS suffered. We conclude that it was an abuse of
discretion.
Earlier, we referenced a statement in dicta from

Scarborough in which we provided a list of the types of
events that would support a ruling that a party’s prejudice
alone should result in a dismissal. In that portion of the
opinion, we said:
If there has been true prejudice to

a party by its adversary’s failure
to file a timely or adequate
pleading, discovery response, or
pretrial statement, that factor
would bear substantial weight in
support of a dismissal or default
judgment. Examples of such
prejudice are the irretrievable loss
of evidence, the inevitable
dimming of witnesses' memories,
or the excessive and possibly
irremediable burdens or costs
imposed on the opposing party.
Scarborough, 747 F.2d at 876. We acknowledge that this list

is exemplary and not exhaustive. Yet, it points to the scenario
that we view as typical in a dismissal: the non-responsible
party’s case is severely impaired because it lacked the
information that was not produced. Although UPS had every
right to receive the original Farber notes, there is nothing in
the record to even suggest that it asked for them in discovery,
or that its case was crippled because it lacked them.
Moreover, UPS itself suggested early in the deliberations, that
33
sanctions short of mistrial (and therefore short of dismissal)
were sufficient to address its prejudice. We conclude from all
of this that UPS’s prejudice is not sufficiently weighty to
support the District Court’s sanction of dismissal with
prejudice.
Moreover, since all sanctions originate from the realm

of equity, we note that UPS’s representations to the District
Court, and this Court, were less than candid. It is fair to say
that UPS expended no small effort in obfuscating to the
District Court and this Court the details of its requests for the
originals. We acknowledge that it was the District Court’s
responsibility to plumb the record to learn that UPS did not,
as it repeatedly asserted, hound Bull for the originals. Yet,
UPS’s counsel crossed a line between effective advocacy and
its duty as an officer of the court to accurately present the
record, and in so doing it encouraged the District Court’s
misunderstanding of the record. 15
15
Based upon the following statement at sidebar during trial,
the District Court plainly was unaware of the fact that UPS
never asked Bull directly for the notes, and that there is no
evidence that she was aware of any such request made to
anyone else; yet, UPS never made any effort to correct the
misunderstanding. “So all they did was in my view what
would be a reasonable request. We find out she wouldn’t
give those [Farber notes] to UPS, she wouldn’t give them to
the people that are the union leaders that are supposed to be
assisting her. Then all discussions stop, a lawsuit starts, and
then all of a sudden, after three years of discovery, in the third
day of trial, we find out that these may exist but nobody ever
looked for them. Now come on. . . .This is egregious.” ECF
No. 56-4, 49:7-17.
34
We conclude that, apart from the merits of the appeal,
without the benefit of clean hands here, UPS should not be
the beneficiary of a sanction that we are, under most
circumstances, already loathe to affirm.
IV.
For all of these reasons, we conclude that the District

Court abused its discretion in ordering that Bull be sanctioned
by dismissing her case with prejudice. Therefore, we will
reverse the order of the District Court and remand the cause
for a re-trial.
35
NOT FOR PUBLICATION

STEPHEN D’ONOFRIO, Civil Action No. 09-6220 (AET)
Plaintiff,
v. MEMORANDUM OPINION
BOROUGH OF SEASIDE PARK, et al.,
Defendants.
BONGIOVANNI, Magistrate Judge
As the parties are aware, currently pending before the Court are several formal motions as
well as a number of informally briefed issues. In this Memorandum Opinion, the Court
addresses only the Borough Defendants1 motion for discovery to reclaim inadvertently produced
privileged documents pursuant to FED .R.CIV .P. (“Rule”) 26(b)(5)(B) (“motion for discovery”)
[Docket Entry No. 134]. Plaintiff Stephen D’Onofrio (“Plaintiff”) opposes the Borough
Defendants’ motion. The Court has fully reviewed all arguments made in support of and in
opposition to the Borough Defendants’ motion. The Court considers the Borough Defendants’
1
The term the “Borough Defendants” refers collectively to: the Borough of Seaside Park;
Former Mayor Robert W. Matthies; Mayor Thomas E. Connors; the Seaside Park Borough
Council; Susan Maday; Maryanne Palmisano; John “Jack” Moyse; Benjamin J. Kaiser; Sharon
Pratico; Nancy Koury; James Jablonski; Robert Brennan; John Coughlin; Seaside Park Police
Department; William A. Beining, III; Edward C. Dickson; Daniel Fitzgerald; Lt. Francis Murphy
Larkin; James Leone; Brian McKay; Stephen Shadiack; Brian Jankowski; Seaside Park
Municipal Planning Board; Faith Liguori; Kathleen Hughes; Anthony DiCaro; Robert Bellantoni;
Seaside Park Municipal Zoning Board; Andrew Sbordone; Martin Wilk, Jr.; Kenneth Deshay;
Michael Giuliano; Francis Losey; Michael Tierny; Raymond Sites; Geoffrey N. Schwartz;
Seaside Park Code Enforcement Department; James Anderson; Robert Nora; Rejean Laliberte;
Gary Swirczynski; Richard Barbarise; and Charles Hollins.
motion without oral argument pursuant to Rule 78. For the reasons set forth more fully below,
the Borough Defednants’ motion for discovery is DENIED.
I. Background
The parties and the Court are all familiar with the facts underlying this litigation. As such,
they are not restated at length herein. Instead, the Court focuses mainly on the facts relevant to
the pending motion for discovery.
Plaintiff is the former owner and operator of an establishment located on the boardwalk in
the Borough of Seaside Park known as the SawMill Café (the “SawMill”). In this matter,
Plaintiff has sued numerous individuals and entities associated with the Borough of Seaside
Park.2 Essentially Plaintiff claims that these individuals and entities inappropriately engaged in
2
Following is a complete list of all of the defendants initially sued in this matter. As
noted, two have since been terminated from this case: (1) the Borough of Seaside Park; (2)
Former Mayor Robert W. Matthies; (3) Mayor Thomas E. Connors; (4) the Seaside Park
Borough Council; (5) Susan Maday (Borough Council Member); (6) Maryanne Palmisano
(Borough Council Member); (7) John “Jack” Moyse (Borough Council Member); (8) Robert
Martucci (Borough Council Member); (9) Benjamin J. Kaiser (Borough Council Member); (10)
Sharon Pratico (Borough Council Member); (11) Nancy Koury (Borough Council Member); (12)
James Jablonski (Borough Council Member); (13) Frank Fritz McHugh; (14) Robert Brennan
(Borough Council Member); (15) John Does 1-20 (Seaside Park Council Members); (16) Jay
Delaney, Jr. (Former Clerk and Borough Administrator); (17) John Coughlin (Former Acting
Clerk and Borough Administrator); (18) Julie Keizer (Clerk and Borough Administrator); (19)
CMX Engineering, Inc.; (20) Robert K. Forsyth (P.E.); (21) John Maczuga (P.E.); (22) John
Does 21-40 (Seaside Park Council, Consultants, Agents, Representatives, Employees and/or
Attorneys); (23) Seaside Park Police Department; (24) William A. Beining, III (Former Chief);
(25) Edward C. Dickson (Lt. and/or Chief); (26) Alex Condos (Sergeant); (27) James Citta
(Former Sergeant); (28) James Boag (Sergeant); (29) Daniel Fitzgerald (Patrolman); (30) Lt.
Francis Murphy Larkin; (31) James Leone (Patrolman); (32) Brian McKay (Sergeant); (33)
Stephen Shadiack (Patrolman); (34) Brian Jankowski (Patrolman); (35) John Does 41-60
(Members of Seaside Park Police Department); (36) Seaside Park Municipal Planning Board;
(37) Joseph J. Stack (Planning Board Chairman); (38) John A. Krayesky (Planning Board Vice-
Chairman) (Terminated 4/23/2010); (39) Donald Kadlac (Planning Board Member) (Terminated
5/27/2010); (40) Faith Liguori (Planning Board Member); (41) Kathleen Hughes (Planning
Board Member); (42) Patricia Degutis (Planning Board Member); (43) Anthony DiCaro
2
tortious, fraudulent and extortionate conduct which violated Plaintiff’s right to control and
operate the SawMill. Plaintiff specifically asserts violations of the Equal Protection and Due
Process Clauses of the 14th Amendment, the 1st Amendment, 42 U.S.C. § 1983 of the Civil
Rights Act; 42 U.S.C. § 1985 of the Civil Rights Act; the Racketeer Influenced and Corrupt
Organizations Act (“RICO”), 18 U.S.C. § 1961 et seq.; and N.J.S.A. 2C:41-1 et seq. (“New
Jersey RICO”), which is the New Jersey statute that prohibits racketeering activity. Defendants
deny all of Plaintiff’s allegations.
As part of discovery in this matter, the Borough Defendants reviewed the contents of 14
boxes of documents for possible production to Plaintiff. Of these 14 boxes, 8 came directly from
the Borough Defendants (the “Borough documents”). The other 6 boxes (the “Ryan/McKenna
(Planning Board Member); (44) Robert Bellantoni (Planning Board Member); (45) John Vanna
(Planning Board Member); (46) John Does 61-80 (Seaside Park Planning Board Members,
Consultants, Agents, Representatives, Employees and/or Attorneys); (47) Seaside Park Municipal
Zoning Board; (48) Andrew Sbordone (Zoning Board Chairman); (49) Martin Wilk, Jr. (Zoning
Board Member); (50) Kenneth Deshay (Zoning Board Member); (51) Michael Giuliano (Zoning
Board Member); (52) Francis Losey (Zoning Board Member); (53) Michael Tierny (Zoning
Board Member); (54) Edward Gallagher (Former Zoning Board Member); (55) Raymond Sites
(Former Zoning Board Member); (56) Geoffrey N. Schwartz (Zoning/Code Enforcement
Officer); (57) John Does 81-100 (Seaside Park Zoning Board Members, Consultants, Agents,
Representatives, Employees and/or Attorneys); (58) Seaside Park Code Enforcement
Department; (59) William Schultz (Construction Official); (60) James Anderson (Subcode
Official); (61) Robert Nora (Subcode Official); (62) Rejean Laliberte (Subcode Official); (63)
Gary Swirczynski (Subcode Official); (64) Richard Barbarise (Subcode Official); (65) John Does
101-120 (Seaside Construction Code Enforcement Department and/or Building Department
Members, Consultants, Agents, Representatives, Employees and/or Attorneys); (66) Taxpayers
Association of Seaside Park; (67) John Tweed (Director, Officer, Trustee and/or Member of
Taxpayers Association of Seaside Park); (68) Thomas Hourihan (President of Taxpayers
Association of Seaside Park); (69) John Does 121-140 (Taxpayers Association of Seaside Park,
Members, Consultants, Agents, Representatives, Employees and/or Attorneys); (70) Seaside Park
Democratic Club; (71) Seaside Park Republican Club; (72) ABC Corporations 1-20; (73) XYZ
Partnerships 1-20; (74) John/Jane Does 141-160; (75) Charles Hollins (Planning Board
Member); and (76) Ryan Stack (Patrolman).
3
documents”) came from the files of Guy P. Ryan, Esq. and Michael J. McKenna, Esq., two
attorneys who represented the Borough Defendants in several matters underlying this litigation.
Before producing the information contained in the Borough documents and Ryan/McKenna
documents counsel for the Borough Defendants (“Brown & Connery” or “counsel”) undertook a
multi-attorney review of same during which counsel spent approximately 250 hours reviewing
the documents for responsiveness to Plaintiff’s requests and privilege.3
While associates from Brown & Connery were responsible for reviewing a significant
portion of the Borough documents, a partner undertook the review of the Ryan/McKenna
documents for privilege. In conducting this review, the partner personally reviewed each of the
Ryan/McKenna documents for privilege, made notes concerning the documents flagged as
privileged and then confirmed all privileged notations with Mr. Ryan or Mr. McKenna. The
partner then instructed a clerical employee to separate the privileged Ryan/McKenna documents
from the non-privileged Ryan/McKenna documents based on the designations included in her
detailed notes. The partner further instructed the clerical employee to have the privileged and
non-privileged documents separately Bates stamped and to have the non-privileged documents
burned onto a disc to be produced to Plaintiff. The privileged documents were to be withheld
and set aside so that they could later be identified on a proper privilege log.
Believing that the aforementioned process had been followed, on August 12, 2010, the
Borough Defendants produced what at the time they believed to be the non-privileged Borough
documents and non-privileged Ryan/McKenna documents to Plaintiff. While counsel did review
3
For purposes of this Opinion, unless otherwise defined, the term “privilege” refers to the
attorney-client privilege and includes attorney work product.
4
the final form of the disc of non-privileged documents prior to producing same to Plaintiff,
counsel did not conduct a page-by-page review of the disc. Counsel argues that such a review
was not required and would have been impracticable given the volume of documents being
produced. Indeed, counsel notes that approximately 30,000 pages of material were contained on
the August 12, 2010 disc.
Unfortunately, unbeknownst to counsel, the clerical employee tasked with separating the
privileged Ryan/McKenna documents from the non-privileged Ryan/McKenna documents only
did so for the first 2 of the 6 boxes of Ryan/McKenna documents. As a result, all of the
information the partner designated as privileged from the latter 4 boxes of Ryan/McKenna
documents was not withheld as intended, but instead was produced to Plaintiff. Again, counsel
did not recognize that privileged information was included on the disc produced to Plaintiff
because counsel did not conduct a page-by-page review of the disc before producing same.
After the disc was produced on August 12, 2010, the clerical employee who assisted with
the production of the Ryan/McKenna documents was reassigned and a new secretary began to
work for the partner at Brown & Connery. As part of assisting the partner in this matter, the new
secretary reviewed the format of the disc produced on August 12, 2012. As part of this review,
the secretary realized two things. First, she noticed that some of counsel’s electronic comments,
which were entitled to be withheld as privileged, appeared on the documents produced on the
disc. Second, she realized that the information contained on the disc could be more efficiently
organized – as originally produced each document and sometimes individual pages of a single,
larger document were scanned as separate and distinct pdf files.
5
In light of these discoveries, the Borough Defendants recalled the August 12, 2010 disc.
The partner instructed her new secretary to create a new disc, sans attorney comments and with
merged pdf files for a cleaner production. Instead of having numerous individual pdf files, the
new disc would contain two pdf files, one for the non-privileged Borough documents and the
second for the non-privileged Ryan/McKenna documents. The Borough Defendants produced
this new and more efficiently organized disc on November 4, 2010.
The Borough Defendants claim that at the time they produced the November 4, 2010 disc
they again did not recognize that privileged Ryan/McKenna documents were contained on same
because counsel did not substantively re-review the documents being produced. Instead, counsel
merely directed its clerical staff to remove the electronic comments accidentally included in the
original disc and to merge the numerous individual pdf files to create a cleaner, more efficient
production. As a result, counsel contends that it did not become aware of the inadvertent
production of privileged Ryan/McKenna documents in producing the November 4, 2010 disc.
Counsel similarly argues that it did not become aware of the disclosure of privileged
information in December 2010 when Plaintiff informed the Borough Defendants that the
November 4, 2010 disc was unreadable. Indeed, counsel claims that the minor technical
difficulty identified by Plaintiff on December 9, 2010, i.e. that the disc was unreadable, did not
necessitate a substantive re-review of the information produced. Instead, the Borough
Defendants simply needed to reburn the November 4, 2010 disc, which they did and produced
same to Plaintiff on December 10, 2010, the day after Plaintiff informed them that the second
disc was unreadable. Further, while counsel did conduct a quality control audit of the documents
being reproduced on the December 10, 2010 disc, counsel argues that this audit was performed
6
simply to confirm that the same documents were being produced on this disc as were produced
on the August 12, 2010 disc and which were attempted to be produced on the November 4, 2010
disc. Counsel claims that this review was not substantive in nature and was delegated to a
clerical employee. Counsel emphasizes that the audit was not substantive in nature and did not
entail a re-review for privilege.
In late November 2010, counsel for the Borough Defendants began creating the privilege
logs for the documents initially produced on August 12, 2010. An associate at Brown & Connery
was responsible for creating the logs and before they were produced to Plaintiff, a partner
reviewed the logs for “description information.” (Cert. of Karen A. McGuinness [Docket Entry
No. 134-2] (the “McGuinness Cert.”) ¶ 16). In generating and producing the aforementioned
privilege logs, counsel did not realize that there were fewer privilege entries for the
Ryan/McKenna documents than there should have been based on the partner’s review of said
documents. Counsel argues that they did not detect this discrepancy both because of the amount
of time that had passed between when the Ryan/McKenna documents were reviewed and when
the privilege logs were generated and because of the shear volume of documents produced by the
Borough Defendants.
On March 15, 2011, Plaintiff informed the Borough Defendants that there was a problem
with the information contained on the third disc, i.e. the disc produced on December 10, 2010.
Specifically, Plaintiff noted that it appeared that the documents burned on the third disc were out
of order. In response to Plaintiff’s complaints, counsel for the Borough Defendants reviewed the
information as produced on the December 10, 2010 disc against the information as originally
scanned. From this review, counsel determined that Plaintiff was correct: documents on the
7
December 10, 2010 disc were produced out of order. In reaching this conclusion, counsel did not
review the entire contents of the December 10, 2010 disc. Instead, counsel simply reviewed a
small sampling from the Borough documents, not the Ryan/McKenna documents. Counsel
claims that this sampling was all that was necessary to confirm that a problem existed with the
December 10, 2010 disc.
After confirming that there was, in fact, a problem with the December 10, 2010 disc,
counsel for the Borough Defendants requested that personnel from the firm’s Information
Technology Department (“IT”) identify the cause of the problem as well as which portions of the
disc were affected so that the problem could be rectified. IT reviewed the December 10, 2010
disc and determined that the problem with the disc was limited to only the Borough documents.
The Ryan/McKenna documents were not affected. Apparently, “the problem was attributable to
a computer glitch due to the manner in which the Borough documents were scanned by the
vendor and then uploaded onto [counsel’s] system and subsequently bate stamped.”
(McGuinness Cert. ¶ 30). The Ryan/McKenna documents were processed differently and thus
unaffected.
Counsel for the Borough Defendants corrected the problem identified by Plaintiff on
March 15, 2011 by restoring the affected documents to their original order and reproducing those
documents on a fourth disc in April 2011. Counsel argues that in correcting the problem with the
December 10, 2010 disc, they never intended to re-review any of the documents for privilege.
Counsel, however, states that in reorganizing the documents, the partner working on this matter
identified that certain documents that had been previously flagged as privileged had not, in fact,
8
been withheld, but had mistakenly been produced.4 As a result, the partner added said documents
to the Borough Defendants’ privilege logs and did not include them on the April 2011 disc.
Counsel claims that the fact that certain privileged information from the Borough documents had
been mistakenly produced to Plaintiff did not trigger a re-review of the Ryan/McKenna
documents because the partner had personally reviewed all of the Ryan/McKenna documents for
privilege whereas a substantial portion of the Borough documents had been reviewed by
associates. As a result, counsel argues that they had no reason to believe that privileged
Ryan/McKenna documents had been produced.
On January 12, 2012, Plaintiff filed a reply brief in further support of his motion for a
preliminary injunction [Docket Entry No. 121]. Plaintiff attached numerous exhibits to that
briefing, including Exhibits G, H & I, documents first produced by the Borough Defendants on
the August 12, 2010 disc.
On January 13, 2012, the Borough Defendants began their review of Plaintiff’s reply
brief. During that review, counsel saw that Plaintiff attached Exhibits G, H & I to his reply.
Exhibits G, H & I were Ryan/McKenna documents that the partner at Brown & Connery had
marked as privileged. Based on this disclosure, counsel for the Borough Defendants re-reviewed
its production and recognized that approximately 1000 pages of privileged Ryan/McKenna
documents had been produced. Consequently, on January 18, 2012, counsel for the Borough
Defendants informed all other parties of this inadvertent production and asked that they discard
and not use said documents pursuant to Rule 26(b)(5)(B).
4
In light of the fact that only the Borough documents had been affected by the computer
glitch, there was no need for the Ryan/McKenna documents to be reorganized.
9
II. The Parties’ Arguments
A. The Borough Defendants
Under these circumstances, the Borough Defendants argue that they should be permitted
to reclaim the privileged Ryan/McKenna documents because their disclosure of same was
inadvertent. In the first instance, the Borough Defendants claim that the documents they seek to
reclaim are clearly privileged. In this regard, the Borough Defendants argue that all of the
documents they seek to reclaim represent attorney-client communications made in Messrs. Ryan
and/or McKenna’s representation of the Borough in the disputes underlying this lawsuit or
protected attorney work product generated in connection with same. The Borough Defendants
additionally claim that none of the documents were discussed in open meetings or produced in
the underlying disputes. Further, they note that their privilege log produced on February 10,
2012 sets forth their specific claims of privilege.
The Borough Defendants next turn to the alleged inadvertent nature of their disclosure.
In this regard, the Borough Defendants claim that the production of the privileged
Ryan/McKenna documents was clearly unintentional and therefore inadvertent. Further, the
Borough Defendants argue that their attorneys took reasonable steps to prevent the disclosure of
privileged information. In this regard, the Borough Defendants note that counsel engaged in a
multiple-attorney privilege review of the documents prior to their production. Indeed, the
Borough Defendants note that approximately 250 hours of attorney time was spent reviewing the
documents prior to their production.
In addition, the Borough Defendants claim that the disclosure of the privileged
Ryan/McKenna documents was not unreasonably large. Specifically, the Borough Defendants
10
claim that as part of discovery in this litigation, they produced nearly 100,000 pages of
documents. They note that their disclosure of the privileged Ryan/McKenna documents, which
involved approximately 1,000 pages of documents, amounts to only 1% of their entire
production.
The Borough Defendants also contend that after they learned of the inadvertent disclosure
they acted promptly to rectify the problem. In this regard, the Borough Defendants claim that
within five days of learning of the inadvertent disclosure, they notified all other parties of same
and requested that the inadvertently produced information be destroyed or sequestered.
Moreover, for the reasons set forth above, the Borough Defendants argue that they had no reason
to be believe that privileged Ryan/McKenna documents had been inadvertently produced until
Plaintiff filed Exhibits G, H and I in connection with his reply brief in further support of his
motion for a preliminary injunction.
Further, the Borough Defendants argue that the interests of justice favor a finding that no
waiver of privilege has been made. First, the Borough Defendants note that they have always
maintained the inadvertently produced, privileged Ryan/McKenna documents as privileged.
Indeed, they point out that additional copies of several of the inadvertently produced documents
were in fact logged as privileged in other instances. Second, the Borough Defendants argue that
finding a waiver would derogate the policy of full and open attorney-client communications
which gives rise to the existence of the privilege in the first place. Third, the Borough
Defendants argue that a finding of waiver would prejudice their ability to defend against
Plaintiff’s claims in this matter because Plaintiff will continue to use and cite to the documents
outside of their proper context. Fourth, the Borough Defendants claim that Plaintiff has no
11
legitimate interest in keeping the inadvertently produced documents. In this regard, the Borough
Defendants argue that there is no inherent fairness interest in permitting Plaintiff to keep and use
the documents. The Borough Defendants claim that this is particularly true in light of the fact
that some of the documents were marked “attorney-client privileged” and, as such, Plaintiff knew
or should have known that the documents were inadvertently produced. Fifth, the Borough
Defendants argue that Plaintiff breached his ethical duties under New Jersey’s Rule of
Professional Conduct (“RPC”) 4.4(b)by failing to alert them to their inadvertent disclosure.
B. Plaintiff
In contrast, Plaintiff argues that the Borough Defendants should not be permitted to
reclaim the allegedly privileged Ryan/McKenna documents that they produced. As an initial
matter, Plaintiff claims that the Borough Defendants have not established that any of the
documents they seek to reclaim are, in fact, privileged. Further, Plaintiff argues that the Borough
Defendants’ disclosure of the allegedly privileged Ryan/McKenna documents was not
inadvertent. In this regard, Plaintiff contends that inadvertent does not simply equate to
unintentional. Instead, Plaintiff argues that the Borough Defendants, as the party resisting
waiver, must demonstrate that they took reasonable precautions to avoid the inadvertent
disclosure of the allegedly privileged documents.
Here, Plaintiff claims that the Borough Defendants did not take reasonable steps to
prevent the disclosure of the allegedly privileged Ryan/McKenna documents. Indeed, Plaintiff
notes that the Borough Defendants produced said documents on three separate occasions, the last
of which occurred after the Borough Defendants conducted an audit of the information contained
on the disc being produced. Moreover, Plaintiff argues that the Borough Defendants inexplicably
12
failed to conduct a second review of the produced Ryan/McKenna documents after they learned
that on three separate occasions they had produced 728 pages of privileged Borough documents.
Further, Plaintiff claims that the documents currently in dispute represent a significant
portion of the Borough Defendants’ production. In this regard, Plaintiff notes that, in total, the
Borough Defendants produced 12,318 pages of Ryan/McKenna documents and that the disputed
documents make up 872 of these pages or 7.1% of said production. In addition, Plaintiff argues
that the Borough Defendants failed to act promptly to address the disclosure of the allegedly
privileged Ryan/McKenna documents because the Borough Defendants should have known long
before January 13, 2012 that these documents had been disclosed and the trigger for promptness
is when the party seeking to avoid waiver knew or should have known that the problem existed.
Here, Plaintiff argues that the Borough Defendants should have learned of their disclosure in
November 2010 or at the latest in March 2011 when they discovered that other privileged
information contained on the same disc as the allegedly privileged Ryan/McKenna documents
was produced.
Finally, Plaintiff argues that the interests of justice favor the denial of the Borough
Defendants’ motion. In this regard, Plaintiff notes that at the time the Borough Defendants
initially sought to reclaim the allegedly privileged Ryan/McKenna documents on January 18,
2012, he had already relied on the documents for over a year and a half.
III. Analysis
Motions to reclaim documents that have allegedly been inadvertently produced are
governed by FED .R.EVID . (“FRE”) 502. This Rule provides as follows:
13
Rule 502. Attorney-Client Privilege and Work Product;
Limitations on Waiver
(b) Inadvertent Disclosure. When made in a federal

proceeding or to a federal office or agency, the disclosure
does not operate as a waiver in a federal or state proceeding
if:
(1) the disclosure is inadvertent;
(2) the holder of the privilege or protection took

reasonable steps to prevent disclosure; and
(3) the holder promptly took reasonable steps to

rectify the error, including (if applicable) following
Federal Rule of Civil Procedure 26(b)(5)(B).
Rule 26(b)(5)(B) states that:
If information produced in discovery is subject to a claim of

privilege or of protection as trial-preparation material, the party
making the claim may notify any party that received the
information of the claim and the basis for it. After being notified, a
party must promptly return, sequester, or destroy the specified
information and any copies it has; must not use or disclose the
information until the claim is resolved; must take reasonable steps
to retrieve the information if the party disclosed it before being
notified; and may promptly present the information to the court
under seal for a determination of the claim. The producing party
must preserve the information until the claim is resolved.
In determining whether inadvertently produced documents should be returned, the Court
engages in a two-step analysis. First, the Court must determine whether the documents at issue
are, in fact, privileged. Peterson v. Bernardi, 262 F.R.D. 424, 427 (D.N.J. 2009). Second, if
privileged, then the Court must determine whether the aforementioned three prongs of FRE
502(b) have been met. Id. The disclosing party has the burden of proving that the elements of
FRE 502(b) have been satisfied. Id. Similarly, the burden of proving that a privilege exists rests
14
with the party asserting the privilege. Louisiana Mun. Police Employees Ret. Sys. v. Sealed Air
Corp., 253 F.R.D. 300, 305-06 (D.N.J. 2008). FRE 502 does not alter the placement of same.
Peterson, 262 F.R.D. at 427.
The approach set forth in FRE 502(b) is essentially the same as that used in Ciba-Geigy
Corp. v. Sandoz Ltd., 916 F.Supp. 404, 411 (D.N.J. 1995). Under the approach outlined in Ciba-
Geigy, the Court engages in a multi-factor analysis to determine whether a waiver occurred.
Factors considered are “‘(1) the reasonableness of the precautions taken to prevent inadvertent
disclosure in view of the extent of the document production; (2) the number of inadvertent
disclosures; (3) the extent of the disclosure; (4) any delay and measures taken to rectify the
disclosure; and (5) whether the overriding interests of justice would or would not be served by
relieving the party of its error.’” Id. (quoting Advanced Medical, Inc. v. Arden Medical Sys., Inc.,
Civ.A. No. 87-3059, 1988 WL 26128 at *2 (E.D. Pa. July 18, 1988)).
B. Discussion
As noted above, the Court begins its analysis by determining whether the inadvertently
produced Ryan/McKenna documents are privileged. In this regard, the Court notes that the
Borough Defendants did not go document by document or category of documents by category of
documents through the Ryan/McKenna documents they seek to reclaim to explain why said
documents are privileged or otherwise protected from discovery. Instead, the Borough
Defendants generally argue that the documents are protected from discovery because they are all
either attorney-client communications made in Messrs. Ryan and McKenna’s representation of
the Borough in the disputes underlying this litigation or protected work product generated in
15
connection with same. The Borough Defendants also attach their privilege log dated February
10, 2012 that corresponds to the Ryan/McKenna documents they seek to reclaim.
While the Borough Defendants’ showing is not the strongest, the Court agreed to review
all of the documents subject to this motion in camera. Based on the information provided by the
Borough Defendants as well as the Court’s review of the Ryan/McKenna documents subject to
this motion, the Court makes the following findings:
1. No privilege has been asserted with respect to SP 24327; SP 24963; SP 25008;

and SP 25049. Indeed, the Nature of Privilege/Redaction column on the Borough
Defendants’ February 10, 2012 privilege log indicates that the aforementioned
documents have been “RELEASED.” As such, to the extent the Borough
Defendants seek to reclaim these documents, that request is denied.
2. No privilege has been asserted with respect to SP 21037-38; SP 21039-40; SP

26151-52; SP 26166-67; SP 26199-208; SP 26210-11; and SP 26209. Instead, the
Nature of Privilege/Redaction column on the Borough Defendants’ February 10,
2012 privilege log states that these documents are “NOT RELEVANT.” While
parties to a litigation need only produce relevant information (see Rule 26(b)(1)),
nothing in FRE 502 or the pertinent case law interpreting same suggests that the
Rule was designed to permit a party to reclaim documents that are merely
irrelevant to the pending matter. In fact, FRE 502 explicitly provides that “[t]he
following provisions apply, in the circumstances set out, to disclosure of a
communication or information covered by the attorney-client privilege or
work-product protection.” (Emphasis added). Similarly, Rule 26(b)(5)(B)
pertains to information produced in discovery that is subject to a claim of
“privilege or of protection as trial-preparation material[.]” (Emphasis added).
Here, no such claims were made. Instead, the documents are merely classified as
“NOT RELEVANT.” The Court finds the documents’ alleged lack of relevance
to be an insufficient basis on which to permit the Borough Defendants to reclaim
same. As a result, the Borough Defendants’ request to clawback these documents
is denied.
3. It does not appear the any privilege has been asserted with respect to SP 18702-
07; SP 18728-32; 18761-66; 19051-56; 26109 or 26610-14.5 Instead, the Nature
5
In conducting Its in camera review of the subject Ryan/McKenna documents, the Court
was unable to locate SP 26610-26614. It appears that this document was not submitted to the
Court. Nevertheless, even without specifically reviewing this particular document, the Court is
16
of Privilege/Redaction column on the Borough Defendants’ February 10, 2012
privilege log for these documents reads “POLICE RECORDS - PROTECTIVE
ORDER.”
The Court notes that on January 19, 2012, It entered a Discovery Confidentiality
Order (“DCO”) [Docket Entry No. 125] that had been agreed upon by the parties.
Pursuant to the terms of the DCO, the DCO applies only to police related
documents. (DCO ¶ 2). All of the aforementioned documents designated as
“POLICE RECORDS - PROTECTIVE ORDER” appear to qualify as police
related documents that, absent some other issue, would be subject to the DCO.
Here, while the Borough Defendants do not explicitly so state, the Court assumes
that they do not seek to actually reclaim the aforementioned documents, but
instead seek to ensure that said documents are subject to the DCO.6 The Court
additionally assumes that the Borough Defendants are doing so because Plaintiff
will not agree to the retroactive protection of these documents under the DCO.7 In
this regard, the Court notes that pursuant to its explicit terms, the DCO applies
only to police related documents produced by Defendants after its entry or police
related documents produced by Defendants before the entry of the DCO that were
identified within 15 days of the date of the entry of the DCO and which Plaintiff
agrees will be retroactively subject to the DCO. (DCO ¶ 4). The documents at
issue here were produced approximately 17 months before the DCO was entered.
As a result, absent Plaintiff’s approval or Court Order, the aforementioned
documents would not be subject to the protection afforded by the DCO.
Despite the fact that SP 18702-07; SP 18728-32; 18761-66; 19051-56; 26109 and
26610-14 were produced nearly a year and a half before the DCO was entered, the
Court shall require that these documents be afforded retroactive protection under
same. As a result, while the Borough Defendants shall not be permitted to reclaim
confident with Its determinations as expressed herein.

6
To the extent the Court is incorrect and the Borough Defendants seek to reclaim these
documents, the Borough Defendants’ request is denied. The Borough Defendants have proffered
no arguments to suggest that these police records are privileged or subject to attorney work
product protection and, as explained with respect to the documents identified in category 2, FRE
502 and Rule 26(b)(5)(B) only apply where claims of attorney-client privilege and or attorney
work product protection have been made.
7
Neither party has explicitly stated that this is the case; instead, the Court is inferring
same from the information presented in the Borough Defendants’ motion and Plaintiff’s
opposition thereto.
17
these documents, Plaintiff, in using these documents, must treat them as protected
material under the DCO.
4. The Borough Defendants claim that the following documents are protected by the
attorney-client privilege: SP 19327-28; SP 25053-54; SP 19669; SP 19674; SP
19673; SP 19682; SP 24327; SP 24963; SP 25008; SP 25049; SP 24901; SP
25005; SP 25065; SP 25066; SP 25067; SP 25072; and SP 26457 (collectively the
Clive Samuels documents”). These documents all involve communications with
engineers at Clive Samuels and Associates, Inc. (“Clive Samuels”). The Borough
Defendants’ claims of privilege regarding the Clive Samuels documents turn on
whether Clive Samuels was the Borough’s expert in one of the underlying
disputes that forms the basis for part of this case or an independent court-
appointed expert. The Borough Defendants argue that Clive Samuels was their
expert whereas Plaintiff contends that Clive Samuels was an independent, court-
appointed expert.
The Court has reviewed the information presented by the parties on this issue and
finds that Clive Samuels was the Borough’s expert. This conclusion is based
largely on this Court’s independent review of the transcript of proceedings that
occurred on October 6, 2006 before Hon. Frank A. Buczynski, Jr., J.S.C., as well
as the Order entered by Judge Buczynski immediately after the hearing. In light of
the Court’s decision that Clive Samuels was the Borough’s expert, the Court finds
that the aforementioned documents are privileged.
5. The Borough Defendants claim that all of the remaining documents entered on the
February 10, 2012 privilege log are protected by the attorney-client privilege.
These documents include: SP 1422; SP 16634-40; SP 16666-67; SP 16668; SP
16773-76; SP 16778-81; SP 16740-42; SP 16743-45; SP 16770; SP 16771; SP
16727-30; SP 16731-36; SP 16738; SP 16785-86; SP 16788; SP 16789-90; SP
16799-800; SP 16801-02; SP 16803-04; SP 16812-13; SP 16850; SP 17492-94;
SP 17495-510; SP 17512-50; 17568-87l SP 18198-209; SP 18212-18; SP 18989-
19037; SP 19507; SP 19512; SP 19695-99; SP 19702-28; SP 19744-54; SP
19758-72; SP 19782-86; SP 20106; SP 21113-19; SP 24298-312; SP 21025-36;
SP 24298-312; SP 25381-83; SP 25385-90; SP 25381-83; SP 25385-90; SP
25500-18; SP 26459-61; SP 17568-77; SP 17594-601; SP 17602-12; SP 17613-
28; SP 17635-37; SP 17638; SP 17651; SP 17652; SP 17653; SP 17654-60; SP
17783-86; SP 18005-06; SP 18039; SP 18985; SP 18087; SP 18086; SP 18095;
SP 18096; SP 18097; SP 18373; SP 18560-62; SP 18661-63; SP 18583; SP
18592; SP 18614-15; SP 18625-26; SP 18629-30; SP 18632-33; SP 18635-36; SP
18659-60; SP 18664-71; SP 18874; SP 19118-20; SP 19125; SP 19372; SP
19373; SP 19374-75; SP 19390; SP 19461; SP 19475; SP 19895; SP 19903; SP
19498; SP 19508; SP 19509; SP19533; SP 19536; SP 19537; SP 19556; SP
19614; SP 20088; SP 19687; SP 19688; SP 20090; SP 20091; SP 20092; SP
18
20093; SP 20094; SP 20147-48; SP 20161; SP 20162; SP 20178-82; SP 20223-
26; SP 25456-59; SP 25463-70; SP 25477-85; SP 25491-98; SP 20933; SP
20937-44; SP 20945-47; SP 20955; SP 21047-48; SP 21141; SP 24327; SP
24963; SP 25008; SP 25049; SP 24832; SP 24901; SP 25005; SP 25058; SP
25072; SP 25081; SP 25241; SP 25338; SP 25425; SP 25431; SP 25433-34; SP
25436-37; SP 25439; SP 25443; SP 25448-49; SP 25450;8 SP 26104; SP 26133;
SP 26151-52; SP 26157; SP 26183-86; SP 26226; SP 26227-60; SP 26261; SP
26262; SP 16845; SP 17557; SP 17629-30; SP 17981; SP 18030-31; SP 18033;
SP 18188-96; SP 18220-52; SP 18254-58; SP 18263-66; SP 18271; SP 19893; SP
19919-57; SP 20043-44; SP 20104; SP 20720; SP 21165; SP 24976; SP 14654;9
SP 14657; SP 14658; SP 15661-72; SP 14673; SP 14674; SP 14678-79; SP
14682; SP 14686; SP 14689; SP 14692; SP 14693; SP 14698; SP 14735; SP
14763; SP 14764; SP 14773; SP 14790; SP 14794; SP 14803; SP 14808; SP
15208; SP 15214; SP 15217; SP 15222; SP 17853; SP 17858; SP 17861; SP
17862; SP 17864; SP 17950; SP 18046; SP 18407; SP 18408; SP 18436; SP
18445; SP 18466; SP 19061; SP 19077; SP 19406; SP 19611; SP 19700; SP
19734; SP 19906; SP 21162; SP 21166; SP 21172; SP 21349; SP 21466; SP
21473-74; SP 21494; SP 21500; SP 21504; SP 21597; SP 21601-03; SP 21625;
SP 22068; SP 22362; SP 22365-66; SP 22369-82; SP 22386-87; SP 22390; SP
22394; SP 22397; SP 22400-01; SP 22406; SP 22442; SP 24296; SP 24316; SP
24322; SP 24935; SP 25123; and SP 25342 (the “Remaining documents”). As
noted above, the Court has reviewed all of these documents in camera. Based on
the Court’s in camera review, the Court finds that these documents are properly
identified as privileged.10
8
The Court notes that an entry for this particular document does not appear on the
February 10, 2012 privilege log submitted by the Borough Defendants.
9
The remaining documents including the foregoing were not withheld in their entirety but
produced in redacted format.
10
The Court notes that in his motion to compel, Plaintiff raises additional reasons, other
than waiver through their production to him, why some of these documents are not properly
subject to the Borough Defendants’ claims of privilege. Examples of such reasons include, but
are not limited to, the crime-fraud exception to the attorney-client privilege, the fact that the
Borough Defendants allegedly placed certain of these otherwise protected communications at
issue and that a waiver of the privilege occurred because the communications were shared with
unnecessary third-parties. The Court has not considered these arguments in considering the
Borough Defendants’ motion for discovery, but shall do so in deciding Plaintiff’s motion to
compel. As a result, the findings described above are made without prejudice to the Court’s later
consideration of Plaintiff’s motion to compel. The same holds true with respect to the Court’s
determination concerning the Clive Samuels documents.
19
Having determined that the Clive Samuels documents and the Remaining documents are
privileged, the Court next addresses whether these categories of documents were inadvertently
produced under Rule 502(b) and the multi-factor test outlined in Ciba-Geigy. Because the Court
has determined that the other categories of Ryan/McKenna documents subject to this motion are
not privileged, the Court need not reach the question of whether they were inadvertently
produced. Peterson, 262 F.R.D. at 427 (holding that “[i]t is axiomatic that FRE 502 does not
apply unless privileged or otherwise protected documents are produced.”) The Court, however,
notes that for practical purposes the remaining analysis would apply equally to these other
categories of documents had the Court determined that they were privileged.
The Court has no doubt that the Borough Defendants’ production of the Clive Samuels
and Remaining documents was inadvertent in the sense that their production was unintentional.
However, while necessary to a finding under FRE 502(b) that the production of these documents
does not amount to a waiver, because “every inadvertent disclosure is an unintentional
disclosure[,] the fact that the Borough Defendants have convinced the Court that said disclosure
was unintentional “does not go far in establishing the absence of waiver.” Ciba-Geigy, 916
F.Supp. at 412. Instead, the other factors would also have to point to an absence of waiver.
Peterson, 262 F.R.D. at 429 (citing Ciba-Geigy, 916 F.Supp. at 411) (noting that producing
party’s “subjective intent is not controlling.”); Sensient, 2009 WL 2905474, at *4 (same).
As to these other factors, the Court finds that the Borough Defendants initially took
reasonable steps to prevent the disclosure of their privileged information. In this regard, the
Court notes that counsel for the Borough Defendants engaged in a multi-attorney review of the
Borough documents and Ryan/McKenna documents devoting approximately 250 hours of
20
attorney time to reviewing these documents for privilege. Moreover, given the likelihood that
privileged information would be contained in the Ryan/McKenna documents, a partner at Brown
& Conner personally reviewed all of the Ryan/McKenna documents for privilege. That partner
took detailed notes concerning which documents she believed were privileged, confirmed the
privilege designations with counsel in the matters underlying this lawsuit and marked in her
notes all privileged documents that were to be withheld from production. This privilege review
was more than adequate and certainly indicates that reasonable precautions were taken to prevent
the inadvertent disclosure of privileged information. Cf. Peterson, 262 F.R.D. at 429 (finding
that conclusory statement that privilege review was conducted without any specific information
concerning when review occurred, how much time was spent reviewing documents, what
documents were reviewed or other basic details of review process is not informative and does
little to establish reasonableness of precautions taken to avoid disclosure).
Also reasonable was counsel’s decision to delegate to a non-attorney, clerical employee
the task of separating the flagged, privileged Ryan/McKenna documents from the rest of the
production. Indeed, the Court finds no reason why, after counsel invested substantial resources
in reviewing the Borough and Ryan/McKenna documents for privilege, the task of physically
separating the documents flagged as privileged from the non-privileged documents and sending
the non-privileged information to a vendor to be scanned on a disc for production would also
have to be performed by an attorney or paralegal. Instead, this is exactly the type of task that
could reasonably be assigned to clerical staff.
Moreover, the Court finds that the Borough Defendants were not obligated to review the
disc before producing same to Plaintiff. Indeed, as explicitly noted in FRE 502(b)’s Explanatory
21
Note, “[t]he rule does not require the producing party to engage in a post-production review to
determine whether any protected communication or information has been produced by mistake.”
Thus, having implemented reasonable steps to prevent the inadvertent disclosure of privileged
information, the Borough Defendants were permitted to rely on same, at least until there were
“any obvious indications that a protected communication or information ha[d] been produced
inadvertently.” Explanatory Note, FRE 502(b).
Turning next to the number and extent of the disclosures, the Court finds these factors to
be neutral. As part of this litigation, the Borough Defendants have produced approximately
100,000 pages of documents, which represents a relatively substantial document production. See
Sensient, 2009 WL 2905474, at *4 (finding that document production consisting of 47,000
documents reviewed with 45,000 documents (135,000 pages) produced was substantial). Given
the size of the Borough Defendants’ production, mistakes were bound to occur. Id. At issue here
is the disclosure of 872 pages of documents, which represents only about 1% of the Borough
Defendants’ total production, which on a total percentage basis is notably small. Nevertheless,
the Court finds that 872 pages of documents is not insignificant.
Moreover, the Court does not necessarily agree with the Borough Defendants that the
extent of their disclosures should be based on their total document production. While that
method was apparently used in Sensient, 2009 WL 2905474, at *4, and Peterson, 262 F.R.D. at
429, the Court in Ciba-Geigy focused not on the defendants’ total production but on the 681
documents counsel had designated for use in two depositions. 916 F.Supp. at 407, 414. If the
Court focuses more narrowly on the total number of documents produced on August 12, 2010
(approximately 26,000 pages) or even more so on only the Ryan/McKenna documents
22
(approximately 12,000 pages), the extent of the disclosure increases to approximately 3% or 7%
respectively, more significant figures.
Furthermore, even were the Court to focus solely on the Borough Defendants’ disclosures
relative to the Borough Defendants’ total document production, i.e., the 1% figure pressed by the
Borough Defendants, the Court would be mindful of the fact that the disclosures at issue in this
motion are not the only unintentional disclosures made by the Borough Defendants in this case.
Instead, though not at issue here, as part of their initial August 12, 2010 production, the Borough
Defendants also produced 728 pages of privileged Borough documents. In addition, as in
Peterson, the Court would also be mindful of “the nature of the disclosures” at issue. 262 F.R.D.
at 429.
Here, the disclosures in question largely involve communications between the Borough
Defendants and their counsel in the legal issues underlying this lawsuit. As in Peterson, these
communications “warranted a significant level of scrutiny.” Id. (finding that documents
representing exchanges between plaintiff and his attorney warranted significant level of scrutiny).
Thus, even assuming that the extent of the Borough Defendants’ disclosures was quite small -
1% of their total document production - this fact would be balanced by the nature of the
disclosures at issue. For these reasons, the Court finds that the number and extent of the
Borough Defendants’ disclosures are neutral factors.
As to the Borough Defendants’ efforts to rectify their disclosure of the privileged
Ryan/McKenna documents, the Court finds that the Borough Defendants did not take reasonable
steps to remedy their error. In this regard, the Court finds that the Borough Defendants should
have been aware that something was amiss with their document production long before Plaintiff
23
relied on three privileged documents as part of his reply brief filed in further support of his
motion for a preliminary injunction on January 12, 2012. While FRE 502(b) “does not require
the producing party to engage in a post-production review to determine whether any protected
communication or information has been produced by mistake[,]” it “does require the producing
party to follow up on any obvious indications that a protected communication or information has
been produced inadvertently.” Explanatory Note, FRE 502(b). As a result, once a producing
party is “on notice that something [i]s amiss with its document production and privilege
review[,]” then that party has an obligation to “promptly re-assess its procedures and re-check its
production.” Sensient, 2009 WL 2905474, at * 5. The Borough Defendants simply did not do so
here, despite being provided with clear signs that something went awry with its document
production and privilege review.
While a number of the seemingly unending problems the Borough Defendants
experienced with their document production did not involve the disclosure of protected
information, several did. Indeed, as early as November 2010, the Borough Defendants were
aware that protected information in the form of counsel’s electronic comments had been
produced to Plaintiff on the August 12, 2010 disc. Yet this disclosure did not prompt the
Borough Defendants to review the entirety of their August 12, 2010 production for other
mistakes. The Borough Defendants seem to suggest that such a review was unnecessary; though
they do no provide much detail as to why. The Court assumes that the Borough Defendants’
position is based primarily on the fact that the inclusion of counsel’s protected electronic
comments would in no way indicate that actual documents that had been identified as privileged
had also been erroneously included in their production.
24
The reasonableness of the Borough Defendants’ decision not to re-review its August 12,
2010 production depends in part on the circumstances surrounding the inclusion of the electronic
comments in the production. The Court, however, has been provided with scant information in
this regard. Indeed, the Court does not know how counsel’s electronic comments came to be
produced on the August 12, 2010 disc, who was responsible for removing same prior to
production, etc. This information is certainly relevant because if the same clerical employee
responsible for sorting the privileged Ryan/McKenna documents from the non-privileged
documents was also responsible for removing the electronic comments, then the Borough
Defendants would have had more reason to suspect that other mistakes occurred in the August
12, 2010 production based on the disclosure of counsel’s electronic comments. Regardless, the
Court is willing to assume for purposes of this motion that the disclosure of counsel’s electronic
comments in the August 12, 2010 production would not, standing alone, reasonably put the
Borough Defendants on notice that actual privileged documents may have been mistakenly
produced to Plaintiff.
The disclosure of counsel’s electronic comments, however, does not stand alone. On
December 3, 2010, the Borough Defendants produced their privilege log relevant to their
production of the Borough and Ryan/McKenna documents. That log included entries for 366
pages of privileged Ryan/McKenna documents. Missing, however, were the 872 pages of
unintentionally disclosed documents that are the subject of the instant motion. In other words,
approximately 70% of the material that should have been included on the Ryan/McKenna portion
of the December 3, 2010 log was missing.
25
The Borough Defendants claim that they did not realize that their privilege log was
missing “fewer” entries than should have been included given the sheer volume of their
production and the passage of time from when counsel reviewed the Ryan/McKenna documents
and when the privilege log was generated. In the first instance, the Court notes that the privilege
log is not simply missing “fewer” entries than should have been included; instead, the bulk of the
entries are missing.11 While the volume of production and the passage of several months could
certainly interfere with an attorney’s recollection of exactly how many documents should have
been logged, the Court is somewhat perplexed how counsel missed the fact that approximately
70% of the information that should have been included was not.
The Court is also somewhat perplexed how the simple difference between the number of
pages logged for the Borough documents as compared to the number of pages logged for the
Ryan/McKenna documents did not raise any flags for the Borough Defendants. While the
December 3, 2010 privilege log contains entries for 1389 pages of Borough documents, it only
contains entries for 366 pages of Ryan/McKenna documents. This disparity strikes the Court as
odd in light of the fact that, while larger than the Ryan/McKenna production, the Borough
production consisted of only 2 additional boxes of documents and, more importantly, the
Ryan/McKenna documents were obtained directly from counsel involved in the legal matters
11
Even if the Court were to consider the December 3, 2010 log in its entirety, as it pertains
to both the Borough documents and Ryan/McKenna documents, a significant amount of
information would be missing from same. Indeed, the December 3, 2010 log included entries for
1753 pages of information, 1389 of which pertained to the Borough documents and 366 of which
pertained to the Ryan/McKenna documents. Missing from the log were 1600 pages of privileged
information, 728 pages of privileged Borough documents and the 872 pages of Ryan/McKenna
documents at issue here. Consequently, as produced the December 3, 2010 log was missing
almost half of the information that should have been included on same.
26
underlying this litigation, suggesting that they were equally likely, and, in reality, probably much
more likely to involve privileged communications than the Borough documents. Further, the
Court notes that this disparity in number of pages logged would have been self evident
regardless of the passage of time or the volume of the Borough Defendants’ document
production.
The Court also has questions about how the Borough Defendants’ December 3, 2010
privilege log was generated. The Court understands that the partner who reviewed the
Ryan/McKenna documents was not responsible for drafting the December 3, 2010 privilege log
even as to the portion of the log pertaining to the Ryan/McKenna documents. Instead, associates
were responsible for creating the log in its entirety. The Borough Defendants, however, do not
provide much detail concerning the exact process used by the associates to create the privilege
log. For example, in generating the log entries pertaining to the Ryan/McKenna documents, did
the associates reference the partner’s detailed notes, which were initially used by the clerical
employee to sort the privileged Ryan/McKenna documents from their non-privileged
counterparts? If not, why? It would appear to the Court that those notes would have been
invaluable to the creation of the December 3, 2010 privilege log. Moreover, if the notes were
utilized, how then could counsel possibly not realize that 872 pages of privileged Ryan/McKenna
documents were mistakenly disclosed.
Based on the foregoing, the Court finds the Borough Defendants’ position with respect to
the December 3, 2010 privilege log, namely that nothing in their generation of same indicated
that there was anything wrong with their document production or privilege review, to be
somewhat questionable. Nevertheless, as with the Borough Defendants’ disclosure of their
27
counsel’s electronic comments, the Court is willing to assume, for purposes of this motion, that
nothing in the creation or the content of the December 3, 2010 privilege log, or more specifically
the fact that numerous log entries were missing from same, would have put the Borough
Defendants on notice that something was amiss with their document production and privilege
review. Consequently, the Court shall not find a waiver based on the Borough Defendants’
failure to reexamine the August 12, 2010 production based on their generation of the December
3, 2010 privilege log.
Again, however, this does not represent the last discovery mishap experienced by the
Borough Defendants involving the disclosure of protected information. Instead, in March 2011,
after having been informed by Plaintiff of organizational issues with the manner in which their
documents appeared on the December 10, 2010 disc,12 the Borough Defendants both realized that
(1) Plaintiff was correct, the documents produced on the December 10, 2010 disc were
disorganized; and (2) 728 pages of privileged Borough documents had been inadvertently
disclosed. It is this disclosure of 728 pages of privileged documents that is pertinent here.
Even completely discounting the fact that the December 3, 2010 privilege log was
missing a significant portion of its entries, the Court finds that the combination of the
inadvertently produced attorney electronic comments and 728 pages of privileged Borough
documents should have put the Borough Defendants on notice that something had gone
profoundly awry with their document production and privilege review. Unlike the Borough
12
The Court notes that the December 10, 2010 disc was the disc generated by the Borough
Defendants after being informed that their disc produced on November 4, 2010, which contained
the Borough and Ryan/McKenna documents initially produced on August 12, 2010 albeit
theoretically more efficiently organized and sans attorney electronic comments, was unreadable.
28
Defendants, the Court does not believe that it was reasonable for them to fail to fail to “re-
review” the Ryan/McKenna portion of the December 10, 2010 disc “[b]ecause the partner knew
she had personally reviewed each of the Ryan/McKenna documents for privilege, because that
group of documents was unaffected by the glitch, and because Plaintiff had not advised of any
problems with that group of documents[.]” (Borough Def. Reply at 4).
In reaching this conclusion, the Court notes that the fact that the computer glitch only
affected the portion of the December 10, 2010 disc that contained the Borough documents is
largely irrelevant as the glitch itself did not cause the Borough Defendants’ privileged
information to be disclosed. Instead, the glitch merely caused the Borough Defendants to take a
slightly closer look at their production of the Borough documents. In doing so, the Borough
Defendants realized that, despite their multi-attorney review during which counsel spent
approximately 250 hours reviewing the Borough documents and Ryan-McKenna documents for
responsiveness to Plaintiff’s requests and privilege, 728 pages of privileged Borough documents
had been disclosed.
In addition, the fact that the partner had personally reviewed each of the Ryan/McKenna
documents for privilege whereas “associates had done a significant part” of the review of the
Borough documents does not excuse the Borough Defendants’ failure to recheck the entire
production. First, while associates may have conducted a significant portion of the review of the
Borough documents, the partner also had some involvement with that review and there is no
evidence that the partner was not responsible for reviewing any of the mistakenly disclosed
Borough documents. Second, and more importantly, the disclosure of the 728 pages of
privileged Borough documents was not the first privilege-related error experienced by the
29
Borough Defendants with respect to information produced on the disc containing the Borough
documents and Ryan/McKenna documents. Instead, as previously stated, at the time the
Borough Defendants learned that they had inadvertently produced 728 pages of privileged
Borough documents (first on August 12, 2010, then again on November 4, 2010 and lastly on
December 10, 2010), they had already also discovered that they had mistakenly produced
counsel’s protected electronic comments in the same production; though the electronic comments
had been removed since November 4, 2010.
The Court finds that the problems experienced by the Borough Defendants with their
production of the Borough documents and Ryan/McKenna documents would have spurred a
reasonable person to recheck the entire contents of the December 10, 2010 disc. Their failure to
do so necessitates a finding that the Borough Defendants did not take prompt and reasonable
steps to rectify its error. This is true despite the fact that Plaintiff’s counsel may have violated
their ethical obligations under RPC 4.4(b) by failing to inform the Borough Defendants that
privileged information had been produced. Regardless of whether Plaintiff’s counsel had an
ethical responsibility under RPC 4.4(b) to inform the Borough Defendants that privileged
information had been disclosed, once “a party is on notice of an error in its document production
[that party] should not wait for its adversary to discover its error and then claim protection under
FRE 502(b).” Sensient, 2009 WL 2905474, at *5, n.12. Instead, once a party is on notice that
“something [i]s amiss with its document production and privilege review[,]” that party must
“promptly re-assess its procedures and re-check its production.” Sensient, 2009 WL 2905474, at
*5.
30
The Court finds that how a party receives notice of its error is largely irrelevant: whether
a party is informed by its adversary that privileged information has been inadvertently produced
as in Sensient or whether the circumstances surrounding a party’s production indicates that
something has gone awry, as is the case here, is of little import. Instead, the key is that once a
party has notice that something is “amiss with its production and privilege review[,]” that party
has an obligation to “promptly re-assess its procedures and re-check its production.” Id. at *5.
As explained above, the Borough Defendants, despite being on notice that something was amiss
with their production of the Borough documents and Ryan/McKenna documents, failed to
promptly re-check their production of same.
Finally, the Court finds that under the circumstances of this case, the interests of justice
favor finding that a waiver occurred. While the Court agrees with the Borough Defendants that
the relevance of the mistakenly produced Ryan/McKenna documents is not dispositive of the
waiver issue (Ciba-Geigy, 916 F.Supp. at 414 (finding “[t]hat the documents are relevant and
probative does not determine the waiver questions” (internal quotation marks and citation
omitted)), the Court also finds however, much like in Ciba-Geigy, “the interests of justice would
be served by a finding of waiver, where, as here, a party’s negligence has resulted in the
inadvertent production of a privileged document.” 916 F.Supp. at 414; see also Peterson, 262
F.R.D. at 429 (same) and Sensient, 2009 WL 2905474 at *5, n.12 (same).
In reaching this conclusion, the Court is aware that, here, Plaintiff did not inform the
Borough Defendants that they had produced privileged material. The Court is also aware that the
Borough Defendants claim that, in failing to do so, Plaintiff’s counsel violated their ethical
obligations under RPC 4.4(b).
31
RPC 4.4(b) provides as follows: “[a] lawyer who receives a document and has reasonable
cause to believe that the document was inadvertently sent shall not read the document or, if he or
she has begun to do so, shall stop reading the document, promptly notify the sender, and return
the document to the sender.” Here, as the Borough Defendants note, there were obvious
indications that at least some of the documents at issue in this motion were privileged. Indeed,
from the Court’s in camera review it appears that at least 23 documents consisting of
approximately 60 of the 872 pages of documents at issue in this motion bore a legend indicating
that they were “PRIVILEGED AND CONFIDENTIAL” or “ATTORNEY/CLIENT
CONFIDENTIAL” communications.13 In light of these explicit designations, the Court finds that
Plaintiff should have known that he received privileged information from the Borough
Defendants. As a result, Plaintiff’s counsel had an obligation under RPC 4.4(b) to, at a
minimum, stop using the marked documents and return them to the Borough Defendants.
Plaintiff’s counsel did not do so.
Plaintiff’s counsel never addresses the Borough Defendants’ argument concerning RPC
4.4(b). Indeed, Plaintiff’s counsel does not even attempt to explain why he failed to direct the
Borough Defendants to their disclosure of privileged information. The Court is aware that
Plaintiff in his pending motion to compel claims that several documents over which the Borough
Defendants have made privilege claims, including a number of the Ryan/McKenna documents at
issue here, have been improperly marked/withheld as privileged. As a result, it is possible that
Plaintiff reasonably believed that the Ryan/McKenna documents marked as “PRIVILEGED
13
The legends may not have appeared exactly as depicted above, but were substantially
similar to the two legends provided.
32
AND CONFIDENTIAL” and “ATTORNEY/CLIENT CONFIDENTIAL” had been purposefully
disclosed by the Borough Defendants. However, without a specific argument in this regard, the
Court cannot excuse Plaintiff’s counsel’s failure to notify the Borough Defendants of their
disclosure. Consequently, it would appear that Plaintiff’s counsel violated their duty under RPC
4.4(b). This fact obviously weighs against finding that a waiver occurred. Nevertheless, the
Court on balance finds that this fact does not overcome the Borough Defendants failure to re-
review their entire production after it became clear that there was something amiss with their
document production and privilege review. Sensient, 2009 WL 2905474 at *5.
As described above, at least as of March 2011 the Borough Defendants should have
known that something went awry with their production and review of the documents initially
produced on August 12, 2010. At a minimum, at that point in time the Borough Defendants
knew that the information initially produced on August 12, 2010, which included the Borough
documents and Ryan/McKenna documents, not only contained their attorneys’ electronic
comments, but also 728 pages of privileged Borough documents. As explained above, these
disclosures should have spurred the Borough Defendants to re-review their entire production of
the Borough documents and Ryan/McKenna documents. The Borough Defendants’ failure to do
so was unreasonable. As a result, everything considered, the Court finds that the interests of
justice favor finding that a waiver occurred.
The Court, however, also finds that the interests of justice favor imposing a waiver that is
narrow in scope. While the Court has determined that the Borough Defendants’ disclosure of the
privileged Ryan/McKenna documents was not inadvertent, meaning the Borough Defendants
shall not be permitted to reclaim same, the Court has also determined that the Borough
33
Defendants’ disclosure was unintentional. Clearly, the Borough Defendants in unintentionally
disclosing over 800 pages of otherwise privileged information were not attempting to use the
attorney-client privilege as both a sword and a shield. Indeed, this does not represent a case
where a purposeful, selective disclosure was made. As a result, the Court shall limit the waiver
to the actual documents disclosed and shall not require a broad subject matter waiver. Koch
Materials Co. v. Shore Slurry Seal, Inc., 208 F.R.D. 109, 120-21 (D.N.J. 2002) (noting that
“[t]he general rule that a disclosure waives not only the specific communication but also the
subject matter of it in other communications is not appropriate in the case of inadvertent
disclosure, unless it is obvious that a party is attempting to gain an advantage or make offensive
or unfair use of the disclosure” and finding that full subject matter waiver was not warranted)
(internal quotation marks and citation omitted).
III. Conclusion
For the reasons set forth above, the Borough Defendants’ motion for discovery to reclaim
inadvertently produced privileged documents is DENIED. An appropriate Order follows.
Dated: May 30, 2012
s/Tonianne J. Bongiovanni
HONORABLE TONIANNE J. BONGIOVANNI
UNITED STATES MAGISTRATE JUDGE
34
NOT FOR PUBLICATION

____________________________________
:
JUSTER ACQUISITION CO., LLC, :
:
Plaintiff, : Civil Action No. 12-3427 (JLL)
:
v. :
:
NORTH HUDSON SEWERAGE : OPINION
AUTHORITY, :
:
Defendant. :
____________________________________:
HAMMER, United States Magistrate Judge
I. INTRODUCTION
This matter comes before the Court by way of defendant North Hudson Sewerage
Authority’s (“defendant” or “NHSA”) motion for a protective order regarding the sixty-seven
(67) word searches of electronically stored information (“ESI”) demanded by plaintiff Juster
Acquisition Co., LLC (“plaintiff” or “Juster”). See Nov. 28, 2012, Motion for Discovery Fees,
ECF No. 14. Should the Court deny defendant’s motion for a protective order, NHSA seeks, in
the alternative, an order for fee-shifting on plaintiff’s requested electronic discovery. If the Court
determines that fee-shifting is in fact warranted, plaintiff submits that the Court must also require
defendant to reimburse it for the electronic discovery costs it has incurred in the course of this
litigation. For the reasons set forth herein, the Court denies with prejudice defendant’s motion
for a protective order and request for discovery fees, and as such, plaintiff’s request for discovery
fees is rendered moot.
1
II. FACTUAL BACKGROUND
Juster filed a four-count Complaint on June 6, 2012, against NHSA to recover
compensatory damages, costs and pre-judgment interest, attorneys’ fees and costs, and such other
relief as the Court deems equitable and just. (Compl., June 6, 2012, ECF No. 1). Juster is a New
York limited liability company that invests in various business opportunities such as complex
financing transactions. (Id. ¶ 1, ECF No. 1). NHSA is a political subdivision and public body
politic, formed pursuant to the Sewerage Authorities Law of New Jersey, N.J.S.A. 40:14A-1 et
seq., for the purpose of acquiring, constructing, maintaining, and operating facilities for
collection, treatment, and disposal of sewage. (Id. ¶ 2). The parties began their business
relationship when NHSA sought to refinance and recapitalize the bulk of its primary debt
obligations. (Id. ¶ 6). NHSA determined it would need help from a private party who possessed
the requisite knowledge to execute the transaction, and Juster responded to NHSA’s Request for
Qualifications. (Id.). Juster alleges that the two parties negotiated and executed a term sheet
(“Term Sheet”), which contained a binding Exclusivity Provision stating that NHSA would
“neither solicit other offers nor disclose any of the terms herein for a period of 18 months from
the date of execution of this Term Sheet.” (Id. ¶ 7). NHSA denies the existence of this
Exclusivity Provision, or that the Term Sheet was ever executed. (Answer ¶ 7, July 13, 2012,
ECF No. 7).
Pursuant to the execution of the Term Sheet, Juster alleges that it devoted substantial time
and effort developing and implementing a plan for NHSA to refinance its primary debt
obligations (“Juster Work Product”). (Compl. ¶ 10, ECF No. 1). NHSA, on the other hand,
2
alleges that Juster undertook such action at its own expense in the hope of becoming NHSA’s
financing agent. (Answer ¶ 10, ECF No. 7). Accordingly, NHSA denies that “it ever agreed to
or has any obligation to reimburse Juster.” (Id.). Juster alleges that on January 20, 2012, NHSA
instructed Juster’s financial advisors to put a halt to the transaction. (Compl. ¶ 11, ECF No. 1).
It contends that NHSA indefinitely suspended the closing, and in the months following the
suspension, NHSA refused to negotiate in good faith with Juster. (Id. ¶ 12). On or about May
24, 2012, NHSA took the Juster Work Product and proceeded to close the contemplated
transaction without Juster, in violation of the Exclusivity Provision. (Id. ¶ 15). NHSA denies
these allegations, but admits that it proceeded to closing on May 24, 2012, using a Certificates of
Participation (“COP”) structure after determining that such a transaction structure would better
meet its requirements for the refinancing. (Answer ¶ 15, ECF No. 7). NHSA alleges that Juster
had the opportunity to participate but declined to do so. (Id.). Juster claims that NHSA’s actions
have damaged Juster in an amount not less than $41 million. (Compl. ¶ 16, ECF No. 1).
III. PROCEDURAL HISTORY
As set forth above, plaintiff filed a four-count Complaint against NHSA on June 6, 2012
(ECF No. 1), and defendant filed its Answer on July 13, 2012 (ECF No. 7). The Court ordered
that Fed. R. Civ. P. 26 disclosures were to be exchanged on or before October 12, 2012, with fact
discovery to remain open through May 22, 2013. (Pretr. Sched. Order ¶¶ 1–2, Oct. 5, 2012, ECF
No. 10). The parties served written discovery, and agreed that all responses would be served on
or before November 5, 2012, and any privilege logs would be served on or before November 12,
2012. (Id. ¶ 3). Plaintiff’s counsel wrote to the Court on November 5, 2012, with intentions of
requesting a revised scheduling order. (ECF No. 12). After a telephone status conference, the
3
Court issued an Order extending the deadline for all responses to written discovery to be served
on or before December 10, 2012, and any responsive privilege log to be served on or before
December 17, 2012. (Order, Nov. 13, 2012, ECF No. 13).
Plaintiff issued its First Request for the Production of Documents on August 15, 2012,
which included forty-nine (49) requests for documents related to defendant’s answer and
affirmative defenses and other relevant documents, as well as an additional list of sixty-seven
(67) proposed search terms. (ECF No. 14-1). On November 28, 2012, defendant filed a letter
requesting that the Court enter a protective order regarding the sixty-seven (67) electronic word
searches demanded by Juster. (ECF No. 14). NHSA argues that it is entitled to a protective
order because it has already produced 8000 pages of responsive documents and the additional
search terms are, in its view, quite broad and vague. (Id.). Alternatively, should the Court deny
its request for a protective order and require these searches, NHSA seeks an order that requires
Juster to pay the costs associated with conducting these searches. (Id.).
In response, Juster filed a letter in opposition to defendant’s application for fee-shifting
on electronic discovery. (Pl.’s Letter, Nov. 28, 2012, ECF No. 15). Juster maintains that it seeks
reasonable discovery that is tailored to the breach issues in this case and is limited to the
timeframe of 2011 and 2012. (Id. at 1). Therefore, Juster asserts that NHSA should conduct the
relevant searches, including the searches involving the sixty-seven (67) proposed terms, and bear
the cost of doing so. (Id. at 1). However, if the Court decides to shift the cost of defendant’s
discovery to plaintiff, Juster requests that the Court order NHSA to reimburse Juster for the
electronic discovery costs it has incurred already. (Id. at 10).
4
IV. DISCUSSION
Fed. R. Civ. P. 26(c)(1) provides, in relevant part:
A party or any person from whom discovery is sought may move

for a protective order in the court where the action is pending – or
as an alternative on matters relating to a deposition, in the court for
the district where the deposition will be taken. The motion must
include a certification that the movant has in good faith conferred
or attempted to confer with other affected parties in an effort to
resolve the dispute without court action. The court may, for good
cause, issue an order to protect a party or person from annoyance,
embarrassment, oppression, or undue burden or expense. . . .
Here, defendant moves for a protective order with regard to the additional sixty-seven
(67) word searches requested by plaintiff. However, defendant has failed to provide any law or
analysis in support of its request for a protective order. That is, defendant presents no
compelling factual basis or sufficient legal background to support its motion for a protective
order. It is true, as defendant argues, that several of the sixty-seven terms appear, at first blush,
to be somewhat broad. See, e.g., Nov. 28, 2012, Motion for Discovery Fees, Exh. 1 (list of
proposed search terms, which include “SEC,” “fee,” “debt,” and “Tax!”). However, NHSA
points out that it has complied with Juster’s request to search approximately one hundred terms,
including “indicative,” “development,” “security,” “fee,” and “finance!” Nov. 28, 2012, Letter,
ECF No. 15, Exh. B (exclamation point in original). Moreover, given the nature of the dispute,
the Court cannot find that the search terms are unreasonable. In addition, NHSA fails to show
how it would be unreasonably cumulative or duplicative to perform the requested search
discovery. Therefore, the Court cannot conclude that there is good cause for a protective order.
Further, defendant has failed to comply with the plain language of Fed. R. Civ. P.
26(c)(1), which requires the inclusion of a certification “that the movant has in good faith
5
conferred or attempted to confer with other affected parties in an effort to resolve the dispute
without court action.” In the absence of such a certification from defendant, the Court cannot
grant its request for a protective order. Therefore, because NHSA has failed to carry its burden
of demonstrating good cause, the Court denies defendant’s request for a protective order.
Next, defendant argues that Juster should bear the cost of the requested ESI because the
scope of information sought is unduly burdensome and unreasonable. Defense counsel alleges
that this matter has been one of the most document-intensive cases that the firm has ever
encountered, and that NHSA has already provided Juster with over 8000 pages in hard-copy
documents in response to Juster’s other discovery requests. (Def.’s Letter, ECF No. 14).
Therefore, NHSA submits that the Court should require Juster to pay for its costs in retaining an
outside vendor to provide the technical support needed to respond to Juster’s ESI discovery
requests. (Id.). However, for the reasons set forth below, the Court denies defendant’s motion
for discovery fees.
There is a general presumption that “the responding party must bear the expense of
complying with discovery requests.” Zubulake v. UBS Warburg LLC, 216 F.R.D. 280, 283
(S.D.N.Y. 2003). Therefore, the responding party “has the burden of proof on a motion for cost-
shifting.” Id. To that end, cost-shifting should only be considered when electronic discovery
imposes an undue burden or expense on the responding party. Id. at 287. As a preliminary
matter, “[c]ost-shifting is potentially appropriate only when inaccessible data is sought.” Id. at
284; see Zubulake v. UBS Warburg LLC, 217 F.R.D. 309, 318 (S.D.N.Y. 2003) (Zubulake II)
(“[W]hether production of documents is unduly burdensome or expensive turns primarily on
whether it is kept in an accessible or inaccessible format (a distinction that corresponds closely to
6
the expense of production).”). “[I]n the world of electronic data, thanks to search engines, any
data that is retained in a machine readable format is typically accessible.” Zubulake II, 217
F.R.D. at 318. Specifically, active, online data, near-line data, and offline storage/archives are
typically identified as “accessible” electronic data. Id. at 318–19. Backup tapes and erased,
fragmented, or damaged data are typically identified as “inaccessible” electronic data. Id. at
319–320. Thus, electronic data that is stored in “a readily usable format” is deemed “accessible”
whereas electronic data that is not readily usable (i.e., data that must be restored, de-fragmented,
or reconstructed) is considered “inaccessible.” Id. at 321.
Here, NHSA, as the responding party, has failed to satisfy its burden of showing that the
ESI sought by Juster is inaccessible. NHSA has not asserted that any of the requested data is
located on backup tapes. (Def.’s Mot., ECF No. 14); (see Pl.’s Opp’n Mem. at 6, ECF No. 15);
cf. Universal Del., Inc., v. Comdata Corp., 2010 U.S. Dist. LEXIS 32158, at *22 (E.D. Pa. Mar.
31, 2010) (finding that “[s]ome cost-shifting is appropriate because the data at issue in this case
is kept in an inaccessible format, i.e., back-up tapes”). It has not asserted that any of the
requested data is erased, fragmented, or damaged in any way. (See ECF No. 14). To the
contrary, by asserting that it has hired an outside vendor to perform the word searches, NHSA
has acknowledged that the ESI is accessible. It has also failed to show that the ESI sought by
Juster imposes an “undue” burden or expense. Rather, the Court finds that NHSA seeks merely
to avoid the cost associated with what it presumes to be duplicative and expensive word searches.
As a result, the Court cannot find that the ESI requested by Juster falls into either category of
“inaccessible” electronic data. Because such data is in fact accessible to NHSA, defendant must
bear the attendant discovery costs. Even though defendant has failed its threshold requirement of
7
demonstrating that Juster seeks “inaccessible” ESI, this Court will nonetheless provide additional
reasons for denying NHSA’s fee-shifting request.
The Zubulake Court set forth a seven-factor test to determine whether discovery costs
should be shifted, which are weighted more-or-less in the following order:
1) the extent to which the request is specifically tailored to

discover relevant information;
2) the availability of such information from other sources;
3) the total cost of production, compared to the amount in
controversy;
4) the total cost of production;
5) the relative ability of each party to control costs and its incentive
to do so;
6) the importance of the issues at stake in the litigation; and
7) the relative benefits to the parties of obtaining the information.
216 F.R.D. at 284.
The Third Circuit has adopted that criteria for when cost-shifting might be appropriate in
electronic discovery. See Wachtel v. Guardian Life Ins., 2007 U.S. Dist. LEXIS 43842 (D.N.J.
June 18, 2007) (following Zubulake); see also Boeynaems v. La Fitness Int’l, 285 F.R.D. 331,
336 (E.D. Pa. 2012) (stating that Zubulake is undoubtedly the leading opinion in regard to
standards for requiring that the requesting party share discovery costs).
Applying the Zubulake factors in light of all of the circumstances, the Court does not find
that fee-shifting is warranted. First, the Court agrees with Juster that the ESI requests are
sufficiently tailored to discover relevant information regarding communications between the
various actors who took part in the transaction at issue. (Pl.’s Opp’n Mem. at 7, ECF No. 15).
The requested searches are reasonable and restricted to the time period of 2011 to 2012, which is
the relevant time frame during which the parties were in negotiation. Further, it is important to
note that after Juster provided NHSA with the proposed discovery request, NHSA agreed in
8
writing that each party would bear its own costs in connection with producing discovery. (Id. at
2; ECF No. 15, Exh. A ¶ 4). Accordingly, the first factor is in favor of Juster.
Second, until NHSA actually runs the requested searches, neither NHSA nor anybody
else can know whether the ESI word searches will turn up information that would have been
available from any other source. While it is true that the Court, on motion or on its own, must
limit the frequency or extent of discovery otherwise allowed if it determines that “the discovery
sought is unreasonably cumulative or duplicative, or can be obtained from some other source that
is more convenient, less burdensome, or less expensive,” Fed. R. Civ. P. 26(b)(2), NHSA fails to
show how it would be unreasonably cumulative or duplicative to perform the requested ESI
discovery. As such, the Court is not compelled to impose such a limitation on plaintiff’s
requested ESI discovery. Further, it is irrelevant that NHSA has already turned over 8000 pages
of documents to Juster because “production of information in ‘hard copy’ documentary form
does not preclude a party from receiving that same information in computerized/electronic form.”
Anti-Monopoly, Inc. v. Hasbro, Inc., 1995 U.S. Dist. LEXIS 16355, at *2 (S.D.N.Y. Nov. 3,
1995). Therefore, the second Zubulake factor also weighs in favor of Juster.
Factors three through five concern the financial aspect of the discovery request. The
amount in controversy here is well in excess of the amount that NHSA estimates it will cost to
perform the requested ESI discovery. NHSA alleges that it will cost between $6000 to $16,000
to process the word searches and eliminate duplicates. (Def.’s Letter, ECF No. 14). The amount
that Juster alleges it is owed is not less than $41 million. (Answer ¶ 15, ECF No. 7). Further,
Juster alleges that “NHSA’s resources dwarf the estimated cost of the requested discovery . . . .
The estimated discovery costs at issue here are negligible in comparison to the NHSA’s
9
resources.” (Pl.’s Opp’n Letter at 9, ECF No. 15). Given the amount of damages at stake in this
litigation, and NHSA’s ability to absorb the cost of the requested ESI discovery, the projected
discovery costs are not so substantial as to support a conclusion that fee-shifting is warranted.
See, e.g., Zubulake, 216 F.R.D. at 287–88 (“[T]he total cost of restoring the remaining seventy-
two tapes extrapolates to $165,954.67 . . . the cost of restoration is surely not ‘significantly
disproportionate’ to the projected value of this case. This factor weighs against cost-shifting.”).
As a result, Zubulake factors three through five weigh in favor of Juster.
Factors six and seven are less critical to the analysis of whether cost-shifting is
appropriate in electronic discovery. See Zubulake, 216 F.R.D. at 284 (listing the factors for
determination of fee-shifting in descending order of importance). As such, factors six and seven,
standing alone, are generally insufficient to support a party’s motion for fee-shifting. Here,
factors six and seven do not weigh in favor of plaintiff, nor do they necessarily weigh in favor of
defendant. The sixth factor examines the actual allegations involved in the litigation. It is
usually invoked where the litigation presents “a particularly novel issue.” Zubulake, 216 F.R.D.
at 289. The litigation here is a private contractual dispute between fairly sophisticated parties.
The issues at stake include the nature of the parties’ negotiations and what one party could have
reasonably understood the other party to have meant during their discussions. The outcome of
this litigation does not have great potential for broad public impact because it does not implicate
any widely-applicable public policy. See Zubulake II, 217 F.R.D. at 321 (noting that cases
involving “toxic tort class actions, environmental actions, so-called ‘impact’ or social reform
litigation, cases involving criminal conduct, or cases implicating important legal or constitutional
questions” are of the ilk that may have broad public impact). Therefore, the sixth factor is not
10
dispositive to the instant matter. See id. at 321 (noting that the sixth factor “will only rarely
come into play”). The seventh factor concerns the potential benefits to either party that may
result from the ESI discovery. However, any such benefits cannot be known until the word
searches are performed. The last factor is thus also neutral.
Having established that the Zubulake factors, on balance, weigh in favor of plaintiff, this
Court finds that principles of fundamental fairness further support its denial of defendant’s fee-
shifting request. See Boeynaems, 285 F.R.D. at 335. Here, both parties have requested ESI
discovery from the other party, and both parties agreed to pay their own costs in producing
discovery. (See Exh. A to ECF No. 15). The amount of discovery requested by each party is
comparable – Juster requested sixty-seven (67) word searches in addition to forty-nine (49)
specific documents, and NHSA requested the results from one hundred (100) search terms. This
is not a situation where one party is unduly burdened by the opposing party’s discovery requests,
which may implicate fee-shifting. See Major Tours, Inc. v. Colorel, 2009 U.S. Dist. LEXIS
97554, at *11, 19 (determining that plaintiffs and defendants should share the cost of searching
defendants’ backup tapes, given that thousands of relevant documents were e-mails that had been
produced already to plaintiffs). Therefore, because defendant has failed to demonstrate that it has
been unduly burdened by plaintiff’s ESI request or that plaintiff’s request is eminently
unreasonable, it fails to meet its burden of showing why cost-shifting is appropriate.
Because the Court finds that defendant is not entitled to a fee-shifting order, plaintiff’s
subsequent request that NHSA should reimburse it for the electronic discovery it has incurred
during the course of this litigation is rendered moot. Therefore, the Court declines to analyze the
merits of plaintiff’s request for fee-shifting.
11
V. CONCLUSION
For the reasons above, the Court denies defendant’s motion for a protective order, as well
as its alternative request for fee-shifting. The Court also denies as moot plaintiff’s request for
fee-shifting. An Order consistent with this Opinion will follow.
s/ Michael A. Hammer
United States Magistrate Judge
Date: February 11, 2013
12
Dictum
The newsletter of the NJSBA Young Lawyers Division
Vol. 37, No. 1 — March 2013
Go to
New Jersey State Bar Association Dictum 1 Index
Recent Developments in New Jersey Law:
Social Media and More
by Fernando M. Pinguelo, Esq. and Chris Borchert
F
rom MySpace and Facebook to Twitter and have reasonably forecasted a substantial disruption. The
LinkedIn, online social networking platforms court also found that the speech did not make its way on
have radically changed the way we share, store campus sufficiently to classify it as ‘school speech.’
and consume information. Today, users can tap social
media outlets for everything from big data storage to Social Media and Juror Bias
daily deals and discounts; and, with recent advances in In U.S. v. Fumo, 2 the Third Circuit considered
phone and tablet technology, accessing online content whether the district court had erred in its refusal to
remotely has never been easier. grant the defendant a new trial on the grounds of a
As a result, state legislatures and courts are begin- jury’s exposure to extraneous information, and the
ning to take notice of the far-reaching implications of purported prejudice and partiality that may have
social and other cyber-related media. In New Jersey resulted. The defendant claimed that a juror who posted
jurisdictions, key cases and statutes have emerged that comments relating to the case on his Facebook and
address these broad legal implications. In fact, in 2011, Twitter accounts brought widespread public attention to
New Jersey enacted the nation’s most comprehensive the jury’s deliberations, creating a “cloud of intense and
anti-cyber bullying law—the Anti-Bullying Bill of Rights widespread media coverage...and [the] public expecta-
Act. It is no secret that today’s social media platforms tion that a verdict [wa]s imminent[,]” thereby violating
are characterized by wide-ranging functionality, and so his Sixth Amendment right to a fair and impartial trial.
lawyers of all practices must stay current on the laws Takeaway: A trial court’s jury instructions before
and cases relating to social and cyber media. What and at the close of a trial should include instructions not to
follows are a few notable cases that caught our attention. communicate information to anyone by any means, including
social media, about the case. The Third Circuit found no
Social Media and School Speech plausible theory for how the defendant suffered any prejudice,
In J.S. ex rel. Snyder v. Blue Mountain School District,1 let alone substantial prejudice, from the juror’s Facebook
a student created a parody MySpace profile of her and Twitter comments.
principal, which contained adult language and sexually
explicit content. Although the profile was created at the Social Media and Shield Law Protections
student’s home on a computer owned by her parents, the In Too Much Media v. Hale, 3 the defendant, a
school nevertheless suspended her for 10 days for twice self-described journalist, posted Internet messages
violating the school’s disciplinary code (false accusation about Too Much Media, LLC (TMM), a company that
about a school staff member and a copyright violation produced software used in the adult entertainment
of its computer use policy for misappropriating the industry. Specifically, the defendant alleged that a
principal’s photograph from the school website). The breach in TMM’s software exposed the personal infor-
student’s parents filed a Section 1983 action against the mation of TMM customers who believed they were
school district, claiming violation of free speech rights, accessing pornographic websites anonymously. Claim-
due process rights, and state law. ing she had conducted a probe on the alleged breach,
Takeaway: Applying the Tinker standard, the Third the defendant posted that TMM violated New Jersey’s
Circuit found that the MySpace page did not substantially identity theft protection laws, threatened people who
disrupt the operation of the school, nor could school officials questioned its conduct, and profited from the alleged
Go to
breach. TMM sued, alleging the defendant’s posts were Cloud Computing and Legal Ethics
defamatory and made in a false light. When TMM While cloud computing (i.e., the delivery of comput-
sought to depose her during discovery to ascertain her ing and storage capacity as a service to users over the
alleged sources, the defendant moved for a protective Internet) can offer significant benefits to lawyers in
order, asserting she was a reporter entitled to protection managing their practices, particularly solo and small
under New Jersey’s Shield Law. firm practitioners for whom flexibility and cost control
Takeaway: While New Jersey’s Shield Law affords are a must, attorneys must take careful steps before
broad protections to news media, including non-traditional employing Internet or cloud-based software programs
news outlets such as blogs, a self-appointed newsperson is or using remote sites to store client-related information.
not necessarily a reporter entitled to Shield Law protec- In Advisory Committee on Professional Ethics Opinion
tions. Those seeking to invoke the privilege must demonstrate 701 (April 2006), the New Jersey Advisory Committee
that the means by which they disseminate ‘news’ is similar to on Professional Ethics addressed the ethical implications
traditional news sources, such as newspapers, magazines, etc. associated with cloud-based data storage.
Takeaway: The ethical issues underlying the use of cloud
Social Media and Employment services are the duty of confidentiality owed to clients and
In Pietrylo v. Hillstone Restaurant Group,4 the plain- the duty to serve competently. Attorneys must take reason-
tiffs, who were employed as servers at a restaurant able, affirmative steps to ensure the confidentiality of client
owned by the defendant, created a MySpace group to information that travels over the Internet or other network.
“vent about any BS we deal with at work without any This obligation includes assuring to a reasonable extent that
eyes spying in on us.” The group was created as a private a third-party cloud services provider is aware of the lawyer’s
group, and invitations were required to join. Central to obligation of confidentiality, and of its own obligation, wheth-
the case is the fact that an invited employee, who was er by contract, professional standards, or otherwise, to assist
an authorized user of the group, accessed the page for in preserving confidentiality.
one of the restaurant managers and later provided the
password to another manager. When managers learned Fernando M. Pinguelo, a partner of Norris McLaughlin
that comments disparaged the restaurant and its & Marcus, P.A. and chair of its cyber security and data
managers and customers, they terminated the plaintiffs’ protection group, is a trial lawyer who devotes his practice
employment. The plaintiffs filed suit, alleging violations to complex business disputes with an emphasis on cyber,
of federal and state stored communications statutes, media, and employment matters in federal and state courts.
wrongful termination, and invasion of privacy. Chris Borchert is a second-year student at the University of
Takeaway: Content found on blogs and public social Connecticut School of Law, where he is pursuing a certificate
networking profiles that employees generate may still enjoy in intellectual property in addition to his J.D. As a summer
the benefit of protected privacy when restrictions on access associate with the firm, he focused primarily on matters
are implemented. Employers should exercise caution when relating to e-discovery, cyber security and social media. This
asking for access to private employee information. article was first distributed at a New Jersey Institute for
Continuing Legal Education seminar.
Endnotes
1. J.S. ex rel. Snyder v. Blue Mountain School Dist., 650 F. 3d 915 (3rd Cir. 2011).
2. U.S. v. Fumo, 655 F. 3d 288 (3rd Cir. 2011).
3. Too Much Media v. Shellee Hale, 206 N.J. 209 (2011).
4. Pietrylo v. Hillstone Restaurant Group (D.N.J. 2008).
Go to
Romano v Steelcase Inc. :: September, 2010 :: New York Other Courts Decisions :: New ... Page 1 of 6
Justia.com Lawyer Directory Legal Answers Law more ? Sign In
in ---- 2010 Other Courts
Search
Romano v Steelcase Inc. Justia on
Like 129,092 people like this. Sign Up to see what

More
Share | your friends like.
Romano v Steelcase Inc. 2010 NY Slip Op 20388 [30 Misc 3d 426] September 21, 2010
Daily Opinion Summaries
Spinner, J. Supreme Court, Suffolk County Published by New York State Law Reporting
Subscribe to Justia's FREE Daily
Bureau pursuant to Judiciary Law § 431. As corrected through Wednesday, February
Newsletter Opinion Summaries
23, 2011
Subscribe Now
[*1] Kathleen Romano, Plaintiff, Today on Verdict

v What’s the Matter with North Dakota
and Arkansas? Two State Legislatures
Steelcase Inc. et al., Defendants.
Pass Highly Restrictive and
Unconstitutional Abortion Laws
Supreme Court, Suffolk County, September 21, 2010
Justia columnist and Hofstra law
APPEARANCES OF COUNSEL professor Joanna Grossman
comments on two states’
Gallagher Gosseen Faller & Crowley, Garden City, for Steelcase Inc., defendant. John T. decisions to pass abortion laws
Ryan & Associates, Riverhead, for Educational & Institutional Cooperative Services Inc., despite the fact that under U.S.
Supreme Court precedent, it is very clear
defendant. Orrick Herrington & Sutcliffe LLP, New York City, for Facebook. Kelner & that these new laws are unconstitutional.
Kelner, New York City, for plaintiff.
By Joanna L. Grossman
{**30 Misc 3d at 427} OPINION OF THE COURT
Jeffrey Arlen Spinner, J. Ask a Lawyer

Question:
Ordered, that defendant Steelcase's motion is hereby granted as set forth herein below.
Defendant Steelcase moves this court for an order granting said defendant access to
plaintiff's current and historical Facebook and MySpace pages and accounts, including
all deleted pages and related information upon the grounds that plaintiff has placed Ask Question
certain information on these social networking sites which is believed to be inconsistent
About Legal Answers
with her claims in this action concerning the extent and nature of her injuries,
especially her claims for loss of enjoyment of life. Connect with Justia
Follow justiacom Fan Justia
The present application was brought on by order to show cause. The court has reviewed Justia
the submissions both in favor of and in opposition to the relief sought, as well as the Like
applicable federal statutory law, specifically the Stored [*2]Communications Act (18
129,092 people like Justia.
USC § 2701 et seq.), which prohibits an entity such as Facebook and MySpace from
disclosing such information without the consent of the owner of the account (see 18
USC § 2702 [b] [3]; Flagg v City of Detroit, 252 FRD 346, 352 [ED Mich 2008]).
Scope of Permissible Discovery Facebook social plugin
Find a Lawyer
Pursuant to CPLR 3101, there shall be full disclosure of all nonprivileged matter which is
material and necessary to the defense or prosecution of an action. To this end, trial
Search
courts have broad discretion in the supervision of discovery, and in determining what is
Browse Lawyers
"material and necessary" (see Allen v Crowell-Collier Publ. Co., 21 NY2d 403 [1968];
Andon v 302 -304 Mott St. Assoc., 94 NY2d 740 [2000]; Cabellero v City of New York, Lawyers - Get Listed Now!
Get a free full directory profile listing
48 AD3d 727 [2d Dept 2008]). Within the context of discovery, "necessary" has been
interpreted as meaning " 'needful' and not indispensable" (see Allen at 407). The
"material and necessary" standard is to be interpreted liberally, requiring disclosure of
"any facts bearing on the controversy which will assist preparation for trial by
sharpening the issues and reducing delay and prolixity. The test is one of usefulness
and reason" (see Allen, 21 NY2d at 406; Andon, supra; Hoenig v Westphal, 52 NY2d
605, 608 [1981] [pretrial discovery is to be encouraged, limited only by a test for
materiality of "usefulness and reason"]).{**30 Misc 3d at 428}
Each discovery request is to be decided on a case-by-case basis, keeping in mind the
http://law.justia.com/cases/new-york/other-courts/2010/2010-20388.html 4/2/2013
strong public policy in favor of open disclosure (see Andon

sought is sufficiently related to the issues in litigation so as to m
it in preparation for trial reasonable, then discovery should be
406 -407; Matter of Beryl, 118 AD2d 705 [2d Dept 1986]). It
information sought may not be admissible at trial as "pretria
only to proof that is admissible but also to matters that may l
admissible proof" (see Twenty Four Hour Fuel Oil Corp. v Hunte
175, 175 [1st Dept 1996]; Polygram Holding, Inc. v Cafaro,
Dept 2007] [" 'disclosure extends not only to admissible proof
documents which may lead to the disclosure of admissible p
which might be used in cross -examination"]).
Information Sought from Internet Sites
Plaintiffs who place their physical condition in

disclosure material which is necessary to the
Westphal, supra). Accordingly, in an action seeking damage
discovery is generally permitted with respect to materials that
the issue of damages and the extent of a plaintiff's injury (se
York, 205 AD2d 755 [2d Dept 1994]), including a plaintiff's cla
of life (see Orlando v Richmond Precast, Inc., 53 AD3d 534
action to recover damages for personal injuries, records so
necessary to the defense regarding plaintiff's claim of loss of en
v City of New York, 276 AD2d 789 [2d Dept 2000]; Mora v
Med. Ctr. of N.Y., 8 Misc 3d 868 [Sup Ct, NY County 2005]).
Thus, in Sgambelluri v Recinos (192 Misc 2d 777 [Sup Ct, Na

action arising out of a motor vehicle accident, the court held
video taken two years after the incident was clearly
permanency of injuries. As a result of the accident,
permanent injuries to her neck and back, and testified at her de
longer participate in certain activities such as running or hors
sought a copy of her wedding video on the basis that it might
various activities such as dancing, which would be relevant
objected on{**30 Misc 3d at 429} the basis of the personal n
court decided in favor of disclosure, noting its relevancy to the
injuries. In so finding, the court reasoned that although the
tape, as contemplated by CPLR 3101 (i), the statute's "langua
encompass any film, photograph or videotape . . . involving
paragraph one of subdivision (a), i.e., a party. This is consisten
of New York courts, allowing liberal disclosure. Moreover,
subdivision (i) only strengthens the argument for open disclo
[internal quotation marks omitted].)
Like the plaintiff in Sgambelluri, plaintiff herein also claims sh

injuries as a result of the incident and that she can no long
activities or that these injuries have affected her enjoyment of
to plaintiff's claims, Steelcase contends that a review of the pub
MySpace and Facebook pages reveals that she has an
to Florida and Pennsylvania during the time
prohibited such activity. In light of this, defendant sought to
deposition regarding her MySpace and Facebook accounts, to
those depositions, served plaintiff with a notice for discovery an
inter alia, "authorizations to obtain full access to and copies
historical records/information on her Facebook and MySpace
refused to provide the requested authorizations.
Both Facebook and MySpace are social networking sites w

information about their personal lives, including posting ph
information about what they are doing or thinking. Indeed, Fac
"it helps you share information with your friends and people
"Facebook is about sharing information with others."[FN1]
"social networking service that allows Members to create unique
in order to find and communicate with old and new friends" an
"online community" where "you can share photos, journals
growing network{**30 Misc 3d at 430} of mutual
lifestyle portal that reaches millions of people around the world."[FN3] Both sites allow growing network{**30 Misc 3d at 430} of mutual
the user to set privacy levels to control with whom they share their information.
The information sought by defendant regarding plaintiff's Facebook and MySpace

accounts is both material and necessary to the defense of this action and/or could lead
to admissible evidence. In this regard, it appears that plaintiff's public profile page on
Facebook shows her smiling happily in a photograph outside the confines of her home
despite her claim that she has sustained permanent injuries and is largely confined to
her house and bed. In light of the fact that the public portions of plaintiff's social
networking sites contain material that is contrary to her claims and deposition
testimony, there is a reasonable likelihood that the private portions of her sites may
contain further evidence such as information with regard to her activities and
enjoyment of life, all of which are material and relevant to the defense of this action.
Preventing defendant from accessing plaintiff's private postings on Facebook and
MySpace would be in [*3]direct contravention to the liberal disclosure policy in New
York State.
Although there is no New York case law directly addressing the issues raised by this
application, there are instructive cases from other jurisdictions. Recently, in Ledbetter v
Wal-Mart Stores, Inc. (2009 WL 1067018, 2009 US Dist LEXIS 126859 [Civ Action No.
06-cv -01958 -WYD -MJW, D Colo, Apr. 21, 2009]), defendant store sought, via
subpoena, production of the content of plaintiffs' social networking sites.[FN4]
Information contained on the public access areas contradicted plaintiffs' allegations
regarding the effect of their injuries on their daily lives. When the networking sites
refused to provide the information absent plaintiffs' consent or request, defendant
moved to compel production and plaintiffs moved for a protective order. Both plaintiffs
had claimed physical and psychological injuries as a result of the accident which gave
rise to lawsuit. By order dated April 21, 2009, Magistrate Judge Watanabe denied
plaintiffs' motion and held that the information sought by the subpoenas{**30 Misc 3d
at 431} was "reasonably calculated to lead to the discovery of admissible evidence as is
relevant to the issues in the case." (2009 WL 1067018, *2, 2009 US Dist LEXIS
126859, *5.)
Likewise, in Leduc v Roman (2009 CarswellOnt 843 [Feb. 20, 2009]), a matter pending
in the Superior Court of Justice, Ontario, Canada, defendant also requested production
of the plaintiff's Facebook pages, including private pages. Plaintiff claimed that as a
result of injuries allegedly sustained in a car accident, his enjoyment for life had
lessened. Canadian law requires that each party disclose every document relating to
any matter in the action over which he has possession or control absent a claim of
privilege. Plaintiff had failed to disclose the information, which defendant only learned
about following a defense psychiatric examination. After only being able to access the
limited portions of plaintiff's public profile page, defendant sought an order requiring
production of all site materials as well as preservation of the materials. The decision
denying the request was reversed on appeal, with the appellate court disagreeing that
defendant was on a fishing expedition. In this regard, Judge Brown noted that it was
"beyond controversy" that a person's Facebook pages may contain relevant documents
(¶ 23); that other Canadian cases had permitted into evidence photographs posted on a
person's Facebook page showing them engaged in activities despite their claim to the
contrary; and that it is reasonable to infer from the social networking purpose of
Facebook that even if a person only maintains a private profile, with the public profile
merely listing their name, relevant information exists on their limited -access private
pages (¶ 36). In deciding to permit the examination into the private Facebook profile,
the court set forth: "To permit a party claiming very substantial damages for loss of
enjoyment of life to hide behind self-set privacy controls on a website, the primary
purpose of which is to enable people to share information about how they lead their
social lives, risks depriving the opposite party of access to material that may be
relevant to ensuring a fair trial."(¶ 35; see also Kent v Laverdiere, 2009 CarswellOnt
1986 [Ont Sup Ct, Apr. 14, 2009] [as plaintiff asserted that accident disfigured her and
lessened her enjoyment of life, any photos on Facebook or [*4]MySpace showing her in
healthy state, enjoying life, would be relevant]; Bishop v Minichiello, 2009 BCSC 358
[Apr. 7, 2009] [defendant's motion for production of{**30 Misc 3d at 432} plaintiff's
computer's hard drive so it could analyze how much time plaintiff spent on Facebook
granted as the information sought was relevant to the issues in the case]; Goodridge v
King, 2007 CarswellOnt 7637 [Ont Sup Ct, Oct. 30, 2007] [in action in which plaintiff
claimed various injuries, including loss of enjoyment of life and disfigurement following
a car accident, photos posted by plaintiff on her Facebook account were evidence to the
contrary, showing her socializing and dating]; Kourtesis v Joris, 2007 CarswellOnt 5962
[Ont Sup Ct, Sept. 24, 2007] [in proceeding concerning costs, court noted that during
trial, Facebook photos of plaintiff were important element of case; apparently plaintiff
testified that she no longer had a social life because of her injuries, yet the photographs
taken after the accident showed her at a party].)[FN5]
Thus, it is reasonable to infer from the limited postings on plaintiff's public Facebook
and MySpace profile pages that her private pages may contain material and information
that are relevant to her claims or that may lead to the disclosure of admissible
evidence. To deny defendant an opportunity to access these sites not only would go
against the liberal discovery policies of New York favoring pretrial disclosure, but would
condone plaintiff's attempt to hide relevant information behind self-regulated privacy
settings.
Plaintiff's Privacy Concerns
Production of plaintiff's entries on her Facebook and MySpace accounts would not be
violative of her right to privacy,[FN6] and any such concerns are outweighed by
defendant's need for the information.
The Fourth Amendment's right to privacy protects people, not places (see Katz v United
States, 389 US 347, 351 [1967] ["What a person knowingly exposes to the public, even
in his own home or office, is not a subject of Fourth Amendment protection"]). In
determining whether a right to privacy exists via the Fourth Amendment, courts apply
the reasonableness{**30 Misc 3d at 433} standard set forth in the concurring opinion
of Justice Harlan in Katz: "first that a person have exhibited an actual (subjective)
expectation of privacy and, second, that the expectation be one that society is prepared
to recognize as reasonable." (Id. at 361 [internal quotation marks omitted].)
New York courts have yet to address whether there exists a right to privacy regarding
what one posts on their on -line social networking pages such as Facebook and
MySpace. However, [*5]whether one has a reasonable expectation of privacy in
Internet postings or e-mails that have reached their recipients has been addressed by
the Second Circuit, which has held that individuals may not enjoy such an expectation
of privacy (see United States v Lifshitz, 369 F3d 173, 190 [2004]): " 'Users would
logically lack a legitimate expectation of privacy in the materials intended for
publication or public posting. They would lose a legitimate expectation of privacy in an
e-mail that had already reached its recipient; at this moment, the e-mailer would be
analogous to a letter-writer, whose expectation of privacy ordinarily terminates upon
delivery of the letter.' " (Quoting Guest v Leis, 255 F3d 325, 333 [6th Cir 2001].)
Likewise, whether one has a reasonable expectation of privacy in e -mails and other
writings that have been shared with others, including entries on Facebook and
MySpace, has been addressed by the United States District Court of New Jersey, which
ordered such entries produced in Beye v Horizon Blue Cross Blue Shield of N.J. (Civ
Case No. 06 -5337 [D NJ, Dec. 14, 2007]). In this regard, the court stated that "[t]he
privacy concerns are far less where the beneficiary herself chose to disclose the
information." As to the entries which had not been shared with others, they were to be
preserved. At issue in Beye were on-line journals and diary entries of minor children
who had been denied health care benefits for their eating disorders (see also Moreno v
Hanford Sentinel, Inc., 172 Cal App 4th 1125, 91 Cal Rptr 3d 858 [Ct App, 5th Dist
2009] [no person would have reasonable expectation of privacy where that person took
affirmative act of posting own writing on MySpace, making it available to anyone with a
computer and opening it up to the public eye]; Dexter v Dexter, 2007 WL 1532084,
2007 Ohio App LEXIS 2388 [Ct App, 11th Dist, Portage County 2007] [no reasonable
expectation of privacy regarding MySpace writings open to public view]).{**30 Misc 3d
at 434}
Indeed, as neither Facebook nor MySpace guarantee complete privacy, plaintiff has no
legitimate reasonable expectation of privacy. In this regard, MySpace warns users not
to forget that their profiles and MySpace forums are public spaces,[FN7] and
Facebook's privacy policy set forth, inter alia, that "[y]ou post User Content . . . on the
Site at your own risk. Although we allow you to set privacy options that limit access to
your pages, please be aware that no security measures are perfect or impenetrable."
Further that "[w]hen you use Facebook, certain information you post or share with third
parties (e.g., a friend or someone in your network), such as personal information,
comments, messages, photos, videos . . . may be shared with others in accordance
with the privacy settings you select. All such sharing of information is done at your own
risk. Please keep in mind that if [*6]you disclose personal information in your profile or
when posting comments, messages, photos, videos, Marketplace listing or other items,
this information may become publicly available."[FN8]
Thus, when plaintiff created her Facebook and MySpace accounts, she consented to the
fact that her personal information would be shared with others, notwithstanding her
privacy settings. Indeed, that is the very nature and purpose of these social networking
sites, else they would cease to exist. Since plaintiff knew that her information may
become publicly available, she cannot now claim that she had a reasonable expectation
of privacy. As recently set forth by commentators regarding privacy and social
networking sites: given the millions of users, "[i]n this environment, privacy is no
longer grounded in reasonable expectations, but rather in some theoretical protocol
better known as wishful thinking."[FN9]
Further, defendant's need for access to the information outweighs any privacy concerns
that may be voiced by plaintiff. Defendant has attempted to obtain the sought -after
information{**30 Misc 3d at 435} via other means: e.g., via deposition and notice for
discovery; however, these have proven to be inadequate since counsel has thwarted
defendant's attempt to question plaintiff in this regard or to obtain authorizations from
plaintiff for the release of this information. The materials, including photographs,
contained on these sites may be relevant to the issue of damages and may disprove
plaintiff's claims. Without access to these sites, defendant will be at a distinct
disadvantage in defending this action.
Ordered, that defendant Steelcase's motion for an order granting said defendant access
to plaintiff's current and historical Facebook and MySpace pages and accounts,
including all deleted pages and related information, is hereby granted in all respects;
and it is further ordered, that, within 30 days from the date of service of a copy of this
order, as directed herein below, plaintiff shall deliver to counsel for defendant Steelcase
a properly executed consent and authorization as may be required by the operators of
Facebook and MySpace, permitting said defendant to gain access to plaintiff's Facebook
and MySpace records, including any records previously deleted or archived by said
operators. Footnotes
Footnote 1: Facebook Principles, http:www.facebook.com/policy.php (last visited Apr.

3, 2009).
Footnote 2: About Us, MySpace.com/index.dfm?fuseaction=misc.aboutus (last visited

June 16, 2009).
Footnote 3: MySpace Safety Highlights, http://www.myspace.com/index.cfm?

frseaction=cms.viewpage & placement=safety (last visited June 18, 2009).
Footnote 4: Facebook, MySpace and Meetup.com.
Footnote 5: See Charles Foster, Uncovering the Truth: Social Networks are a Treasure
Trove of Information, Claims Canada, Oct./Nov. 2008,
http://www.claimscanada.ca/issues/article.aspx?aid=1000224989 (last viewed June 18,
2009).
Footnote 6: In New York, there is no common-law right to privacy. (See Ava v NYP
Holdings, Inc., 20 Misc 3d 1108[A], 2008 NY Slip Op 51281[U] [Sup Ct, NY County
2008].)
Footnote 7: MySpace General Tips, http://www.myspace.com/index.cfm?

frseaction=cms.veiwpage & placement=safety_pagetips (last visited June 18, 2009).
Footnote 8: Facebook Principles, effective as of November 26, 2008,

http://www.facebook.com/policy.php (last viewed June 18, 2009).
Footnote 9: Dana L. Fleming and Joseph M. Herlihy, What Happens When the College
Rumor Mill Goes Online?, Privacy, Defamation and Online Social Networking Sites, 53
Boston BJ [No. 1] 16 (Jan./Feb. 2009).
Copyright © Justia :: Company :: Terms of Service :: Privacy Policy :: Contact Us
"Private" Facebook and MySpace Postings are Discoverable : E-Discovery Law Alert: M ... Page 1 of 2

Posted at 1:44 PM on October 11, 2010 by Melissa DeHonney
"Private" Facebook and MySpace Postings are Discoverable
A New York trial court has ordered a personal injury plaintiff to produce her Facebook and MySpace
postings, notwithstanding that plaintiff self-designated them as private. Justice Jeffrey Arlen Spinner, in
Romano v. Steelcase Inc., 2010 N.Y. Slip Op. 20388, 2010 N.Y. Misc. LEXIS 4538 (N.Y. Sup. Ct.,
Suffolk Cty. Sept. 21, 2010), reasoned that New York’s "liberal discovery policies" favored allowing
access to posts that might undermine plaintiff's claim for loss of enjoyment of life and further that, "as
neither Facebook nor MySpace guarantee complete privacy, Plaintiff has no legitimate reasonable
expectation of privacy." Read the decision here.
The Romano opinion suggests that, if a discovering party makes a threshold showing that content posted
on social media websites is relevant, discovery may be had: "In light of the fact that the public portions
of Plaintiff’s social networking sites contain material that is contrary to her claims and deposition
testimony, there is a reasonable likelihood that the private portions of her sites may contain further
evidence such as information with regard to her activities and enjoyment of life, all of which are
material and relevant to the defense of this action." As with traditional modes of discovery, it appears
that "fishing expeditions" will not be allowed.
Most of the cases the Romano court cited where access to social media was granted were likewise
personal injury actions in which the individual plaintiffs were ordered to produce their online journals.
Still, the opinion signals to businesses that they, too, might have to produce content posted on social
media sites. With more and more businesses promoting themselves through this medium, the attendant
preservation obligations should be considered. Posting potentially discoverable information with third-
parties can complicate compliance.
For the same reason, businesses should be mindful of what their employees post on social networking
sites. An employee’s postings may be deemed to reflect on the company and might even bear on matters
relevant to a lawsuit in which the company is a party. All too often, what people say online is as candid
as what they might say to a few close co-workers behind closed doors. Yet, online posts are available for
the world to see. Even seemingly innocuous posts -- for example, "I was in way over my head today at
work" -- can prove damaging in litigation. The most prudent approach would include both restricting
employees' access to these sites at the office and implementing a social media policy addressing
employees' online postings.
Additional discussions on this topic from members of the Gibbons Employment Law Department and
the E-Discovery Task Force can be found here and here.
http://www.ediscoverylawalert.com/2010/10/articles/legal-decisions-court-rules/private-face... 4/2/2013
"Private" Facebook and MySpace Postings are Discoverable : E-Discovery Law Alert: M ... Page 2 of 2
Melissa DeHonney is an Associate on the Gibbons E-Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2010/10/articles/legal-decisions-court-rules/private-face... 4/2/2013
McCann v Harleysville Ins. Co. of N.Y. :: November, 2010 :: New York Appellate Divisi ... Page 1 of 1
Justia.com Lawyer Directory Legal Answers Law more ? Sign In
in ---- 2010 Appellate Division, Fourth Department
Search
Justia > US Law > US Case Law > New York Case Law > New York Appellate Division, Fourth Department Decisions > November, 2010 > M c C a n n v
Harleysville Ins. Co. of N.Y.
N E W - Receive Justia's FREE Daily Newsletters of Opinion Summaries for the US Supreme Court, all US Federal Appellate
Courts & the 50 US State Supreme Courts and Weekly Practice Area Opinion Summaries Newsletters. Subscribe Now
McCann v Harleysville Ins. Co. of N.Y. Justia on

129,094 people like this. Sign Up to see what
Like
Share | your friends like.
McCann v Harleysville Ins. Co. of N.Y. 2010 NY Slip Op 08181 [78 AD3d 1524]
Daily Opinion Summaries
November 12, 2010 Appellate Division, Fourth Department Published by New York State
Subscribe to Justia's FREE Daily
Law Reporting Bureau pursuant to Judiciary Law § 431. As corrected through
Newsletter Opinion Summaries
Wednesday, January 19, 2011
Subscribe Now
Kara R. McCann, Respondent, v Harleysville Insurance Company of New York, Today on Verdict
Appellant. (Appeal No. 1.) W h a t’s the Matter with North Dakota
and Arkansas? Two State Legislatures
— [* 1] Chelus, Herdzik, Speyer & Monte, P.C., Buffalo (Christopher R. Poole of counsel), Pass Highly Restrictive and
Unconstitutional Abortion Laws
for defendant- appellant.
Justia columnist and Hofstra law
Anspach Meeks Ellenberger LLP, Buffalo (David M. Stillwell of counsel), for plaintiff - professor Joanna Grossman
respondent. comments on two states’
decisions to pass abortion laws
Appeal from an order of the Supreme Court, Erie County (James H. Dillon, J.), entered despite the fact that under U.S.
Supreme Court precedent, it is very clear
August 19, 2009 in a personal injury action. The order denied the motion of defendant that these new laws are unconstitutional.
to compel disclosure.
By Joanna L. Grossman
It is hereby ordered that the order so appealed from is unanimously affirmed without
costs.
Ask a L a w y e r
Memorandum: Plaintiff commenced an action seeking damages for injuries she Question:
sustained when the vehicle she was operating collided with a vehicle driven by
defendant's insured. Plaintiff thereafter settled that action and commenced the instant
action against defendant seeking "supplementary uninsured/underinsured motorist
coverage." In appeal No. 1, defendant appeals from an order denying its motion to Ask Question
compel disclosure of photographs and seeking "an authorization for plaintiff's Facebook About Legal Answers
account." According to defendant, the information sought was relevant with respect to
Connect with Justia
the issue whether plaintiff sustained a serious injury in the accident. We conclude in
Follow justiacom Fan Justia
appeal No. 1 that Supreme Court properly denied defendant's motion "as overly broad,"
Justia
without prejudice "to service of new, proper discovery demands" (see generally Slate v
Like
State of New York, 267 AD2d 839, 841 [1999]).
129,094 people like Justia.

In appeal No. 2, defendant appeals from an order denying its subsequent motion
seeking to compel plaintiff to produce photographs and an authorization for plaintiff's
Facebook account information and granting plaintiff's cross motion for a protective
order. Although defendant specified the type of evidence sought, it failed to establish a
Facebook social plugin
factual predicate with respect to the relevancy of the evidence (see Crazytown Furniture
Find a Lawyer
v Brooklyn Union Gas Co., 150 AD2d 420, 421 [1989]). Indeed, defendant essentially
sought permission to conduct "a fishing expedition" into plaintiff's Facebook account Search
based on the mere hope of finding relevant evidence (Auerbach v Klein, 30 AD3d 451, Browse Lawyers
452 [2006]). Nevertheless, although we conclude that the court properly denied
Lawyers - Get Listed Now!
defendant's motion in appeal No. 2, we agree with defendant that the court erred in
Get a free full directory profile listing
granting plaintiff's cross motion for a protective order. Under the circumstances
presented here, the court abused its discretion in prohibiting defendant from seeking
disclosure of plaintiff's Facebook [*2]account at a future date. We therefore modify the
order in appeal No. 2 accordingly. Present— Martoche, J.P., Lindley, Sconiers, Pine and
Gorski, JJ.
Copyright © Justia : : Company : : Terms of Service : : Privacy Policy : : Contact Us
http://law.justia.com/cases/new-york/appellate-division-fourth-department/2010/2010-0818... 4/2/2013
New York's Appellate Division Finds Facebook Accounts Off-Limits When Discovery D ... Page 1 of 2

Posted at 12:54 PM on January 31, 2011 by Robert D. Brown, Jr.
New York's Appellate Division Finds Facebook Accounts Off-Limits When Discovery
Demands are Non-Specific
In McCann v. Harleysville Insurance Co. of New York, 910 N.Y.S.2d 614, 2010 N.Y. App. Div. LEXIS
8396 (N.Y. App. Div. Nov. 12, 2010), New York’s Appellate Division, Fourth Department affirmed the
trial court’s refusal to compel Plaintiff to produce information regarding or provide access to her
Facebook account. Plaintiff was injured in an auto accident with one of Harleysville’s insured. She filed
a personal injury suit against the insured, which resulted in a settlement. Plaintiff thereafter commenced
a new action directly against Harleysville for certain uninsured/underinsured auto insurance benefits.
During discovery, Harleysville sought disclosure of photographs from Plaintiff and an authorization for
access to her Facebook account. Upon Plaintiff’s refusal, Harleysville moved to compel disclosure of
Plaintiff’s Facebook account information. Harleysville argued that the information sought was relevant
to whether McCann suffered a serious injury in the accident without specifying why. The trial court
disagreed, finding that Harleysville’s request was “overly broad,” but noted that the denial was without
prejudice “to service of new, proper discovery demands.” Harleysville then made a second request for
access to Plaintiff’s Facebook account, which the Fourth Department noted specified the type of
evidence Harleysville sought. Plaintiff again refused to disclose her Facebook account information and
Harleysville again moved to compel. The trial court again denied the motion, finding that Harleysville
failed to establish the relevancy of the evidence sought, and further granted Plaintiff’s cross-motion for a
protective order.
On appeal by Harleysville, the Fourth Department, in a succinct opinion, held that the trial court
properly denied Harleysville’s two motions to compel. The Court likened Harleysville’s motions to a
request for permission to conduct a “fishing expedition” into Plaintiff’s Facebook account in hopes of
finding relevant evidence. It disagreed, however, with the trial court’s entry of a protective order,
determining that Harleysville should be allowed to seek disclosure of Plaintiff’s Facebook account in the
future.
The McCann decision is consistent with Romano v. Steelcase Inc., 2010 N.Y. Slip Op. 20388, 2010
N.Y. Misc. LEXIS 4538 (N.Y. Sup. Ct. Sept. 21, 2010) and McMillen v. Hummingbird Speedway, Inc.,
Case No. 113-2010 CD, 2010 WL 4403285 (Pa. Ct. of Com. Pl. Sept. 9, 2010), which were previously
discussed here and here. Implicit in all three decisions is that non-public communications and
information found on social networking websites will not be protected from disclosure when it is
properly demonstrated that the sought-after information is relevant to the litigation. As was previously
discussed here, publicly accessible information on such sites has already been found to be discoverable.
Unlike the boilerplate and catch-all discovery requests in personal injury actions, requests for access to a
http://www.ediscoverylawalert.com/2011/01/articles/legal-decisions-court-rules/new-yorks-... 4/2/2013
New York's Appellate Division Finds Facebook Accounts Off-Limits When Discovery D ... Page 2 of 2
litigant’s Facebook or other social networking account must be specifically tailored and, if possible, tied
to actual facts or information found on the public portions of a litigant’s account.
Robert D. Brown, Jr. is an Associate on the Gibbons E-Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2011/01/articles/legal-decisions-court-rules/new-yorks-... 4/2/2013
PRO I HO NOTARY
FRANKLIN COIJNTY PA
nail NOV 8 P L: 3
I rIfl
bL.
P RUT HO NOTARY
DL PUT Y_O____----..
OPINION AND ORDER
"Facebook helps you connect and share with the people in your life." But what if the
people in your life want to use your Facebook posts against you in a civil lawsuit? Whether and
to what extent online social networking information is discoverable in a civil case is the issue
currently before the Court.
s TRUE,
1. http://www.facebook.com .
DALLUD, PROTHONOTARY
Largent v. Reed No. 2009-1823
Opinion Page 2
Defendant Jessica Rosk0 2 has filed a Motion to Compel Plaintiff Jennifer Largent’s
Facebook Login Information. Rosko has a good faith belief that information on Jessica Largent’s
Facebook profile is relevant to Rosko’s defense in this matter. For the following reasons, the
Court holds that the information sought is discoverable, and we will grant the motion to compel.
Background
I. Underlying Facts
This case arises out of a chain-reaction auto accident that occurred four years ago.
According to the pleadings, Plaintiff Keith Largent was driving a 1986 Honda Shadow
Motorcycle on Lincoln Way East in Chambersburg, with Plaintiff Jessica Largent as a passenger.
Compl. 114, 19. At an intersection, Rosko collided with a minivan driven by Additional
Defendant Sagrario Pena, pushing the van into Plaintiffs’ motorcycle. j4 116-7. As a result of
the crash, Plaintiffs allege serious and permanent physical and mental injuries, pain, and
suffering. Id. TT 11-14, 17, 21-24, 27.

On April 27, 2009, Plaintiffs filed their four-count Complaint against Rosko. Plaintiffs
allege two counts each of negligence and loss of consortium. On July 20, Rosko filed an Answer
with New Matter, to which Plaintiffs filed a Reply on August 11. The pleadings joining Pena as
an additional defendant are not relevant here, and he is not a party to the instant motion.
During the deposition of Jennifer Largent, taken May 18, 2011, Defense counsel
discovered that she has a Facebook profile, that she had used it regularly to play a game called
FrontierVille, and that she last accessed it the night before the deposition. Def. ’5 Mot. to Compel,
Ex. A, Dep. of Jennifer Elaine Largent, 90-91, 94. Rosko refused, however, to disclose any
2. Ms. Rosko was formerly known as Jessica Reed. Def.’s Answer with New Matter 13. The Court will refer to her
by her current name.
Opinion Page 3
information about the account, and Plaintiffs’ counsel advised that it would not voluntarily turn
over such information. Id. This motion to compel followed on August 1, 2011.
II. Facebook
Facebook is a free social networking site. To join, a user must set up a profile, which is
accessible only through the user’s ID (her email) and a password. Facebook allows users to
interact with, instant message, email, and friend or unfriend other users; to play online games;
and to upload notes, photos, and videos. Facebook users can post status updates about what they
are doing or thinking. Users can post their current location to other friends, suggest restaurants,
businesses, or politicians or political causes to "like," and comment or "like" other friends’
posts. 3
Social networking websites like Facebook, Google+, and MySpace are ubiquitous.
Facebook, which is only seven years old, has more than 800 million active users, 50% of whom
are active on the site at any given day. 4 Facebook has spawned a field of academic research,
books, and a movie. Social networking websites also have a dark sidethey have caused
criminal investigations and prosecutions and civil tort actions. See, e.g., Chapman V. Unemp’t
Comp. Bd. of Review, 20 A.3d 603 (Pa. Cmwlth. 2011) (employee fired for posting on Facebook
while at work); United States v. Drew, 259 F.R.D. 449 (C.D. Cal. 2009) (woman criminally
prosecuted for breaching MySpace’s terms of use); In re Rolando 5., 129 Cal. Rptr. 3d 49 (Ct.
App. 2011) (prosecution of a juvenile who hacked another child’s Facebook account and posted
3. Facebook currently does not allow a person to "dislike" (or in Facebook parlance, "un-like") a friend’s post,
probably for good reason.
4. Facebook Statistics http://www.facebook.com/press/info.php?statistics (last visited Oct. 25, 2011).

Opinion Page 4
vulgar material therein); Finkel v. Dauber, 906 N.Y.S.2d 697 (Sup. Ct. Nassau 2010) (lawsuit
concerning allegedly defamatory material posted in a Facebook group).
Facebook has a detailed, ever-changing privacy policy. Only people with a user account
can access Facebook. For all practical purposes, anyone with an email account can set up a
Facebook account. 5 Users can set their privacy settings to various levels, although a person’s
name, profile picture, and user ID are always publically available. At the least restrictive setting,
named "public," all 800 million users can view whatever is on a certain user’s profile.’ At an
intermediate level, only a user’s Facebook friends can view such information, and at the least
restrictive, only the user can view his or her profile. Facebook also currently allows users to
customize their privacy settings.
Facebook alerts users that Facebook friends may "tag" them in any posting, such as a
photograph, a note, a video, or a status update. A tag is a link to a user’s profile:
If someone clicks on the link, they will see your public information and anything
else you let them see.
Anyone can tag you in anything. Once you are tagged in a post, you and your
friends will be able to see it. For example, your friends may be able to see the post
in their News Feed or when they search for you. It may also appear on your
profile.
You can choose whether a post you’ve been tagged in appears on your profile.
You can either approve each post individually or approve all posts by your
friends. If you approve a post and later change your mind, you can always remove
it from your profile.
5. To comply with federal law, one must be 13 or older to have a Facebook account. This policy is apparently hard
to enforce and is openly flouted.
6. Facebook Data Use PolicySharing and finding you on Facebook,

http://www.facebook.com/aboutlprivacy/your-info-on-fb#controlprofile (last visited Oct. 27, 2011).
Opinion Page 5
If you do not want someone to tag you in their posts, we encourage you to reach
out to them and give them that feedback. If that does not work, you can block
them. This will prevent them from tagging you going forward.
If you are tagged in a private space (such as a message or a group) only the people
who can see the private space can see the tag. Similarly, it you are tagged in a
comment, only the people who can see the comment can see the tag. 7
Therefore, users of Facebook know that their information may be shared by default, and a user
must take affirmative steps to prevent the sharing of such information.
Facebook also alerts users that it may reveal information pursuant to legal requests:
Responding to legal requests and preventing harm
We may share your information in response to a legal request (like a search

warrant, court order or subpoena) if we have a good faith belief that the law
requires us to do so. This may include responding to legal requests from
jurisdictions outside of the United States where we have a good faith belief that
the response is required by law in that jurisdiction, affects users in that
jurisdiction, and is consistent with internationally recognized standards. We may
also share information when we have a good faith belief it is necessary to: detect,
prevent and address fraud and other illegal activity; to protect ourselves and you
from violations of our Statement of Rights and Responsibilities; and to prevent
death or imminent bodily harm. 8
Discussion
Rosko has moved to compel Jennifer Largent to disclose her Facebook username and
password. She claims that, as of January 2011, Largent’s Facebook profile was public, meaning
that anyone with an account could read or view her profile, posts, and photographs. Rosko says
that certain posts on Largent’s Facebook account contradict her claims of serious and severe
7. Facebook Data Use PolicySharing and finding you on Facebook,

http://www.facebook.com/about/privacy/your-info-on-fb#friefldSShare.
8. Facebook Data Use PolicySome things you need to know, http://www.facebook.comlaboutlprivacy/other

(internal hyperlinks removed).
Opinion Page 6
injury. Specifically, Rosko claims that Largent had posted several photographs that show her
enjoying life with her family and a status update about going to the gym.
Jennifer Largent responds that the information sought is irrelevant and does not meet the
prima facie threshold under Pennsylvania Rule of Civil Procedure 4003.1. She further argues that
disclosure of her Facebook account access information would cause unreasonable embarrassment
and annoyance. Finally, she claims that disclosure may violate privacy laws such as the Stored
Communications Act of 1986, Pub. L. No. 99-508, tit. II, § 201, 100 Stat. 1848 (codified as
amended at 18 U.S.C. §§ 2701-12).
I. Discovery Standard
In Pennsylvania,
a party may obtain discovery regarding any matter, not privileged, which is
relevant to the subject matter involved in the pending action, whether it relates to
the claim or defense of the party seeking discovery or to the claim or defense of
any other party, including the existence, description, nature, content, custody,
condition, and location of any books, documents, or other tangible things and the
identity and location of persons having knowledge of any discoverable matter.
Pa. R.C.P. 4003.1(a). It is no objection that the material sought will be inadmissible at trial, so
long as the material "appears reasonably calculated to lead to the discovery of admissible
evidence." Pa. R.C.P. 4003.1(b). Therefore, the material Rosko seeks must be relevant and not
privileged.
The Pennsylvania discovery rules are broad, and the relevancy threshold is slight.
George v. Schirra, 814 A.2d 202, 204 (Pa. Super. 2002). Relevancy is not limited to the issues
raised in the pleadings, and it carries a broader meaning than the admissibility standard at trial.
Id. at 205; 9 Goodrich Amram 2d § 4003.1(a):(6).

Opinion Page 7
There are no Pennsylvania appellate opinions addressing whether material contained on
social networking websites is discoverable in a civil case. This is most likely because social
networking is a recent phenomenon and issues are just beginning to percolate in the courts. $
Evan B. North, Comment, Facebook Isn’t Your Space Anymore: Discovery of Social Networking
Websites, 58 U. Kan. L. Rev. 1279, 1308 (2010) (noting social networking websites’ effect on
discovery).
Because of the lack of binding authority, the Parties have cited trial courts in this State
and others. Rosko cites two cases in which the court permitted discovery of material on social
network websites. Zimmerman v. Weis Mkts., Inc., No. CV-09-1535, 2011 WL 2065410 (Pa.
C.P. Northumberland May 19,2011); McMillen v. Hummingbird Speedway, Inc., No. 113-2010
CD, 2010 WL 4403285 (Pa. C.P. Jefferson Sept. 9, 2010). In response, Plaintiffs cite two cases
where courts have denied discovery of Facebook material. Piccolo v. Paterson, No. 2009-4979
(Pa. C.P. Bucks May 6, 2011); Kennedy v. Norfolk S. Corp., No. 100201437 (Pa. C.P. Phila. Jan
15, 20l1).
Courts in other jurisdictions have also allowed discovery of social networking data in
civil lawsuits. See, e.g., Offenback v. LM Bowman, Inc., No. 1:10-CV-1789, 2011 WL 2491371
(M.D. Pa. June 22, 2011); EEOC v. Simply Storage Mgmt., LLC, 270 F.R.D. 430 (S.D. Ind.
2010); Romano v. Steelcase, Inc., 907 N.Y.S.2d 650 (Sup. Ct. Suffolk 2011).
In Offenback, a personal injury case, the court ordered the plaintiff to turn over data
contained on his Facebook page in a form mutually agreeable to the parties. Offenback, 2011
9. Counsel’s citation of these cases (via an online Pittsburgh Post-Gazette article and an article written by an
attorney from a law firm specializing in plaintiffs’ personal injury representation) is inadequate. Counsel is
reminded that the proper way to cite an unreported case is, at minimum, to provide a docket number so that the
Court does not need to conduct a wild goose chase to find the case.
Opinion Page 8
WL 2491371, at *2..3. Specifically, the court ordered Offenback to turn over information that
contradicted his claim of injury. j4 In Simply Storage, a Title VII sexual harassment case, the
court allowed discovery of Facebook material including status updates, communications between
two plaintiffs who alleged emotional distress injuries, and photographs and videos. Simply
Storage, 240 F.R.D. at 436. In Romano, a personal injury action, the court ordered access - to all
the plaintiffs’ social networking website information. Romano, 907 N.Y.S.2d at 657.
As far as the threshold relevancy inquiry is concerned, it is clear that material on social
networking websites is discoverable in a civil case. Pennsylvania’s discovery rules are broad,
and there is no prohibition against electronic discovery of relevant information. Furthermore,
courts in other jurisdiction with similar rules have allowed discovery of social networking data.
Rosko claims a good faith basis for seeking material contained on Jennifer Largent’s
Facebook account. Largent has pleaded that she suffers from, among other things, chronic
physical and mental pain. Compi. 121. At her deposition, Largent testified that she suffers from
depression and spasms in her legs, and uses a cane to walk. Def.’s Mot. to Compel, Ex. A 65, 85.
Rosko claims that Largent’s formerly public Facebook account included status updates about
exercising at a gym and photographs depicting her with her family that undermine her claim for
damages. The information sought by Rosko is clearly relevant. The information sought by Rosko
might prove that Largent’s injuries do not exist, or that they are exaggerated. Therefore, Rosko
satisfies the relevancy requirement.
II. Privilege and Privacy Concerns
Having determined that Rosko satisfies the threshold relevancy requirement, the Court
must determine whether privilege or privacy rights protect against discovery. Privileged matter is
Opinion Page 9
not discoverable. Pa R.C.P. 4011(c); 4003.1(a). The term "privilege" refers only to those
recognized by the common law, statutory law, or the Constitution. S.M. ex rel. R.M. v. Children
& Youth Servs. of Del. Cnty., 686 A.2d 872, 874-75 (Pa. Cmwlth. 1996). If either
Pennsylvania’s law of privilege or statutory law, such as the Stored Communications Act,
prohibits disclosure, the relevant information Rosko seeks is not discoverable.
A. Privilege Under Pennsylvania Law
Pennsylvania disfavors privileges, and the law recognizes only a limited number of
privileges. Joe v. Prison Health Servs., Inc., 782 A.2d 24, 30-31 (Pa. Cmwlth. 2001).
There is no confidential social networking privilege under existing Pennsylvania law.
McMillan 2010 ’WL 4403285. There is no reasonable expectation of privacy in material posted
on Facebook. Almost all information on Facebook is shared with third parties, and there is no
reasonable privacy expectation in such information. 10 Cf. Commonwealth v. Proetto, 771 A.2d
823, 828 (Pa. Super. 2001).
When a user communicates on Facebook, her posts may be shared with strangers.
McMillan, 2010 WL 4403285. And making a Facebook page "private" does not shield it from
discovery. Simply Storage Mgmt., 270 F.R.D. at 434; Patterson v. Turner Constr. Co.,
N.Y.S.2d ---, 2011 N.Y. Slip Op. 07572, 2011 WL 5083155 (App. Div. Oct. 27, 2011). This is so
because, as explained above, even "private" Facebook posts are shared with others.
The Court holds that no general privacy privilege protects Jennifer Largent’s Facebook
material from discovery. No court has recognized such privilege, and neither will we. By
10. There may be a reasonable expectation of privacy in undelivered Facebook email messages. That expectation of
privacy vanishes once the email reaches the intended recipient. Commonwealth v. Proetto, 771 A.2d 823, 828 (Pa.
Super. 2001); accord United States v. Lifshitz, 369 F.3d 173, 190 (2d Cir. 2004). ("[Computer users have no
reasonable] expectation of privacy in transmissions over the Internet or e-mail that have already arrived at the
recipient.").
Opinion I
Page 10
definition, there can be little privacy on a social networking website. Facebook’s foremost
purpose is to "help you connect and share with the people in your life." That can only be
accomplished by sharing information with others. Only the uninitiated or foolish could believe
that Facebook is an online lockbox of secrets.
B. The Stored Communications Act
The Court next must determine whether the Stored Communications Act (SCA or Act)
prohibits disclosure of Jennifer Largent’s Facebook information." The SCA is part of the
Electronic Communications Privacy Act, Pub. L. No. 99-508, 100 Stat. 1848 (1986). The SCA
fills the gaps left by the Fourth Amendment, which weakly protects the digital and electronic
worlds. Orin S. Kerr, A User’s Guide to the Stored Communications Act, and a Legislature’s
Guide to Amending It, 72 Geo. Wash. L. Rev. 1208, 1212-13 (2004). The SCA does this by
creating limits on the government’s ability to compel Internet Service Providers (ISPs) to
disclose information about their users, and it places limits on ISPs’ ability to voluntarily disclose
information about their customers and subscribers to the government. See 18 U.S.C. §§ 2702-03.
Crucial to the resolution of this motion, the SCA regulates only ISPs or other types of
network supporters. It divides ISPs into two categories: electronic communications services
(ECSs) and remote computing services (RCSs). An ECS is "any service which provides to users
thereof the ability to send or receive wire or electronic communications." 18 U.S.C. § 2510(15).
An RCS stores data long-term for processing or storage. j4 § 2711(2). The terms are somewhat
confusing because they reflect the state of computing technology as it existed in 1986 (a time
before smartphones, Facebook, and the World Wide Web). Kerr, 72 Geo. Wash. L. Rev, at 1213.
11. Largent argues that disclosure of her Facebook information "may violate privacy laws." Pis.’ Answer to Def. ’S
Mot. to Compel 121. As she cites only the SCA, it is the only authority that the Court addresses.
Opinion Page 11
To simplify greatly, RCSs store information for longer periods of time than ECSs. 12 The SCA
applies differently to each but, as will be apparent below, the minutiae are irrelevant for our
purposes.
Only one court has addressed whether Facebook is an entity covered by the SCA. See
Crispin v. Christian Audigier, Inc., 717 F. Supp. 2d 965 (C.D. Cal. 2010). In Crispin, the
defendants served subpoenas upon Facebook and other social networking sites seeking
information about the plaintiff’s online postings. Id. at 969. The plaintiff filed a motion to quash
the subpoenas arguing, among other things, that the SCA prohibited disclosure. Id. In a
comprehensive opinion, the court held that Facebook is both an ECS and an RCS, depending on
which function of the site is at issue. Id. at 987-88, 990.
The court granted the motion to quash. In doing so, it held that civil subpoenas are never
permissible under the SCA. I4 at 975-76 (quoting Viacom Int’l, Inc. v. YouTube, Inc., 253
F.R.D. 256, 264 (S.D.N.Y. 2008); In re Subpoena Duces Tecum to AOL, LLC, 550 F. Supp. 2d
606, 611 (E.D. Va. 2008); O’Grady v. Super. Ct., 139 Cal. App. 4th 1423 (2006)).
Crispin is distinguishable. In that case, the defendants sought information via subpoena to
Facebook and other social networking sites. In this case, Rosko seeks the information directly
from Jennifer Largent. The SCA does not apply because Largent is not an entity regulated by the
SCA. She is neither an RCS nor an ECS, and accessing Facebook or the Internet via a home
computer, smartphone, laptop, or other means does not render her an RCS or ECS. See Kerr, 72
Geo. Wash. L. Rev, at 1214. She cannot claim the protection of the SCA, because that Act does
not apply to her. "The SCA is not a catch-all statute designed to protect the privacy of stored
12. For a much more comprehensive explanation of the SCA, see the law review article cited herein and Crispin v.
Christian Audigier, Inc., 717 F. Supp. 2d 965, 971-73 (C.D. Cal. 2010).
Opinion Page 12
Internet communications." Id. Rather, it only applies to the enumerated entities. Largent being
neither an ECS nor an RCS, the SCA does not protect her Facebook profile from discovery.
III. Breadth of Discovery Request
Finally, having determined that the information sought by Rosko is relevant and not
privileged, the Court must consider whether her request is overbroad. No discovery is permitted
if it is not relevant to the pending action or if it would cause unreasonable annoyance,
embarrassment, oppression, burden, or expense. Pa. R.C.P. 4011. The mere existence of some
annoyance or embarrassment is insufficient to bar discovery. 9A Goodrich Amram 2d
§ 4011(b): 1. Unreasonableness is determined on a case-by-case basis.
As we noted above, Largent has no privacy rights in her Facebook postings, and there is
no general Facebook social networking privilege. Furthermore, she cannot claim the protections
of the Stored Communications Act.
We further note that, in filing a lawsuit seeking monetary damages, Largent has placed
her health at issue, which vitiates certain privacy interests. Any posts on Facebook that concern
Largent’ s health, mental or physical, are discoverable, and any privilege concerning such
information is waived. Gormley v. Edgar, 995 A.2d 1197, 1206 (Pa. Super. 2010); Kraus v.
Taylor, 710 A.2d 1142 (Pa. Super. 1998), alloc. granted, 727 A.2d 1109 (Pa. 1999), and alloc.
dismissed, 743 A.2d 451 (Pa. 2000).
Largent complains that Rosko’s motion is akin to asking her to turn over all of her private
photo albums and requesting to view her personal mail. Pls.’ Answer to Def.’s Mot. to Compel
¶ 23. But those analogies are mistaken in their characterization of material on Facebook.
Photographs posted on Facebook are not private, and Facebook postings are not the same as
Opinion Page 13
personal mail. Largent points to nothing specific that leads the Court to believe that discovery
would cause unreasonable embarrassment. Bald assertions of embarrassment are insufficient. As
the court stated in McMillan, Facebook posts are not truly private and there is little harm in
disclosing that information in discovery. 13
Nor does the Court believe that allowing Rosko access to Largent’s Facebook profile will
cause unreasonable annoyance. The court notes that the entire cost of investigating Largent’s
Facebook information will be borne by Rosko. Also, Largent can still access her account while
Rosko is investigating. As Rosko argues, this is one of the least burdensome ways to conduct
discovery.
Finally, the Court finds it significant that the only two Pennsylvania trial courts (of which
we are aware) have granted discovery in identical situations. Zimmerman, 2011 WL 2065410;
McMillen 2010 WL 4403285. The cases cited by Largent, though to the contrary, lack any
persuasive authority because those orders are unsupported by any written opinion or
memorandum.
We agree with Rosko that information contained on Jennifer Largent’s Facebook profile
is discoverable. It is relevant and not covered by any privilege, and the request is not
unreasonable. We will thus allow Rosko access to Largent’s Facebook account to look for the
necessary information. Plaintiff Jessica Largent must turn over her Facebook login information
to Defense counsel within 14 days of the date of the attached Order. Defense counsel is allotted a
13. The Court does not hold that discovery of a party’s social networking information is available as a matter of
course. Rather, there must be a good faith basis that discovery will lead to relevant information. Here, that has
occurred because Jennifer Largent’s profile was formerly public. In other cases, it might be advisable to submit
interrogatories and requests for production of documents to find out if any relevant information exists on a person’s
online social networking profiles.
Opinion Page 14
21-day window in which to inspect Largent’s profile. After the window closes, Plaintiff may
change her password to prevent any further access to her account by Defense counsel.
An Order follows.
ORDER OF CO URT
November Z_, 2011, the Court having reviewed Defendant Jessica Rosko’ Motion to
Compel Plaintiff Jennifer Largent’s Facebook Login Information, Plaintiffs’ Answer thereto, the
briefs, the record, and the law,
IT IS HEREBY ORDERED that Defendant’s Motion to Compel be, and hereby is,
GRANTED.
IT IS FURTHER ORDERED that Plaintiff Jennifer Largent shall turn over to Defense
counsel her Facebook username email and password within 14 days of the date of this Order.
Plaintiff shall not delete or otherwise erase any information on her Facebook account. After 35
days from the date of this Order, Plaintiff may change her Facebook login password to prevent
further access by Defense counsel.

EST TRUE PY
LI DA L. BEARD PR’)?H )7 1Y
Order of Court Page 2
Pursuant to the requirements of Pa. R.C.P. 236 (a)(2), (b), (d), the Prothonotary shall
immediately give written notice of the entry of this Order, including a copy of this Order, to each
party’s attorney of record, or if unrepresented, to each party; and shall note in the docket the
giving of such notice and the time and manner thereof
By the Court,
J.
The Prothonotary shall give notice to:

Christopher T. Moyer, Esq., Counsel for Plaintiffs
Donald L. Carmelite, Esq., Counsel for Defendant
Stephen J. Magley, Esq., Counsel for Additional Defendant
Case 2:11-cv-03892-DOC-SS Document 136 Filed 09/07/12 Page 1 of 12 Page ID #:2740
1
2
3
4
5
6
7
8 UNITED STATES DISTRICT COURT
9 CENTRAL DISTRICT OF CALIFORNIA
10
11 DANIELLE MAILHOIT, ) No. CV 11-03892 DOC (SSx)
)
12 Plaintiff, ) ORDER GRANTING IN PART AND DENYING
) IN PART DEFENDANT’S MOTION TO
13 v. ) COMPEL FURTHER RESPONSES TO
) DEFENDANT’S REQUEST FOR PRODUCTION
14 HOME DEPOT U.S.A., INC., et ) REGARDING SOCIAL NETWORKING SITE
al., ) MATERIAL (SET ONE)
15 )
Defendants. ) (Dkt. No. 105)
16 )
)
17
18
19 I.
20 INTRODUCTION
21
22 On August 7, 2012, Defendant filed a Motion to Compel Further
23 Responses to Defendant’s Request for Production of Documents (Set One).
24 (Dkt. No. 105). The parties filed a Joint Stipulation concurrently with
25 the Motion pursuant to Local Rule 37, (“Jt. Stip.”), including the
26 declarations of Elizabeth A. Falcone in support of the Motion, (Dkt. No.
27 107), and Kenneth Helmer in opposition to the Motion. (Dkt. No. 110).
28
1 The Court held a hearing on the Motion on August 28, 2012. For the
2 reasons stated below, the Motion is GRANTED IN PART and DENIED IN PART.
3
4 II.
5 THE PARTIES’ CONTENTIONS
6
7 Defendant requests an Order compelling Plaintiff to produce
8 documents responsive to Requests for Production Nos. 46-49, which
9 collectively seek:
10
11 (1) Any profiles, postings or messages (including status updates,
12 wall comments, causes joined, groups joined, activity streams,
13 blog entries) from social networking sites from October 2005
14 (the approximate date Plaintiff claims she first was
15 discriminated against by Home Depot), through the present,
16 that reveal, refer, or relate to any emotion, feeling, or
17 mental state of Plaintiff, as well as communications by or
18 from Plaintiff that reveal, refer, or relate to events that
19 could reasonably be expected to produce a significant emotion,
20 feeling, or mental state;
21
22 (2) Third-party communications to Plaintiff that place her own
23 communications in context;
24
25 (3) All social networking communications between Plaintiff and any
26 current or former Home Depot employees, or which in any way
27 refer [or] pertain to her employment at Home Depot or this
28 lawsuit; or
2
1 (4) Any pictures of Plaintiff taken during the relevant time

2 period and posted on Plaintiff’s profile or tagged1 or
3 otherwise linked to her profile.
4
5 (Jt. Stip. at 2).2
6
7 Defendant argues that it is entitled to Plaintiff’s communications
8 posted on social networking sites (“SNS”) such as Facebook and LinkedIn
9 to test Plaintiff’s claims about her mental and emotional state. (Id.
10 at 1). According to Defendant, Plaintiff testified at her deposition
11 that she suffers from post traumatic stress disorder, depression and
12 isolation, and has cut herself off from communication with friends
13 because of Defendant’s alleged wrongdoing. (Id.). Defendant argues
14 that SNS communications are particularly likely to contain relevant
15 information because “in this day and age, many communications between
16 friends and/or about an individual’s emotional state are communicated
17 via social media.” (Id.). Defendant states that it has evidence
18 suggesting that Plaintiff maintains Facebook and LinkedIn accounts and
19
1
“‘Tagging’ is the process by which a third party posts a picture
20
and links people in the picture to their profiles so that the picture
21 will appear in the profiles of the person who ‘tagged’ the people in the
picture, as well as on the profiles of the people who were identified in
22 the picture.” EEOC v. Simply Storage Mgmt., LLC, 270 F.R.D. 430, 436
n.3 (S.D. Ind. 2010).
23
2
As written, Requests for Production Nos. 46-49 appear to require
24 production of the entire contents of Plaintiff’s SNS accounts and are
25 overbroad, as Defendant appears to have recognized. (See Jt. Stip. at
4, 15-16, 26 & 37). The instant Motion is limited to a request for an
26 Order compelling production of only the four categories of documents
described above, which arguably overlap several different requests for
27 production as originally written. (See Jt. Stip. at 2, 9). The Court
will therefore address these four categories instead of the four
28 original production requests that they supersede.
3
1 that publicly available information from those sites undermines

2 Plaintiff’s claims of isolation and loss of friendship. (Id. at 8).
3
4 Plaintiff acknowledges that “social media is discoverable to the
5 extent it is adequately tailored to satisfy the relevance standard,” but
6 argues that Plaintiff’s requests are impermissibly overbroad. (Id. at
7 11). According to Plaintiff, rather than tailor its requests, Defendant
8 seeks “to rummage through the entirety of [Plaintiff’s] social media
9 profiles and communications in the hope of concocting some inference
10 about her state of mind.” (Id. at 3). Plaintiff further argues that
11 the requested discovery is unduly burdensome because she has already
12 testified about her emotional distress, as well as produced or agreed to
13 produce “documents and communications pertaining to her emotional
14 distress damages going as far back as 2004,” (id.), which Plaintiff
15 maintains constitute “sufficiently relevant responses.” (Id. at 14).
16 In particular, Plaintiff asserts that she has already responded to
17 requests for her communications with sixteen different current or former
18 Home Depot employees, which Plaintiff contends “presumably” include her
19 communications via social media. (Id.).
20
21
22
23
24
25
26
27
28
4
1 III.
2 DISCUSSION
3
4 A. Discovery Requests For Social Networking Site Content Must Be
5 Reasonably Calculated To Lead To The Discovery Of Admissible
6 Evidence And Describe The Information To Be Produced With
7 “Reasonable Particularity”
8
9 A party may “obtain discovery regarding any nonprivileged matter
10 that is relevant to any party’s claim or defense -- including the
11 existence, description, nature, custody, condition, and location of any
12 documents or other tangible things.” Fed. R. Civ. P. 26(b)(1).
13 Relevancy is construed broadly to encompass “any matter that bears on,
14 or that reasonably could lead to other matter[s] that could bear on any
15 issue that is or may be in the case.” Chavez v. DaimlerChrysler Corp.,
16 206 F.R.D. 615, 619 (S.D. Ind. 2002) (internal quotations omitted). The
17 Supreme Court has instructed that the limitation on discovery to
18 “relevant” materials must be “firmly applied,” as “the discovery
19 provisions, like all of the Federal Rules of Civil Procedure, are
20 subject to the injunction of Rule 1 that they ‘be construed to secure
21 the just, speedy, and inexpensive determination of every action.’”
22 Herbert v. Lando, 441 U.S. 153, 177, 99 S. Ct. 1635, 60 L. Ed. 2d 115
23 (1979) (quoting Fed. R. Civ. P. 1) (emphasis in original).
24
25 Pursuant to Federal Rule of Civil Procedure 34(a), a party may
26 request documents “in the responding party’s possession, custody, or
27 control.” Rule 34(b) requires the requesting party to describe the
28 items to be produced with “reasonable particularity” and specify a
5
1 reasonable time, place, and manner for the inspection. Fed. R. Civ. P.
2 34(b)(1-2). “The test for reasonable particularity is whether the
3 request places a party upon ‘reasonable notice of what is called for and
4 what is not.’” Bruggeman ex rel. Bruggeman v. Blagojevich, 219 F.R.D.
5 430, 436 (N.D. Ill. 2004) (quoting Parsons v. Jefferson–Pilot Corp., 141
6 F.R.D. 408, 412 (M.D. N.C. 1992)); see also Regan-Touhy v. Walgreen Co.,
7 526 F.3d 641, 649-50 (10th Cir. 2008) (“Though what qualifies as
8 ‘reasonabl[y] particular’ surely depends at least in part on the
9 circumstances of each case, a discovery request should be sufficiently
10 definite and limited in scope that it can be said ‘to apprise a person
11 of ordinary intelligence what documents are required and [to enable] the
12 court . . . to ascertain whether the requested documents have been
13 produced.’”) (quoting Wright & Miller, 8A Federal Practice and Procedure
14 § 2211, at 415). “‘All-encompassing demands’ that do not allow a
15 reasonable person to ascertain which documents are required do not meet
16 the particularity standard of Rule 34(b)(1)(A).” In re Asbestos
17 Products Liability Litigation (No. VI), 256 F.R.D. 151, 157 (E.D. Pa.
18 2009).
19
20 The Court recognizes that social networking site content may be
21 subject to discovery under Rule 34. “Generally, SNS content is neither
22 privileged nor protected by any right of privacy.” Davenport v. State
23 Farm Mut. Auto. Ins. Co., 2012 WL 555759 at *1 (M.D. Fla. Feb. 21,
24 2012). However, “[d]iscovery of SNS requires the application of basic
25 discovery principles in a novel context.” Simply Storage Mgmt., 270
26 F.R.D. at 434. In particular, several courts have found that even
27 though certain SNS content may be available for public view, the Federal
28 Rules do not grant a requesting party “a generalized right to rummage at
6
1 will through information that [the responding party] has limited from
2 public view” but instead require “a threshold showing that the requested
3 information is reasonably calculated to lead to the discovery of
4 admissible evidence.” Tompkins v. Detroit Metropolitain Airport, 278
5 F.R.D. 387, 388 (E.D. Mich. 2012); see also Davenport, 2012 WL 555759 at
6 *1 (“A request for discovery [of SNS content] must still be tailored
7 . . . so that it ‘appears reasonably calculated to lead to the discovery
8 of admissible evidence.’”) (quoting Fed. R. Civ. P. 26(b)(1));
9 Mackelprang v. Fidelity Nat’l Title Agency of Nevada, Inc., 2007 WL
10 119149 at *7 (D. Nev. Jan 9, 2007) (“Ordering . . . release of all of
11 the private email messages on Plaintiff’s Myspace.com internet account
12 would allow Defendants to cast too wide a net for any information that
13 might be relevant and discoverable.”).
14
15 Where discovery requests seek SNS communications in connection with
16 claims involving the responding party’s mental or emotional health,
17 several courts have also found that “the simple fact that a claimant has
18 had social communication is not necessarily probative of the particular
19 mental and emotional health issues in the case. Rather, it must be the
20 substance of the communication that determines relevance.” Simply
21 Storage Mgmt., 270 F.R.D. at 435; Holter v. Wells Fargo and Co., 281
22 F.R.D. 340, 344 (D. Minn. 2011). As one court reasoned, “To be sure,
23 anything that a person says or does might in some theoretical sense be
24 reflective of her emotional state. But that is hardly justification for
25 requiring the production of every thought she may have reduced to
26 writing, or, indeed, the deposition of everyone she may have talked to.”
27 Rozell v. Ross-Holst, 2006 WL 163143 at *3-4 (S.D. N.Y. Jan. 20, 2006).
28 Thus, while a party may conduct discovery concerning another party’s
7
1 emotional state, the discovery itself must still comply with the general
2 principles underlying the Federal Rules of Civil Procedure that govern
3 discovery.
4
5 “A court can limit discovery if it determines, among other things,
6 that the discovery is: (1) unreasonably cumulative or duplicative; (2)
7 obtainable from another source that is more convenient, less burdensome,
8 or less expensive; or (3) the burden or expense of the proposed
9 discovery outweighs its likely benefit.” Favale v. Roman Catholic
10 Diocese of Bridgeport, 235 F.R.D. 553, 558 (D. Conn. 2006) (internal
11 citations and quotation marks omitted). “The district court enjoys
12 broad discretion when resolving discovery disputes, which should be
13 exercised by determining the relevance of discovery requests, assessing
14 oppressiveness, and weighing these factors in deciding whether discovery
15 should be compelled.” Id. (internal citations and quotation marks
16 omitted).
17
18 B. The Majority Of Defendant’s Social Media Requests Fail Rule
19 34(b)(1)(A)’s Reasonable Particularity Requirement And Therefore
20 Are Not Reasonably Calculated To Lead To The Discovery Of
21 Admissible Evidence
22
23 The Court finds that three of the four categories of SNS
24 communications sought by Defendant fail Rule 34(b)(1)(A)’s “reasonble
25 particularity” requirement, and, as such, are not reasonably calculated
26 to lead to the discovery of admissible evidence. Consequently, the
27 Court DENIES Defendant’s Motion with respect to Categories 1, 2 and 4 of
28 the revised requests. (See Jt. Stip. at 2).
8
1 Category 1 seeks any “profiles, postings or messages (including

2 status updates, wall comments, causes joined, groups joined, activity
3 streams, blog entries)” from any social networking site from October
4 2005 through the present “that reveal, refer, or relate to any emotion,
5 feeling, or mental state of Plaintiff, as well as communications by or
6 from Plaintiff that reveal, refer, or relate to events that could
7 reasonably be expected to produce a significant emotion, feeling, or
8 mental state.” (Jt. Stip. at 2). Plaintiff has placed her emotional
9 state at issue in this action and it is conceivable that some SNS
10 communications may support or undermine her claims of emotional
11 distress. Nonetheless, the extremely broad description of the material
12 sought by this category fails to put a “reasonable person of ordinary
13 intelligence” on notice of which specific documents or information would
14 be responsive to the request, and therefore fails to satisfy Rule
15 34(b)(1)(A)’s requirement that production requests be stated with
16 reasonable particularity.
17
18 Even if the first part of this category, which seeks communications
19 relating to “any emotion,” could be understood to encompass only
20 communications containing specific emotive words (which the request does
21 not identify), the category would still arguably require the production
22 of many materials of doubtful relevance, such as a posting with the
23 statement “I hate it when my cable goes out.” The second part of the
24 category, which seeks communications relating to “events” that could
25 “reasonably be expected to produce a significant emotion,” is similarly
26 vague and overbroad. Arguably, watching a football game or a movie on
27 television is an “event” that may produce some sort of “significant
28 emotion,” but it is unclear whether Plaintiff would be required to
9
Case 2:11-cv-03892-DOC-SS Document 136 Filed 09/07/12 Page 10 of 12 Page ID
#:2749
1 produce messages relating to such activities. Without more specific

2 guidance, Category 1 is not “reasonably particular.” The language of
3 the request does not provide sufficient notice to the responding party
4 of what should be considered responsive material. Defendant fails to
5 make the “threshold showing” that the request at issue is reasonably
6 calculated to lead to the discovery of admissible evidence.
7
8 Category 2, which requests “third-party communications to Plaintiff
9 that place her own communications in context,” also fails. To the
10 extent that the reference to Plaintiff’s “own communications” means
11 communications regarding “emotions” produced in response to Category 1,
12 Category 2 is entirely predicated on Category 1 and fails for the same
13 vagueness concerns discussed above. Apart from these deficiencies, even
14 if the universe of documents referred to as Plaintiff’s “own
15 communications” could be reasonably circumscribed and understood, the
16 phrase “in context” is vague and also fails to provide notice to
17 Plaintiff of which specific third party communications are and are not
18 called for by the request.
19
20 Finally, Category 4, which requests “any pictures of Plaintiff
21 taken during the relevant time period and posted on Plaintiff’s profile
22 or tagged or otherwise linked to her profile,” is impermissibly
23 overbroad. Defendant fails to make the threshold showing that every
24 picture of Plaintiff taken over a seven-year period and posted on her
25 profile by her or tagged to her profile by other people would be
26 considered relevant under Rule 26(b)(1) or would lead to admissible
27 evidence. See Simply Storage Mgmt., 270 F.R.D. at 436 (“[A] picture
28 posted on a third party’s profile in which a claimant is merely
10
#:2750
1 ‘tagged’[] is less likely to be relevant.”) (footnote omitted). “All

2 encompassing” production requests do not meet Rule 34(b)(1)(A)’s
3 reasonably particularity requirement, In re Asbestos Products Liability
4 Litigation (No. VI), 256 F.R.D. at 157, and discovery rules do not allow
5 a requesting party “to engage in the proverbial fishing expedition, in
6 the hope that there might be something of relevance in [the producing
7 party’s] Facebook account.” Tompkins, 278 F.R.D. at 388 (emphasis in
8 original).3
9
10 In contrast, Category 3, which requests all SNS communications
11 “between Plaintiff and any current or former Home Depot employees, or
12 which in any way refer . . . to her employment at Home Depot or this
13 lawsuit,” adequately places Plaintiff on notice of the materials to be
14 produced and is reasonably calculated to lead to the discovery of
15 admissible evidence. Plaintiff notes that she has already responded to
16 requests for communications between Plaintiff and sixteen different
17 current or former Home Depot employees, which “would presumably include
18
19 3
The Court acknowledges that Categories 1, 2 and 4 are closely
modeled after three categories of SNS communications that the court in
20 Simply Storage Mgmt. ordered produced. See Simply Storage Mgmt., 270
21 F.R.D. at 436. That court recognized, however, that the categories were
not “drawn . . . with the precision litigants and their counsel
22 typically seek.” Id. The court admonished counsel to make “judgment
calls” “in good faith” pursuant to the guidelines articulated by the
23 court in carrying out the order and stated that the requesting party
could challenge the production if it believed it fell short of those
24 guidelines. Id. These admonishments suggest that the court itself was
25 concerned about the parties' ability to carry out the order. As noted
in the discussion above, this Court finds that the requests suggested in
26 Simply Storage Mgmt. are overbroad and vague. These requests fail to
provide enough direction to the responding party to comply with Rule
27 34(b)(1)(A). Accordingly, this Court declines to compel responses to
the discovery modeled on requests described in the Simply Storage Mgmt.
28 case.
11
#:2751
1 communications via social media.” (Jt. Stip. at 36). Plaintiff’s

2 responses to those requests indicate that a search for the
3 communications described in Category 3 is both technically feasible and
4 not overly burdensome. (See, e.g., Helmer Decl., Exh. A at 8-9).
5 Plaintiff did not provide argument or evidence to the contrary in
6 opposition to the current motion. Consequently, the Court GRANTS
7 Defendant’s Motion with respect to Category 3.
8
9 IV.
10 CONCLUSION
11
12 For the foregoing reasons, Defendant’s Motion to Compel Further
13 Responses to Defendant’s Request for Production of Documents (Set One)
14 is GRANTED IN PART and DENIED IN PART. Defendant’s Motion is DENIED
15 with respect to Categories 1, 2 and 4. Defendant’s Motion is GRANTED
16 with respect to Category 3. Plaintiff is ORDERED to serve a written
17 response and to produce documents responsive to Category 3, if any
18 exist, within fourteen (14) days of the date of this Order.
19
20 IT IS SO ORDERED.
21
22 DATED: September 7, 2012
23
24 /S/
SUZANNE H. SEGAL
25 UNITED STATES MAGISTRATE JUDGE
26
27
28
12
No Fishing Expeditions Allowed When It Comes to Discovery of Social Media : E-Disco... Page 1 of 2

Posted at 9:55 AM on October 12, 2012 by Gibbons P.C.
No Fishing Expeditions Allowed When It Comes to Discovery of Social Media
A recent decision in California, Mailhoit v. Home Depot, U.S.A., Inc. et al., Civ. No. 11-03892 (D.E.
105, C.D. Cal. Sept. 7, 2012) reiterates the limits to which social media information is discoverable.
Consistent with Fed. R. Civ. P. 26(b)(1) and 34, the Court made clear in the context of a motion to
compel that “discovery requests for social networking site content must be reasonably calculated to lead
to the discovery of admissible evidence and describe the information to be produced with ‘reasonable
particularity.’”
At issue in Mailhoit were four categories of discovery sought by the defendant in connection with the
plaintiff’s claims of discrimination and emotional distress:
1. Any profiles, postings or messages. . . from social networking sites from October 2005 (the
approximate date Plaintiff first claimed she was discriminated against by Home Depot), through
the present, that reveal, refer, or relate to any emotion, feeling, or mental state of Plaintiff;
2. Third-party communications to Plaintiff that place her own communications in context;
3. All social networking communications between Plaintiff and any current or former Home Depot
employees that pertain to her employment at Home Depot or this lawsuit; and
4. Any pictures of Plaintiff taken during the relevant time period and posted on Plaintiff’s profile.
In analyzing categories 1, 2 and 4, the Court determined that although some communications could
support or refute Plaintiff’s claims of emotional distress, Defendant had failed to put “a reasonable
person of ordinary intelligence’ on notice of which specific documents or information would be
responsive to the request,” thereby failing Rule 34(b)(1)(A)’s “reasonable particularity” requirement.”
As a result, Defendant failed to make the “‘threshold showing’ that the request at issue is reasonably
calculated to lead to the discovery of admissible evidence.” Indeed, although evidence from social
networking sites is neither privileged nor protected for privacy reasons, the requesting party does not
have “a generalized right to rummage at will through information that [the responding party] has limited
from public view.” Rather, consistent with Fed. R. Civ. P. 26(c), information requests must be “tailored”
so that it “appears reasonably calculated to lead to the discovery of admissible evidence.” Simply
because a communication may, for example, reflect one’s state of mind, that communication is not
necessarily discoverable under Fed. R. Civ. P. 26(c) if it does not relate to the cause of action or
damages at issue.
In contrast, Category 3 was deemed a permissible request because it placed Plaintiff on adequate notice
of the materials sought and was reasonably calculated to lead to the discovery of admissible evidence.
http://www.ediscoverylawalert.com/2012/10/articles/legal-decisions-court-rules/no-fishing-... 4/2/2013
No Fishing Expeditions Allowed When It Comes to Discovery of Social Media : E-Disco... Page 2 of 2
As a practical matter, when seeking social media discovery, attorneys should ensure that the requests are
tailored to the claims at issue. What does that mean? Based upon the reasoning contained in this order, if
you are dealing with a personal injury plaintiff who claims disability and loss of enjoyment, you should
request, for example, any profiles, posts or messages that reflect the plaintiff’s participation in any
travel, sports, recreation or related activities. Simply making a blanket request for all posts will not
suffice under the standard set forth in Mailhoit.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2012/10/articles/legal-decisions-court-rules/no-fishing-... 4/2/2013
NOT FOR PUBLICATION

__________________________________________
)
THE KATIROLL COMPANY, INC., )
)
Plaintiff, )
)
v. ) Civil Action No. 10-3620 (GEB)
)
KATI ROLL AND PLATTERS, INC. AND )
NIRAJ JIVANI, ) MEMORANDUM OPINION
)
Defendants. )
_________________________________________ )
BROWN, Chief Judge
This matter comes before the Court on a motion by Plaintiff for spoliation sanctions and
to compel discovery (Doc. No. 108). The Court has considered the parties’ submissions and
decided the motion without oral argument pursuant to Federal Rule of Civil Procedure 78. For
the reasons set forth below, that motion is denied in part and granted in part.
I. BACKGROUND
This is a trademark infringement case involving two restaurants that sell a similar type of
food called katirolls. Plaintiff originally brought this action on March 3, 2010, in the Southern
District of New York alleging, among other things, infringement of their distinctive trade dress.
(Compl.; Doc. No. 1). The Southern District of New York transferred the case to New Jersey
based on a lack of personal jurisdiction. (Memorandum Order dated July 9, 2010; Doc. No. 31).
Shortly after the transfer, this Court granted a motion for a preliminary injunction after a
two-day evidentiary hearing. After setting forth its findings of fact and conclusions of law, this
1
Court found that Plaintiff was entitled to a preliminary injunction on its trade dress claims but
found that it had not carried its burden on infringement of the trademark. (Doc. Nos. 72, 73).
This motion focuses on the discovery positions of the parties. Particularly, this motion
focuses on whether Defendants’ fact discovery has been late forthcoming and whether
Defendants purposely and deliberately destroyed important discoverable information. Fact
discovery in this case is ongoing and does not close until September 30, 2011. (Doc. No. 110).
II. SPOILATION SANCTIONS AND MOTION TO COMPEL
Parties to litigation often are required to preserve litigation evidence. This duty “arises
when the party in possession of the evidence knows that litigation by the party seeking the
evidence is pending or probable and the party in possession of the evidence can foresee the harm
or prejudice that would be caused to the party seeking the evidence if the evidence were to be
discarded.” Kounelis v. Sherrer, 529 F. Supp. 2d 503, 518 (D.N.J. 2008). If the party does not
so preserve the evidence, they are said to have spoliated that evidence, which can give rise to
sanctions. “In determining whether spoliation sanctions are appropriate, the two key
considerations are the ‘degree of fault of the party who altered or destroyed the evidence’ and
‘the degree of prejudice suffered by the opposing party.’” Id. (quoting Schmidt v. Milwaukee
Elec. Tool Corp., 13 F.3d 76, 79 (3d Cir. 1994)).
Sanctions for spoliation include: “dismissal of a claim or granting judgment in favor of a
prejudiced party; suppression of evidence; an adverse inference, referred to as the spoliation
inference; fines; and attorneys’ fees and costs.” Id. A spoliation inference is the mildest sanction
and “permits a jury to draw an adverse inference that the spoliated evidence might or would have
been unfavorable to the position of the offending party.” Veloso v. Western Bedding Supply Co.,
2
Inc., 281 F.Supp.2d 743, 746 (D.N.J. 2003) (internal quotation marks omitted). To qualify for a
spoliation inference, the movant must show at least four things: “First, it is essential that the
evidence in question be within the party’s control. Second, it must appear that there has been
actual suppression or withholding of the evidence. Third, the evidence destroyed or withheld
was relevant to claims or defenses. And fourth, it was reasonably foreseeable that the evidence
would later be discoverable.” Mosaid Tech., Inc. v. Samsung Elec. Co., Ltd., 348 F.Supp.2d 332,
336 (D.N.J. 2004).
There is a split in this district’s jurisprudence as to the fault required to support the
second factor – the suppression or withholding of the evidence. Some courts have required that
spoliation to be the result of intentional conduct before giving an adverse inference instruction.
Veloso v. Western Bedding Supply Co., 281 F.Supp.2d, 743, 746-49 (D.N.J. 2003); Costello v.
City of Brigantine, No. 99-4072, 2001 WL 732402, at *26 (D.N.J. Jun. 28, 2001). Others have
held that the inference is justified where the party possessing the evidence destroyed it
negligently. Kounelis, 529 F. Supp. 2d at 518 (appropriate where either negligently or
intentionally destroyed, but balancing the degree of fault with the prejudice produced); Mosaid
Techn., Inc. v. Samsung, 348 F.Supp.2d 332, 337-38 (D.N.J. 2004); Scott v. IBM Corp., 196
F.R.D. 233, 249 (D.N.J. 2000) (finding that spoliations might be appropriate where there was
nothing to indicate documents were intentionally destroyed).
This Court concludes that the best rule is to use the amount of prejudice to the opposing
party to help to determine the degree of fault required: Where there is substantial prejudice to
the opposing party, negligence may be sufficient to warrant a spoliation inference. Where there
is minimal prejudice to the opposing party, intentional conduct is required.
3
B. Application
The Plaintiff complains to this Court about a series of discovery abuses that allegedly
require a spoliation inference. However, the Court concludes in its discretion that, as to some
conduct, the prejudice to Plaintiff of the alleged destruction of evidence is minimal and that it
was partially at fault. Thus, no spoliation inference is appropriate. In other instances, the Court
lacks sufficient information to make a determination but is troubled by aspects of Defendants’
failed disclosures. While the Court finds it is premature to enter a spoliation inference in these
instances, the Court is open to spoliation sanctions in the future and grants the motion to compel
in some instances. In still other instances, the Court determines that a spoliation instruction is
warranted.
1. The Appearance of the Restaurant
This Court’s February 1, 2011 order gave Defendants twenty days to change the
infringing color of the restaurant from orange to another color. (Doc. No. 73). Thereafter,
Defendants’ counsel agreed that Defendants would leave the restaurant in its current condition
until February 4, 2011, so that Plaintiff could take video and pictures of the restaurant in its
infringing form. However, Plaintiff complains that the Defendants painted a few test patches of
new colors on the wall prior to the pictures. (Pl.’s Br. at 11).
There is little question that the change in the appearance of the restaurant fits the first,
third and fourth criterion for the spoliation instruction: the appearance was within the
Defendants’ control, it was ultimately changed before evidence could be taken, and Defendants
knew that Plaintiff wanted to take pictures of the restaurant. The only real question is about the
intentionality of the conduct and the prejudice to the Plaintiff.
4
Plaintiff has suffered little prejudice from this action, and Defendants are minimally at
fault for the issue. See Kounelis, 529 F. Supp. 2d at 519. First, Plaintiff has suffered minimal if
any prejudice. Defendants’ counsel represented that the test patches of a gold color were 2 feet
by 1 foot, and were small compared to the size of the restaurant; Plaintiff has submitted no
evidence to contradict this. As such, the jury will not have difficulty imagining the entire
restaurant was orange rather than orange with two small test patches of gold on either the video
or the pictures taken on the day of inspection. This is particularly true in light of Plaintiff’s prior
assertion before this Court that Mr. Jivani’s use of gold is so similar to orange that for certain
lightings and on certain computer screens, it will appear as orange. Further, Plaintiff is in
possession of pictures of the restaurant prior to the change, so the evidence is somewhat
cumulative. (Dolich Decl. at Ex. E).
Second, while Mr. Jivani was at fault for painting prior to the video and pictures, such
fault was minimal because the painting was undertaken to comply with this Court’s preliminary
injunction order. Such conduct was not intentional. Thus, the fault and prejudice do not weigh
in favor of a spoliation sanction and the Court will not order one.
After Plaintiff realized that Mr. Jivani had painted the test patches, Plaintiff requested the
security surveillance video taken prior to the repainting, which the Defendants agreed to provide.
Defendants claim that they were unable to save the video but cite no support for that statement.
(Defs.’ Br. at 11). As the Court acknowledged above, there is minimal prejudice to Plaintiff
because the video that Plaintiff could have taken during the inspection would have been
sufficient. However, the Court is not satisfied with Defendants’ explanation and finds it highly
suspect. Indeed, after having represented that they had the video, Defendants somewhat lamely
state that they were unable to save the video. (Id.). As such, the Court orders Defendants to
5
provide it with a full, sworn explanation of what happened to the surveillance video from each
person involved in its preservation. If the Court is not satisfied with that explanation, it will
conclude that the noncompliance is intentional and grant a spoliation inference.
2. Mr. Jivani’s Old Facebook Pages
Plaintiff asks for a spoliation sanction as a result of Mr. Jivani’s failure to preserve his
Facebook pages in their original state – in other words, Plaintiff wanted PDFs of these pages
prior to their being taken down. (Pl.’s Br. at 5-6). However, Plaintiff does not dispute that
Facebook took these pages down because of its own take-down request. (See Dolich Decl. at Ex.
B). To hold Defendants responsible for this subsection of pages would be unjust. 1
Plaintiff also requests that the Court create an inference of spoliation based upon Mr.
Jivani changing his profile picture on Facebook from a picture displaying the infringing trade
dress without preserving that evidence. Defendants, for their part, claim that Plaintiff is in
possession of all relevant posts prior to the change. Both parties agree that changing the user’s
Facebook profile picture changes the picture associated with each and every post that user has
made in the past.
There is little question that the evidence fulfills the third requirement, but the other
requirements are at issue.
Defendants argue that the Court should not impose a spoliation inference because the
website was public and Plaintiff could have printed the website itself. This seems to be directed
at the first requirement, that the discovery be in Defendants’ control. However, Courts in this
district have still found public websites to be within the control of parties who own them.
Arteria Property Pty Ltd. v. Universal Funding V.T.O., Inc., 2008 WL 4513696 at *5 (D.N.J.
1
The fact that it was not legally improper for Plaintiff to serve this notice (as acknowledged in this Court’s prior
opinion), does not undermine the fact that it was the primary and but-for cause of the spoliation.
6
2008). Further, this is an attempt to “pass the buck” to Plaintiff to print websites that
Defendants are obliged to produce. Given that Defendants have a discovery obligation to
produce them and that only Defendants knew when the website would be changed, it is more
appropriate for Defendants to have that burden. Thus, the Court concludes that the first factor,
Defendants’ control, is present.
The presence of remaining two factors is less than clear. As to the forseeability of the
discoverability of the evidence, the Court notes that the change of a profile picture on Facebook
is a common occurrence. Active users often change their profile pictures weekly. 2 Therefore, it
is hardly surprising that Mr. Jivani has changed his profile picture during the pendency of this
litigation. Further, while Mr. Jivani was on notice that he had to preserve evidence, it would not
have been immediately clear that changing his profile picture would undermine discoverable
evidence. As such, the Court determines that this spoliation was unintentional; however, it
nonetheless concludes that the loss of this evidence is somewhat prejudicial to Plaintiff. See
Mosaid Tech., Inc. v. Samsung Elec. Co., Ltd., 348 F.Supp.2d at 336.
Given these considerations of fault, the Court determines that a less imposing alternative
is the best solution. Mr. Jivani must to coordinate with Plaintiff’s counsel to change the picture
back to the allegedly infringing picture 3 for a brief time so that Plaintiff may print whatever posts
it thinks are relevant – such action shall not be considered an additional act of infringement. The
Court however, determines that it is incumbent on Plaintiff to, during the appointed time, print
2
Further, the Court notes that it may even be somewhat misleading to have printouts of Defendant’s posts with the
infringing picture, as if they always appeared that way. Indeed, it is likely that many posts existed for a substantial
portion of their lifetime without the allegedly infringing photograph. This is an issue that can be resolved in the
proper questioning of witnesses.
3
These photographs have not been destroyed, they been attached in several PDFs to this Court. It would be a simple
process to recover the photograph from those PDFs. Then, Plaintiff may print the posts it thinks are relevant to the
case.
7
such posts as it thinks are necessary to make its case. Thereafter, Mr. Jivani must immediately
change his profile picture back to his noninfringing picture.
3. Documents from Defendants’ Website
Plaintiff complains that it does not have a copy of the Defendants’ website showing what
the restaurant looked like prior to its repainting. Defendants respond that Plaintiff is in
possession of pictures showing the restaurant prior to repainting, which are sufficient for Plaintiff
to make its case. Again, the first, third, and fourth elements of the test are met. The question
revolves around the second element, whether such evidence was destroyed intentionally or
negligently, and what prejudice accrued to Plaintiff as a result.
Essentially, this is Plaintiff’s complaint about Defendants’ repainting prior to the
inspection again. As discussed, Plaintiff had the opportunity to take pictures and video of the
restaurant with a minor change. The Court does not know whether Plaintiff took advantage of
that opportunity but, as discussed, the minor change in appearance was not very prejudicial to
Plaintiff (particularly in light of the fact that Plaintiff has maintained that the gold color of the
test patches is similar to the original orange). Consequently, Plaintiff suffered minimal
prejudice.
However, the Court concludes that it is implausible that Mr. Jivani has no other electronic
copies of these pictures on his hard drive or in emails to his web developer, etc. As such, Mr.
Jivani is directed to search his home computer and any other computers he frequently uses and
disclose any pictures he has of the restaurant prior to its repainting. The Court directs Defense
counsel to remind Mr. Jivani of the consequences of destroying evidence before a tribunal.
8
The Court also determines that, without the printout of the website, the evidence will be
insufficient to show that the website itself embodied pictures of the infringing restaurant and
substantial prejudice has accrued to Plaintiff’s ability to show that website itself infringed its
trade dress. As such, on this limited issue, the Court will grant a spoliation instruction unless
Defendants produce an image of the portion of the website that included the photographs within
one week of the order accompanying this opinion.
4. Annual Report
Plaintiff claims that a spoliation inference should result from Defendants’ failure to
provide its 2010 annual report. Plaintiff requested the report to confirm Defendants’
representation that Rasik Jivani was no longer a shareholder of the corporation because he had
transferred the stock to Niraj Jivani. Defendants claimed that they were unable to produce the
report because it was no longer in their possession, despite the fact that they filed it during the
pendency of this litigation. (See Doc. No. 59, Ex. A, p. 1). Defendants did provide documents
related to the transfer of stock and stock certificates and corporate bylaws. (Defs.’ Br. at 12).
It does appear that Defendants were at the very least grossly negligent in failing to keep a
copy of a report that they agreed to provide to opposing counsel and the Court. Mosaid Tech.,
348 F.Supp.2d at 336 ; (See Doc. No. 57-1, ¶3). However, apparently this is a public document
available from the New Jersey Secretary of State. As a result, the Court cannot see any prejudice
that accrues from the failure to produce it – it is still available to both parties.
Indeed, the Court observes that it would have been easier for Plaintiff to request the filing
than to draft this portion of the motion. The Court further observes that it would have been
easier for Defendants to request the filing than to respond to this portion of the motion. Given
the lack of prejudice to Plaintiff, the Court finds that a spoliation sanction is inappropriate,
9
despite Defendants’ fault in not preserving their copy of the report. See Kounelis, 529 F. Supp.
2d at 519.
However, counsel for Defendants is directed to request the report from the Secretary of
State and provide it to Plaintiff if it has not already so provided.
5. Emails Between Mr. Jivani and the Web Developer
According to Mr. Jivani’s testimony, in October of 2011, Mr. Jivani hired a web
developer for his website. Mr. Jivani, in failing to comply with this Court’s preliminary
injunction order, alleged that he was unable to contact the developer to put the disclaimer on the
website and change the color. (4/4/11 tr. at 12-13). Plaintiff requested the emails between Mr.
Jivani and his web developer to confirm that he did indeed hire the developer as of October.
(Pl’s. Br. at 14). Plaintiff asserts that Defendants never produced the emails. Defendants claim
that all of the emails in their possession were produced and that all other communication took
place in person or by telephone.
The emails produced consist of one exchange, where, on February 8, 2011, Mr. Jivani’s
web developer sent him a copy of the image of his home page prior to adding the disclaimer:
Hi Niraj,
As per your phone request, Please see your website’s home page’s html file, in
the attachment. (Before adding the disclaimer part in the ‘Home Page’).
Best Regards,
Dr. Hema, Phd.

Nadiyam Technologies, Inc.
(Dolich Decl. at Ex. G) (emphasis added). Mr. Jivani did not reply to the substance of this
email, but replied on a related topic on March 16, 2011 (and immediately forwarded the reply
email to counsel):
10
Hema I have called you on March 3rd and 4th to change the colors[.] I am on a
court order [as] I have told you before . . . you MUST change the color[.]
[T]his is horrible service being a few weeks that have gone by and no action
has been taken to change the colors!
(Id.) (ellipsis in original, emphasis added).
There is no doubt that Dr. Hema Latha’s email and Niraj Jivani’s email reflect that
telephone conversations had taken place between the two of them. As such, if the emails have
not been altered or somehow fabricated, it is clear that some communication took place by
telephone. Therefore, there may be no emails to be had.
However, there may be emails that were not produced or deleted. It seems suspicious
that a person with as active an online persona as Mr. Jivani only ever sent one email to his web
developer, an email happens to cast him in a positive light. Indeed, the Court wonders how Dr.
Latha came into possession of the photographs that were allegedly posted on the website if there
were no emails or other electronic communication between the two. 4 Further, there are relatively
easy ways to confirm these questions that do not require a spoliation sanction at this juncture.
In order to confirm whether Mr. Jivani sent other emails, the Court directs Defendants as
follows:
(1) Mr. Jivani shall allow his counsel to conduct a search of each of his email
accounts, particularly “katirollandplatters@gmail.com” for emails from his
web developer; defense counsel shall conduct such a search and turn over any
emails procured; 5
(2) Mr. Jivani will request, via a signed request communicated through
counsel, that Dr. Hema Latha forward to defense counsel directly, any emails
that Mr. Jivani and Dr. Latha have exchanged, assuming he has not deleted
them;
4
It would have been possible to transfer such things in person. It is also possible that the pictures were on the
website prior to Dr. Latha starting his work with the website. However, both of these possibilities seem less likely
than an email.
5
See Local Civil Rule 26.1(d)(1). The Court also notes that “archive” is the primary email management function on
“Gmail.” As such, it is unusual though not unheard of, for users to actually delete emails.
11
(3) Mr. Jivani shall have no contact, except through counsel, with Dr. Latha
until such time as Dr. Latha has responded to the request for emails;
(4) Mr. Jivani, with the assistance of counsel, is to request phone records for
each phone on which he made these calls for the months of October, 2010, and
February and March of 2011 to confirm Mr. Jivani’s alleged calls to Dr. Latha,
to be delivered, if possible, directly to his counsel; such records shall be
produced to the Court with appropriate redactions of calls not to Dr. Latha.
Counsel for the Defendants shall report every week to Judge Bongiovanni by a brief letter
submission on the progress of these investigations and file, as an attachment, any resulting
emails or phone records, appropriately redacted or under seal. Counsel may choose the day that
he will consistently place such a letter on the docket. If no emails are recovered and Mr. Jivani
is unable to produce records of phone calls or text messages to Dr. Latha (or alleges that such
phone calls were made on the phones of any third parties or pay phones), the Court will consider
sanctions against him. As of now, the existence of such emails is too hypothetical to impose
sanctions.
7. Defendants’ New Facebook Page
Plaintiff argues that Defendants made a new Facebook page and did not disclose its
existence in a discovery response. The Court agrees that Defendants should have disclosed its
existence, but because Plaintiff has found the page independently and Defendants created the
page in order to comply with this Court’s preliminary injunction, sanctions are not appropriate.
See Kounelis, 529 F. Supp. 2d at 519 (prejudice to discovering party considered when imposing
sanctions).
8. Financial Documents
Plaintiff requested that Defendants produce restaurant sales reports as well as other
financial data. Plaintiff does not complain about the production of many of these items.
12
However, Plaintiff does complain that it has not received the point of sale reports, hand logs for
cash transactions, that the corporation’s tax returns were unsigned, the corporation’s bank
account statements redacted several ATM transactions (which may be relevant for piercing
analysis), and that Defendants did not produce a copy of their insurance policy.
Defendants respond that the proprietary point of sale software has created problems for
such production, that counsel is investigating the redaction to the bank account statements, and
that they did produce the copy of the insurance policy. Defendants make no representations
about the hand logs or tax returns. Nor do Defendants argue that the requests are in any way
overbroad.
The Court finds that these are run-of-the-mill discovery issues, on which the parties
should confer to resolve. The Court does not believe a sanction or order compelling production
is appropriate at this juncture. There has yet to be a litigation in which every document was
produced in a timely fashion. The Court also sees no reason why Defendants must produce
electronic records of their bank accounts to Plaintiff or need bear the cost of putting the data into
a spreadsheet. Defendants’ counsel shall address each of these discovery issues in his weekly
report until they are resolved.
9. Deposition of Niraj Jivani
While both parties agree that Plaintiff will have another opportunity to depose Niraj
Jivani, Plaintiff argues that it should be compensated for the costs of the additional deposition.
Plaintiff suggests that this is justified because Defendants were late in the production of several
items, including their insurance policy, prior to Mr. Jivani’s deposition. The Court disagrees.
As suggested by Defense counsel, the need for a supplemental deposition was also
necessitated by Plaintiff’s insistence on taking Mr. Jivani’s deposition only a few weeks after
13
new counsel arrived on the case, without the opportunity for that counsel to come up to speed
with the discovery requested. Further, it appears that Plaintiff has four more hours of questions,
not all of which can be related to the insurance policy. As such, Plaintiff would have had to
schedule an additional day of deposition anyway. Thus, the Court awards no fees for the
additional deposition.
10. Counsel Are Directed to Conduct Themselves with the Collegiality and
Good Faith that is Befitting of New Jersey Attorneys
The Court reminds counsel that attorneys in this jurisdiction are expected to conduct
themselves with professionalism, to creatively find solutions to common problems, and to
cooperate with each other. The email and other correspondence that has appeared before the
Court do not reflect these high standards. Emails accusing opposing counsel of ethical
violations, resistance to simple requests, attempts to pass responsibility to the other party for
simple discovery requests, and inflexibility are reflected in many recent communications in this
case. (See, e.g., Dolich Ex. I; Doc. No. 114; Doc. No. 117; Doc No. 108-2; Doc. No. 118; Doc.
129 Ex. 2). The Court also reminds the parties that their attempts to pass the burden of discovery
to each other often result in more work in emails than the work it would take to obtain the
document themselves.
The Court understands that there is animosity between the parties. The testimony at the
preliminary injunction hearing was enough to make that abundantly clear. However, despite the
animosity between their clients, attorneys before this Court are expected to confer in good faith
prior to bringing issues before the Court.
14
III. CONCLUSION
For the foregoing reasons, the Court grants in part and denies in part the request for
spoliation sanctions and grants certain requests for the production of documents. The Court files
an appropriate order concurrent with this opinion.
Dated: August 3, 2011
/s/ Garrett E. Brown, Jr.

GARRETT E. BROWN, JR., U.S.D.J.
15
Chief Judge Finds That Alteration of Facebook Page Can Lead to Spoliation Inference : E... Page 1 of 2

Posted at 9:40 AM on December 16, 2011 by Gibbons P.C.
Chief Judge Finds That Alteration of Facebook Page Can Lead to Spoliation Inference
In a trademark infringement case involving two restaurants, Katiroll Company, Inc. v. Kati Roll and
Platters, Inc. et al., Plaintiff sought a spoliation inference, alleging various discovery abuses involving
several types of evidence including social media. Specifically, Plaintiff requested sanctions for the
individual Defendant’s failure to preserve his Facebook pages in two different ways. Recognizing that
Facebook users change their pages frequently given the nature of the media at issue, Chief Judge Brown
of the District of New Jersey crafted a creative remedy, which was based in large part on the level of
prejudice to Plaintiff.
Regarding the first Facebook issue, Plaintiff sought PDF versions of Defendant's Facebook pages before
they were taken down pursuant to Plaintiff’s take-down request. The Court declined to sanction
Defendants for actions taken at Plaintiff’s request because it would be "unjust."
Plaintiff also sought a spoliation inference because the individual Defendant altered his profile picture
on Facebook. The prior picture reflected the infringing trade dress of the restaurant at issue but was not
preserved. The Court recited the four requirements for a spoliation inference, which the Chief Judge
described as the mildest of sanctions: (1) whether the evidence was in the party’s control; (2) whether
the evidence was actually suppressed or withheld; (3) whether the evidence was relevant vis-à-vis the
claims or defenses at issue; and (4) whether it was reasonably foreseeable that the evidence at issue
would subsequently be discoverable. The Court concluded that the most important consideration in
determining what level of fault is required to support the second factor (an issue that is disputed within
the District) is the degree of prejudice to the movant. Specifically, the Court concluded that a negligence
standard may be appropriate if there was substantial prejudice whereas intentional conduct would be
required if minimal prejudice resulted.
The Court had little difficulty finding that Plaintiff met the third requirement. It similarly found that the
first requirement was met because "defendants have a discovery obligation to produce [websites]” and
“only defendants knew whether the website would be changed." As for the second and fourth
requirements, the Court found that, although the individual knew he had to preserve evidence, “it would
not have been immediately clear that changing his profile would undermine discoverable evidence"
given that a change of a profile picture "changes the picture associated with each and every post that
user has made in the past.” Although the Court found that the spoliation was not intentional, the Court
also determined that the loss of information was “somewhat prejudicial.” But instead of imposing an
adverse inference instruction (as Plaintiff had requested), the Court directed Defendant to post the image
of the allegedly infringing picture for a short period of time so that Plaintiff could then print relevant
posts; thereafter, Defendant was permitted to re-post the non-infringing picture.
http://www.ediscoverylawalert.com/2011/12/articles/legal-decisions-court-rules/chief-judge... 4/1/2013
Chief Judge Finds That Alteration of Facebook Page Can Lead to Spoliation Inference : E... Page 2 of 2
Katiroll demonstrates the Court’s understanding that the nature of Facebook invites changes to postings
and pictures, and it teaches that the prejudice to the movant is a significant factor in determining the
level of fault required before an adverse inference instruction will be imposed. Katiroll also
demonstrates that courts will fashion novel remedies for spoliation in the social media context.
Mara E. Zazzali-Hogan is a Director on the Gibbons E -Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2011/12/articles/legal-decisions-court-rules/chief-judge... 4/1/2013
Case 2:09-cr-00932-JLL Document 61-1 Filed 10/21/10 Page 1 of 20 PageID: 997
NOT FOR PUBLICATION

UNITED STATES OF AMERICA, Criminal Action No.: 09-932 (JLL)
Plaintiff,
OPINION
v.
ANTHONY SUAREZ and VINCENT

TABBACHINO,
Defendants.
LINARES, District Judge.
Currently before this Court is a Motion filed by Defendants requesting, inter alia, that as a
result of the Government's failure to produce certain electronic text messages related to its
investigation of Defendants the Court either suppress related evidence or issue an adverse
inference instruction. The Court conducted hearings and heard counsel’s arguments regarding
the “missing” text messages on July 27, 2010, September 29, 2010, and October 8, 12, and 15,
2010.
I. Background
In 2008, federal law enforcement authorities began using Solomon Dwek as a cooperating
witness to investigate public corruption within the state of New Jersey. The cooperating witness
held himself out to be real estate developer David Essenbach and claimed to be interested in
developing the greater Hudson and Bergen County, New Jersey, areas. Agents within the Federal
Page 1 of 20
Bureau of Investigation (F.B.I.) instructed the cooperating witness to express interest in such
development to local public officials and offer these officials payments in exchange for
expediting his projects and providing other official assistance. During the period from March to
July, 2009, the cooperating witness, posing as developer David Essenbach, spoke and met with
the defendants herein on multiple occasions. During the relevant time period of the investigation
pertaining to the defendants herein and during meetings with said defendants, the cooperating
witness exchanged numerous Short Message Service electronic communications (“text
messages”) with F.B.I. Agents William Waldie, Donald Russ, and Sean McCarthy.
In December 2009, Defendants made their first request for discovery of the text messages
exchanged between the cooperating witness and the F.B.I. agents. Receiving no response,
Defendants moved on June 16, 2010 for the Government to produce the text messages. The
Court held oral argument on July 27, 2010. At oral argument, the Government indicated that it
would produce the text messages. (Oral Argument Tr. 26:22-27:4, July 27, 2010.)
Subsequently, in a letter to Defendants dated August 31, 2010, the Government stated
that it was unable to produce any of the text messages sent by the cooperating witness to the
agents because the witness had used a personal cellular telephone, and his wireless carrier
“retained text messages for a period of only 3 to 5 days from the date of creation.” Defs.’ Mot. in
Limine Ex. D. The Government further stated that it could not produce text messages sent by
Agents Waldie, Russ, or McCarthy to the cooperating witness during the time period of March 1,
2009 through March 16, 2009, nor could it produce any text messages sent by Agents Russ or
McCarthy during the entire month of July 2009. Id. The Government explained that this was
because “the FBI retained text messages of its agents only for so long as limited storage space on
Page 2 of 20
its servers allowed.” Id.
The following month, however, the Government did in fact produce text messages from
Agent Waldie for the time period from March 1 through 16, 2009. The Government still did not
produce any text messages from Agents Russ or McCarthy for either of the relevant time periods.
On September 21, 2010, Defendants made the present Motion, and oral argument was held on
September 29. At oral argument, the Government confirmed that it had produced text messages
from other time periods, including May 2009, but represented to the Court that no text messages
existed for Agents Russ or McCarthy for the March and July time periods. The Government
offered the following explanation:
THE GOVERNMENT: We do not believe -- have no reason to believe that any text
messages are missing. We made a misstatement in our view to counsel that in fact the
period of time in our review of the [text messages] show that there were no [text
messages] in March for Agents Waldie, Russ and McCarthy. We have since found Agent
Waldie’s and produced to the defendants all of Agent Waldie’s texts for March. We also
stated that in July, there was a period of July where we thought there were no [text
messages] for McCarthy and Donny Russ, and we learned at least for some periods during
that time, both were on --
THE COURT: McCarthy and Russ were on vacation.
THE GOVERNMENT: -- correct.
***
THE COURT: Well, what is your representation to this Court regarding the March text
messages?
THE GOVERNMENT: We do not believe that any text messages are missing. We have
no reason to believe that.
(Oral Argument Tr. 94:22-96:4, Sept. 29, 2010.)
After oral argument, the Government notified the Court that these statements were made
Page 3 of 20
in error. The Government now maintains that Agent Russ and Agent McCarthy were not on
vacation during July 2009 and that text messages were in fact sent by those agents to the
cooperating witness during the relevant time periods.
II. Text Message Hearings
As a result of the lack of information regarding the exact cause and circumstances
surrounding the retention and deletion of the aforesaid messages, the Court ordered a hearing.
Said hearing was held on October 8, 12, and 15, 2010. The three F.B.I. agents who
communicated with the cooperating witness over text message, Agents Waldie, Russ, and
McCarthy, each testified. The agents were asked to describe the contents of the text messages
and to indicate whether they had personally attempted to preserve the messages. Each agent
testified that he had deleted the messages from his handheld Blackberry® device. One of the
reasons given was a fear of overloading the device’s local persistent memory. (Trial Tr. 6.12:2-
16, 6.52:2-12, 6.100:16-6.101:17, Oct. 8, 2010 [hereinafter “Hr’g. Tr.”].) Each agent testified
that he was never instructed to preserve text messages. (Hr’g. Tr. 6.16:3-6, 6.45:22-6.55:5,
6.102:11-20.) The agents also testified that on occasion they did give instructions to the
cooperating witness via text message and that the witness did relay to them the witness’s
interpretation of what was said during the meetings. (Hr’g. Tr. 6.25:15-6.26:3.) The agents,
however, testified that said interpretation was irrelevant because the meetings were audio and
video recorded. (Hr’g. Tr. 6:14:12-22, 6.50:12-6.51:22.)
The first witness, Agent Waldie, testified as to the conditions under which he would
document events or conversations that occurred during a meeting that was being recorded by a
cooperating witness: “If it was something that couldn't be heard or seen, yes. But if it was
Page 4 of 20
something that was said, no. The tape would capture that.” (Hr’g. Tr. 6.14:17-22.) As Agent
Waldie finished his testimony, an Assistant United States Attorney left the courtroom and spoke
with Agent Russ, who had been sequestered, as he had not yet testified, and was waiting in
hallway. An attorney for Defendants followed the Government attorney into the hallway. When
questioned by the Court about the substance of the conversation, both the Assistant United States
Attorney and Agent Russ explained that the attorney was merely attempting to remind Agent
Russ of what they had previously discussed during witness preparation, specifically that the
meetings between the cooperating witness and Defendants had been recorded and that any
conversations described in any text messages would be preserved in the recording. (Hr’g. Tr.
6.42:22-24, 6.64:7-20.) This was in violation of a Sequestration Order entered by this Court.
The Court takes violations of its Orders very seriously, and although the conversation with Agent
Russ appears to have pertained to a matter of small importance in the context of the hearing, the
Court nevertheless admonished the attorney in question and made it clear that the Court was
troubled by his actions.
After the F.B.I. agents testified, the Court heard testimony from three F.B.I. information
technology specialists regarding the procedures that the agency uses to preserve and retrieve data
generated by the agents’ handheld devices. Rodrick Summers, the F.B.I.’s Information
Technology Unit Chief, testified as to how the F.B.I. preserves text messages sent by its agents.
He stated that when an agent sends a text message from his government-issued handheld device,
the message is first transmitted to a commercial wireless carrier via the carrier’s cellular tower
infrastructure. (Trial Tr. 7.12:17-7.14:19, Oct. 12, 2010 [hereinafter “Hr’g. Tr.”].) Then, the
commercial carrier routes one copy of the message to the designated recipient (again, via the
Page 5 of 20
commercial cellular tower infrastructure) and routes another copy of the message to a server
maintained by Research in Motion Limited (RIM). (Hr’g. Tr. 7.8:14-23, 7.15:2-7.16:23.) The
RIM server then transmits the message to one of a number of Blackberry® Enterprise Servers
(BES) maintained by the F.B.I. at its Clarksburg, West Virginia, facility. (Id. at 7.9:10-12.)
During the time period relevant to this case, the F.B.I. maintained fourteen BES servers. (Id. at
7.9:7-9.) After a text message leaves an agent’s handheld device and until it reaches a BES
server, the message is handled entirely by non-government entities. (Id. at 7.15:24-7.16:2.)
Mr. Summers testified that the BES servers contain only a few days of text message data
at any given time, but they are backed up to a “log server” on a daily basis. (Id. at 7.23:3-4.) The
BES server data is also backed up daily to a series of “backup tapes.” (Id. at 7.25:1-9.) The log
server and backup tapes are all located in Clarksburg, West Virginia. (Id. at 7.11:13-15.) Mr.
Summers testified that absent a litigation hold, F.B.I. policy permits that after a period of 90 days
the backup tapes may be overwritten or destroyed, and log server data may be deleted. (Id. at
7.25:1-19.) No litigation hold was in place during the time period in which the missing text
messages were sent, nor was a litigation hold in place during the 90 day period following that
time. The Government first ordered a litigation hold on or about January 11, 2010. (Id. at
7.39:9-13.)
At the hearing, the Government produced an F.B.I. Corporate Policy Directive that
provided both the data retention policy of the F.B.I. and its policy regarding litigation holds.
(Text Message Hr’g. Ex. 3 [hereinafter “Hr’g. Ex.”].) This document contains the 90 day
retention period described in Mr. Summers’ testimony and further indicates that “[l]itigation
holds or other special inquiries are the only exceptions to this data backup policy. In these
Page 6 of 20
instances, backups must be maintained until resolution of the litigation or other inquiry.” (Id.)
The document further indicates that this exception extends to “reasonably anticipated or pending
litigation.” (Id.)
Brian Modest, a security engineer contracted by the F.B.I., described his efforts during
October through December 2009 to retrieve from the F.B.I. servers the text messages from the
missing periods described above. (Hr’g. Tr. 6.131:7-11.) Mr. Modest testified that he used a
computer tool to transfer data from the log servers onto his own database, from which he
conducted keyword searches. (Id. at 6.124:6-6.125:23.) Mr. Modest testified that he was unable
to locate the missing text messages, even through a manual search. (Id. at 6.141:3-14.)
Justin Grover, a information security engineer with the F.B.I.’s Enterprise Security
Operations Center (ESOC), testified about an audit performed by ESOC staff in an effort to
locate the missing text messages. (Id. at 6.149:16-25.) Mr. Grover indicated that he personally
performed the same audit again on October 1, 2010. (Id. at 6.150:15-21.) Mr. Grover testified
that ESOC maintains a separate database of text message data from the ones described by the
other witnesses. On a daily basis, ESOC transfers text message data from the BES servers
through an intermediary server and onto ESOC’s “classified analytical network.” (Id. at
6.147:21-6.148:1.) The classified analytical network maintains text message data back to
November 2007. (Id. at 6.147:7-18.) Mr. Grover testified that if information never resided on
the BES servers, it would not be found on the classified analytical network maintained by ESOC.
(Id. at 6.125:5-8.) None of the witnesses were able to explain why some text messages from the
same time period were preserved while others were apparently deleted. Likewise no reasonable
explanation was offered as to why there was a need to place a litigation hold on text message data
Page 7 of 20
in January of 2010, yet no need to do so prior to that time.
III. Discussion
The Government has no “constitutional duty routinely to deliver his entire file to defense
counsel.” United States v. Agurs, 427 U.S. 97, 111 (1976). In contrast to the breadth of
discovery permitted in civil cases, Rule 16 of the Federal Rules of Criminal Procedure
“delineates the categories of information to which defendants are entitled in pretrial discovery in
criminal cases, with some additional material being discoverable in accordance with statutory
pronouncements and the due process clause of the Constitution.” United States. v. Ramos, 27
F.3d 65, 68 (3d Cir. 1994). In general, this “additional material” is limited to material
discoverable under the Jencks Act and under Brady v. Maryland, 373 U.S. 83 (1963). Id.
Therefore, before deciding whether sanctions are warranted for the Government’s failure to
produce the missing text messages, the Court must determine whether those text messages fall
under Rule 16 or whether they constitute Jencks or Brady material.
A. Whether the Text Messages are Discoverable
Federal Rule of Criminal Procedure 16(a) outlines the Government’s basic disclosure
requirements. Most pertinent to the facts of this case is Rule 16(a)(1)(E), which provides that
“[u]pon a defendant’s request, the government must permit the defendant to inspect and to copy
or photograph books, papers, documents, data . . ., if the item is within the government’s
possession, custody, or control and (i) the item is material to preparing the defense; (ii) the
government intends to use the item in its case-in-chief at trial; or (iii) the item was obtained from
or belongs to the defendant.” Fed. R. Crim. P. 16(a)(1)(E). However, Rule 16(a)(2) clarifies that
this rule does not “authorize the discovery or inspection of statements made by prospective
Page 8 of 20
government witnesses except as provided in 18 U.S.C. § 3500 [the Jencks Act].” Fed. R. Crim.
P. 16(a)(2). Since the text messages at issue in this case were sent between F.B.I. agents and the
cooperating witness, they were statements made by “prospective government witnesses.” The
Court must therefore determine whether production of the text messages is required by the
Jencks Act.
The Jencks Act and Federal Rule of Criminal Procedure 26.21 require the Government to
disclose prior recorded statements of its witnesses after they testify. 18 U.S.C. § 3500(b); Fed. R.
Crim. P. 26.2. The purpose of the Jencks Act and Rule 26.2 is to permit defense counsel to have
the materials necessary to impeach a Government witness. United States v. Rosa, 891 F.2d 1074,
1076-77 (3d Cir.1989) (citing Jencks v. United States, 353 U.S. 657, 667, 77 (1957)). Rule 26.2
provides in relevant part:
After a witness other than the defendant has testified on direct examination, the court, on
motion of a party who did not call the witness, must order an attorney for the government
or the defendant and the defendant's attorney to produce, for the examination and use of
the moving party, any statement of the witness that is in their possession and that relates
to the subject matter of the witness's testimony.
Fed. R. Crim. P. 26.2(a). A “statement” by a witness includes any “written statement that the
witness makes and signs, or otherwise adopts or approves.” Fed. R. Crim. P. 26.2(f)(1).
In the context of the destruction of evidence by F.B.I. agents, the Third Circuit has held
that “the rough interview notes of F.B.I. agents should be kept and produced so that the trial court
can determine whether the notes should be made available to the appellant under the rule of
1
The Jencks Act and Federal Rule of Criminal Procedure 26.2 “can be used
interchangeably because Rule 26.2 incorporates the relevant provisions of the Jencks Act without
any substantive change.” United States v. Heilman, 377 Fed. Appx. 157, 195 (3d Cir. 2010)
(internal quotation omitted).
Page 9 of 20
[Brady], or the Jencks Act.” United States v. Vella, 562 F.2d 275, 276 (3d Cir. 1977). The
appellate court held that while the F.B.I.’s destruction of the interview notes amounted to only
harmless error, the Government nonetheless had a duty to preserve the documents in case they
did in fact constitute Jencks material. Id. The Third Circuit has affirmed this rule and extended
it to cover draft reports by agents. See United States v. Ammar, 714 F.2d 238, 259 (3d Cir.
1983) (“the government must retain and, upon motion, make available to the district court both
the rough notes and the drafts of reports of its agents to facilitate the district court's determination
whether they should be produced”).
Here the Court is presented with a similar quandary as in Vella and Ammar. When a
document is destroyed or otherwise unavailable, the court is left to speculate whether its contents
constituted a “statement” relevant “to the subject matter of the witness's testimony” and whether
the document was ever even in the “possession” of the Government. But, regardless of the
content of the text messages, the Court can at least be certain here that the agents and the
cooperating witness each adopted and approved each message they sent when they affirmatively
chose to transmit it. Thus, the Court deems the text messages “statements” under the Jencks Act.
As to the content of the text messages sent between the agents and the cooperating
witness, Agent Waldie testified that while his communications typically concerned the logistics
of the cooperating witness’s meetings with Defendants, he also gave “instructions” to the
witness. (Hr’g. Tr. 6.25:15-6.26:3.) The agent testified that he encouraged the witness to be
“blatant” and to “dirty up the money,” meaning that the agent encouraged the witness to make it
clear to Defendants that he intended to offer them a bribe. (Id. at 6.25:20, 627:21.) While each
agent maintained that none of the cooperating witness’s text messages to them contained any
Page 10 of 20
information that he believed to be exculpatory, what the cooperating witness may have said to the
agents during the crucial meetings and whether he abided by the agents’ instructions are fertile
ground for cross examination. (Id. at 6.11:19-23, 6.49:20-6.50:3, 6.99:22-24.) For example, the
cooperating witness testified at trial that “[i]t was very infrequent, if at all, that [he] actually
texted an agent” and that he would typically only send text messages to agents when “safety
issues” arose or “before the payment went down.” (Trial Tr. 6.161:14-6.162:20.) This testimony
is belied by telephone records produced at the text message hearing, and the cooperating witness
could have been further impeached on cross examination by recitation of the cooperating
witness’s actual text messages, had such messages been preserved. (Hr’g. Ex. 1, 2.) The need
for effective cross examination of Government witnesses lies at the heart of the Jencks Act and
echoes the concerns expressed in Vella and Ammar. Thus, the content of the text messages sent
by the cooperating witness to the agents, and the agents’ response or instructions thereto, relate
“to the subject matter of the witness's testimony.”
Informational technology specialists from the F.B.I. testified regarding two theories as to
why the missing text messages between the agents and the cooperating witness were not
recovered. Mr. Summers, the Information Technology Unit Chief, offered the following two
theories: 1) that the messages were deleted from the F.B.I. servers after 90 days pursuant to the
agency’s destruction policy, or 2) that the messages never reached the F.B.I. servers. (Hr’g. Tr.
7.37:19-7.38:21.) Thus, under the first theory, the F.B.I. was in “possession” of the messages
both while they existed on the agents’ handheld devices and for the approximately 90 days that
they resided on the F.B.I. servers. Under the second theory, the F.B.I. was only in “possession”
of the messages while they existed on the agents’ handheld devices. In both scenarios, the
Page 11 of 20
messages existed on the F.B.I. handheld devices until the agents manually deleted them. Each
agent testified that he intentionally deleted the messages in order to free space in the phone’s
memory and not pursuant to any destruction policy. Indeed, neither the agents nor the specialists
in charge of maintaining the F.B.I. servers were under any sort of directive to maintain the text
messages in anticipation that criminal charges would be filed against Defendants.
The F.B.I.’s own “Corporate Policy Directive” in fact anticipates this sort of “litigation
hold” on its document destruction policy. It states that the F.B.I. policy that a “full data backup
will be retained for 90 days” does not apply “in cases of reasonably anticipated or pending
litigation.” (Hr’g. Ex. 3.) Thus, the F.B.I. appears to have been well-equipped to preserve
documents relevant to this litigation had the United States Attorneys’ Office requested it do so.
This is especially important in the context of the case at bar, where the investigation was
undertaken as a joint operation by the New Jersey United States Attorneys’ Office and the F.B.I.,
and the United States Attorney was aware of the importance of preserving documents relevant to
the litigation and could have requested a litigation hold on the text messages from the inception
of the investigation.
That request, however, did not come until January of 2010–seven months after the
investigation concluded and three months after the F.B.I. began searching its servers for the
missing text messages. Without an appropriately timed litigation hold, the Court is left to
speculate as to the contents of the missing text messages and to entertain various theories as to
the cause of their disappearance. However, it is without dispute that each missing text message
at one time resided on an F.B.I. agent’s handheld device. It is also clear that the United States
Attorneys supervising the investigation were aware that text messaging was being used by the
Page 12 of 20
cooperating witness, as such use was evident on the videos. These text messages were
indisputably in the Government’s “possession” at one time. Thus, given the F.B.I.’s analogous
preservation duty under Vella and Ammar and the failure of the Government to preserve relevant
data in the midst of an ongoing investigation specifically aimed at prosecution, and thus where
litigation was reasonably anticipated, the Government had a duty to preserve the Jencks material
contained in the text messages.
B. Appropriate Sanction for the Destruction of Jencks Material
Rule 26.2(e) provides that “[i]f a party who called a witness disobeys an order to produce
or deliver [Jencks material], the court must strike the witness’s testimony from the record,” and
further “[i]f an attorney for the government disobeys the order, the court must declare a mistrial if
justice so requires.” Fed. R. Crim. P. 26.2(e). The Rule thus specifies potentially severe
sanctions for disobedience of a court order to produce Jencks material. The Rule does not,
however, mandate any particular sanction against parties who destroy Jencks material outside the
presence of a court order.
Defendants first request that the testimony of the cooperating witness and all tape
recordings in which he is a party be suppressed. In United States v. Ramos, the Third Circuit
addressed whether the destruction of rough interview notes, as in Vella, warranted suppression of
related evidence. The Court reasoned that “the mere fact that Vella and Ammar each established
rules for the government to follow does not suggest that we intended the automatic suppression
of evidence when those rules are violated.” 27 F.3d at 69. Instead, the Court applied a “good
faith” test to determine whether suppression is warranted under the Jencks Act. Id. (citing
Arizona v. Youngblood, 488 U.S. 51, 109 (1988)). Here, Defendants have presented little
Page 13 of 20
evidence, and the Court does not find, that the Government lost or destroyed the text messages in
bad faith or that the text messages clearly contained exculpatory evidence. Accordingly,
suppression of evidence related to the cooperating witness is not warranted, and to the extent that
Defendants move for such sanctions such request is denied.
Defendants alternatively request that the Court give an adverse inference instruction to
the jury regarding the missing text messages. Since the facts of this case lie outside of the
mandates of Rule 26.2(e), the Court has discretion to apply alternative sanctions. See United
States v. Jackson, 649 F.2d 967, 972 n.6 (3d Cir. 1981) (“it has been held that the statutory
alternatives of striking the witness’ testimony or declaring a mistrial are not mandatory, and
whether to follow a different course rests with the discretion of the trial judge”); United States v.
Ferber, 1991 WL 16751, at *6 (E.D. Pa. 1991) (“where the Jencks Act is violated through
oversight or negligence not amounting to a conscious election, the Court is entitled to apply such
a remedy as the justice of the case might require under the circumstances”) (internal quotations
omitted). Given the relatively little criminal case law in the Third Circuit regarding the
application of adverse inference instructions based on spoliation of evidence,2 the Court consults
the more thoroughly developed civil case law on the subject. See United States v. Bunty, 617 F.
Supp. 2d 359 (E.D. Pa. 2008) (using the civil standard to select a spoliation sanction in a criminal
case).
The key considerations for determining the appropriate spoliation sanction (e.g.,
2
The most factually similar criminal case that the Court has found is United States v.
Fleming, 818 F. Supp. 845, 851 (E.D. Pa. 1993), in which the district court refused to give the
jury an adverse inference instruction where two witnesses had destroyed evidence before they
began cooperating with the Government. Since the destruction occurred before cooperation
began, Fleming provides little guidance here.
Page 14 of 20
dismissal, suppression, fines, or an adverse inference instruction) are:
(1) The degree of fault of the party who altered or destroyed the evidence;
(2) The degree of prejudice suffered by the opposing party; and
(3) Whether there is a lesser sanction that will avoid substantial unfairness to the
opposing party and, where the offending party is seriously at fault, will serve to
deter such conduct by others in the future.
Schmid v. Milwaukee Elec. Tool Corp., 13 F.3d 76, 79 (3d Cir. 1994).
In this case there is little evidence to suggest that the Government acted in bad faith in
destroying the text messages. There is, however, evidence that indicates that the defense is in
fact prejudiced by the loss of this evidence. The credibility of the cooperating witness and his
interpretation of what was transpiring and being said at the meetings with the defendants herein
is of paramount importance, and the text messages would have enabled the defense to conduct a
more effective and thorough cross examination of the cooperating witness. The Court, having
already concluded that the relatively severe sanction of suppression is not warranted in this case,
agrees with Defendants that the “lesser sanction” of an adverse inference instruction is
appropriate, considering the Schmid factors. The Court finds the adverse inference instruction
appropriate because:
(1) The text messages were within the Government’s control;
(2) The text messages were intentionally deleted by the agents, and the U.S.
Attorneys’ Office failed to take steps to preserve them;
(3) The text messages that were deleted or not preserved were relevant to claims or
defenses; and
(4) It was reasonably foreseeable by the Government that in the context of this
investigation and in light of the actions of the cooperating witness the text
messages would later have been discoverable.
Page 15 of 20
These findings by the Court fall squarely within the four elements set forth in Mosaid
Technologies Inc. v. Samsung Electronics Co., 348 F. Supp. 2d 332 (D.N.J. 2004). Under
Mosaid, the Court may only give an adverse inference instruction based on spoliation if the
following elements are satisfied:
(1) The evidence in question must be within the party's control;
(2) It must appear that there has been actual suppression or withholding of the
evidence;
(3) The evidence destroyed or withheld was relevant to claims or defenses; and
(4) It was reasonably foreseeable that the evidence would later be discoverable.
348 F. Supp. 2d at 336.
As to the first element, the Court has already addressed the extent to which the text
messages were within the Government’s “control.” Even if the messages never found their way
to the F.B.I. servers, they persisted on the agents’ handheld devices until the agents deleted them.
Thus, the text messages were within the Government’s “control.”
As to the second element, the key inquiry is the meaning of “actual suppression.” The
Third Circuit has explained that “[n]o unfavorable inference arises when the circumstances
indicate that the document or article in question has been lost or accidentally destroyed, or where
the failure to produce it is otherwise properly accounted for.” Brewer v. Quaker State Oil Ref.
Corp., 72 F.3d 326, 334 (3d Cir. 1995). In Mosaid, the district court directly addressed whether
the “actual suppression” requirement implies a requirement of bad faith. 348 F. Supp. 2d 332.
After a thorough analysis of Third Circuit case law in light of the spoliation inference’s
“remedial function” of “leveling the playing field after a party has destroyed or withheld relevant
Page 16 of 20
evidence,” the court in Mosaid adopted a “flexible approach,” which takes into account both the
culpability of the offending party and the prejudice to the innocent party. Id. at 337-38. It
explained that cases will arise in which “the offending party’s culpability is largely irrelevant as
it cannot be denied that the opposing party has been prejudiced.” Id. The Court finds this
flexible approach particularly persuasive in light of the complementary flexibility that the Court
maintains to tailor jury instructions to the needs of specific cases. See Pension Comm. v. Banc
of Am. Sec., 685 F. Supp. 2d 456, 470-71 (S.D.N.Y. 2010) (categorizing different forms of
adverse inference instructions based on the severity of the offending party’s misconduct).
In this case there is little evidence to suggest that the Government acted in bad faith in
destroying the text messages. The most that can be said is that the Government failed to instruct
the F.B.I.–and the F.B.I. failed independently to recognize the need–to preserve potential Jencks
material in anticipation of bringing criminal charges against Defendants. Nonetheless, the
prejudicial effect of this failure cannot be ignored. Defendants are now without cross
examination material that could be used to thoroughly test the credibility of the cooperating
witness and determine whether the cooperating witness actually did or did not follow the
instructions given in the F.B.I. agents’ text messages, or whether the interpretation that the
witness gave in the text messages as to what was said at the meetings was or was not at odds with
the interpretation that the witness gave at trial. Thus, while the culpability of the Government at
worst falls within the realm of gross negligence, the prejudice to Defendants is clear. The Court
therefore finds “actual suppression or withholding” of the text messages by the Government
under the flexible approach set forth in Mosaid.
As to the third element, the Court must determine whether the destroyed evidence was
Page 17 of 20
relevant to the claims and defenses in this case. Thorough cross examination of government
cooperating witnesses is vital to a defendant’s presentation of his or her case. Evidence relevant
to cross examination is therefore likewise relevant to the defendant’s case. Thus, the destroyed
evidence here is “relevant to claims or defenses” in this case.
Finally, as to the fourth element, the Court must address whether “it was reasonably
foreseeable that the evidence would later be discoverable.” Mosaid, 348 F. Supp. 2d at 336.
Here, the Government used the cooperating witness in a joint investigation with the F.B.I. to
gather evidence of alleged public corruption throughout New Jersey. Indeed, the cooperating
witness, who is often seen text messaging on the videos of his meetings with Defendants, was the
prosecution’s key trial witness and was cooperating in the investigation by virtue of a plea and
cooperation agreement. Thus, the Government must have foreseen that any defense attorney
viewing the same videos that the Government was privy to would seek discovery of the text
messages and that the messages were therefore a potential source of Jencks material. The Court
thus concludes that it is at least “reasonably foreseeable” that communications between the
cooperating witness and the Government agents instructing him during the investigation would
be discoverable in this case.
Having concluded that an adverse inference instruction is warranted in this case, the
Court now addresses how the instruction should be phrased. In Pension Committee v. Banc of
America Securities, the court delineated three levels of adverse inference instruction, differing in
severity based on the nature of the spoliating party’s conduct. 685 F. Supp. 2d at 470. The court
explained:
In its most harsh form, when a spoliating party has acted willfully or in bad faith, a jury
Page 18 of 20
can be instructed that certain facts are deemed admitted and must be accepted as true. At
the next level, when a spoliating party has acted willfully or recklessly, a court may
impose a mandatory presumption. Even a mandatory presumption, however, is
considered to be rebuttable.
The least harsh instruction permits (but does not require) a jury to presume that the lost
evidence is both relevant and favorable to the innocent party. If it makes this
presumption, the spoliating party’s rebuttal evidence must then be considered by the jury,
which must then decide whether to draw an adverse inference against the spoliating party.
This sanction still benefits the innocent party in that it allows the jury to consider both the
misconduct of the spoliating party as well as proof of prejudice to the innocent party.
Such a charge should be termed a “spoliation charge” to distinguish it from a charge
where the a jury is directed to presume, albeit still subject to rebuttal, that the missing
evidence would have been favorable to the innocent party, and from a charge where the
jury is directed to deem certain facts admitted.
Id. at 470-71 (emphasis in original).
As already discussed, the Banc of America approach complements the flexible analysis
advocated in Mosaid. Not only is the culpability of the spoliating party relevant to selecting an
appropriate adverse inference instruction, so is the resulting prejudice to the innocent party.
Given the lack of evidence suggesting bad faith on the part of the Government and in light of the
particular circumstances of this case, the Court deems the least harsh, “spoliation charge” most
appropriate. At the close of evidence, the Court will issue the following charge to the jury:
During the course of this trial you have heard evidence by way of stipulation and
testimony that during the Government’s investigation of Defendants, the cooperating
witness, Solomon Dwek, exchanged numerous text messages with F.B.I. agents
supervising the investigation. The Government was obligated to preserve all of these text
messages, but they were either deleted by the agents themselves or not preserved by the
Government. Specifically, although some text messages were in fact preserved, the
Government failed to preserve other text messages, which pertained to Agent Russ and
Agent McCarthy, from two key time periods: March 1 through March 16, 2009 and the
entire month of July 2009. You may infer from the Government’s failure to preserve
these messages, or the fact that they were deleted by agents, that the missing text
messages were relevant to this case and that they were favorable to both Defendant
Suarez and Defendant Tabbachino. You are not required to make this inference,
however, and you must consider any rebuttal evidence that has been offered by the
Page 19 of 20
Government with regard to this issue. Whether you ultimately choose to make the
inference is your decision as the finder of fact.
IV. Conclusion
For the foregoing reasons, the Court grants in part and denies in part Defendant’s Motion.
An appropriate Order accompanies this Opinion.
DATED: October 19, 2010 /s/ Jose L. Linares

JOSE L. LINARES
UNITED STATES DISTRICT JUDGE
Page 20 of 20
Federal Judge Rules Government Failed to Preserve Text Messages and Orders Adverse I ... Page 1 of 2

Posted at 10:08 PM on November 2, 2010 by Jennifer Marino Thibodaux
Federal Judge Rules Government Failed to Preserve Text Messages and Orders
Adverse Inference Instruction in Criminal Case
On October 21, 2010, in the highly publicized New Jersey government corruption case U.S. v. Suarez, et
ano., No. 09-932, 2010 U.S. Dist. LEXIS 112097 (D.N.J.), the Honorable Jose L. Linares, U.S.D.J., held
that the FBI had a duty to preserve Short Message Service electronic communications (i.e., text
messages) exchanged between its agents and their cooperating witness, Solomon Dwek, during the course
of the investigation of defendants Anthony Suarez (mayor of Ridgefield, NJ) and Vincent Tabbachino
(former Guttenberg, NJ councilman and police officer). Despite the lack of evidence of bad faith on the
part of the government, because the text messages were not preserved, the Court found clear prejudice to
defendants and ordered that the appropriate sanction was a “permissive” adverse inference jury
instruction.
From March to July 2009, Dwek assumed the identity of a real estate developer to assist the FBI with its
corruption investigation of local public officials. Specifically, the FBI instructed Dwek to meet with the
officials to express interest in real estate development projects and to offer bribes to them in exchange for
expediting the projects and providing other official assistance. In substance, the text messages exchanged
between Dwek and the agents during this period concerned investigation logistics, Dwek’s impressions of
what was transpiring during the investigation and the agents’ instructions to Dwek in carrying out the
investigation (e.g., encouraging Dwek to be “blatant” and to “dirty up the money”).
Shortly after defendants moved to compel production of the text messages, the government agreed to
produce them but later advised that it could not do so. The government explained that Dwek used a
personal cell phone to transmit text messages, which his carrier retained for only 3 to 5 days. As to the
text messages originating from or received on the agents’ Blackberry® handheld devices, the government
ultimately admitted that it failed to preserve many of these messages, notwithstanding its document
retention policy concerning “reasonably anticipated or pending litigation.” The government did not issue
a litigation hold until on or about January 11, 2010, long after the subject data would have been destroyed
in the normal course of business.
The Court found that the text messages were discoverable under the Jencks Act, 18 U.S.C. § 3500, and
Federal Rule of Criminal Procedure 26.2 and should have been preserved and produced. The Court
declined to impose the harsh sanction of suppressing Dwek’s testimony and all tape recordings in which
he participated, but carefully analyzed the propriety of giving the jury one of various forms of adverse
inference instructions. Consulting the Honorable Shira A. Scheindlin’s decision in Pension Committee of
the University of Montreal Pension Plan v. Banc of America Securities, LLC, 685 F. Supp. 2d 456
(S.D.N.Y. 2010) (corrected in part by Order dated May 28, 2010), the Court ordered the most “flexible”
and “least harsh” adverse inference instruction: permitting, but not requiring, the jury to presume that the
http://www.ediscoverylawalert.com/2010/11/articles/legal-decisions-court-rules/federal-jud... 4/1/2013
Federal Judge Rules Government Failed to Preserve Text Messages and Orders Adverse I ... Page 2 of 2
lost text messages were both relevant and favorable to defendants, but mandating that the jury consider
rebuttal evidence on these issues. The Court acknowledged that, although defendants were clearly
prejudiced in their ability to impeach the government’s cooperating witness, there was “little evidence” to
suggest that the government acted in bad faith. On October 27, 2010, the jury acquitted defendant Suarez
on all counts (conspiracy to commit extortion, attempted extortion and bribery) and convicted defendant
Tabbachino of attempted extortion and bribery.
As with all decisions imposing sanctions for spoliation of evidence, U.S. v. Suarez underscores the
critical importance of promptly issuing a litigation hold and properly preserving evidence. Failure to
discharge these now well established duties, even absent bad faith, can result in severe sanctions that may
negatively impact the prosecution of a case, both in the criminal and the civil context.
Mara E. Zazzali-Hogan is a Director in the Gibbons Business & Commercial Litigation Department. Jennifer Marino Thibodaux , an Associate in the Gibbons Business & Commercial
Litigation Department, co -authored this post.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2010/11/articles/legal-decisions-court-rules/federal-jud... 4/1/2013
How a "Stink Bomb" E-Mail and Its Proof That Facebook Pictures Were Deleted Might ... Page 1 of 2

Posted at 12:55 PM on August 8, 2011 by Gibbons P.C.
How a "Stink Bomb" E-Mail and Its Proof That Facebook Pictures Were Deleted
Might Have Blown Up a $10.6 Million Verdict
Parties in all types of cases often post pictures and messages on Facebook that might be detrimental to
their cases. After his wife died tragically in an automobile accident, and he brought a wrongful death
case, Isaiah Lester did just that when he posted a photo of himself wearing an “I [love] hot moms” t-shirt
and garter belt on his head while he had a beer in hand. That was his first bad choice.
The defense attorneys, who represented the driver of the vehicle and his employer, saw the picture and
requested similar pictures and screen shots from Lester’s Facebook account. The next day, Lester’s
attorney, Matthew B. Murray (who is the Virginia Trial Lawyers Association’s immediate past-
president), allegedly instructed his paralegal to advise Lester to delete some Facebook pictures. Lester,
however, purportedly informed Murray that he had deleted his Facebook account. Clearly, these actions
reflect additional unwise decisions. Defense attorneys are now challenging the $10.6 million verdict,
which may be the state’s largest wrongful death case.
What was even more egregious according to defense counsel, however, was that Murray denied that
Lester had a Facebook account on two specific dates on which he in fact did. They further contend that
to add insult to injury, Murray denied up until the time of trial that he dictated any alteration of the
Facebook account. Defense attorneys also contend that if the misrepresentations made to the court and
opposing counsel were not sufficiently disconcerting, that Murray then deliberately withheld evidence,
namely, a “stink bomb” e-mail reflecting that indeed, Murray had essentially directed his client to delete
pictures from his Facebook page. Lester and Murray were also accused of other improprieties.
Ultimately, defense counsel proposed to the court there possible solutions: (1) dismiss Lester’s claim
and enter a verdict in favor of the defendants; (2) throw out Lester’s claim, set aside the verdict and
allow a new trial, which would limit Lester’s recovery of damages and prohibit Murray or his firm from
representing Lester; or (3) reduce the verdict to $2.2 million or $1.1 million and preclude Lester’s
attorney or the firm from collecting any contingent fees. A hearing regarding this issue will be held on
September 23, 2011.
Murray has reportedly since retired from the practice of law and resigned from his firm on July 6, 2011.
It is unclear if any ethical or disciplinary proceedings will follow. But suffice it to say that while
attorneys should advise their clients not to post potentially damning photographs or messages to social
media websites, they should be wary of the consequences of advising their clients to delete or alter such
information -- particularly when it is the subject of a discovery request. (Click here for information on
how to obtain Facebook discovery or preserve postings/pages through Facebook’s “Download Your
http://www.ediscoverylawalert.com/2011/08/articles/legal-decisions-court-rules/how-a-stink... 4/3/2013
How a "Stink Bomb" E-Mail and Its Proof That Facebook Pictures Were Deleted Might ... Page 2 of 2
Information” feature.) Indeed, that may be an obvious and prudent course of action for most attorneys.
At least one attorney, however, failed to answer the clue phone on that issue.
Mara E. Zazzali -Hogan is a Director on the Gibbons E -Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2011/08/articles/legal-decisions-court-rules/how-a-stink... 4/3/2013
Lester v. Allied Part 2: "Clean Up" of Compromising Social Media Evidence Can Result i ... Page 1 of 2

Posted at 11:30 AM on February 21, 2012 by Gibbons P.C.
Lester v. Allied Part 2: "Clean Up" of Compromising Social Media Evidence Can
Result in Severe Sanctions
Though some practitioners might be in denial, the follow-up sanctions orders in Lester v. Allied
Concrete Co. et al. dated May 27, 2011 and September 23, 2011 should leave no room for doubt that
preservation of social media is as important as any other electronic data or discovery. Similarly, the
penalty for intentionally destroying such evidence may reach beyond the purse strings.
As reported late last year in a blog posting regarding Lester Part I , defense counsel moved for sanctions
against plaintiff Isaiah Lester and his attorney, Matthew B. Murray, for spoliation of Facebook evidence.
Ultimately, although defense counsel’s request for a new trial was denied after the jury had reached a
$10.6 million verdict related to the tragic death of Lester’s wife in an automobile accident, Lester and
Murray were ordered to pay defense counsel respectively, $180,000 and $542,000. But the sanctions did
not end there; Murray’s misconduct was referred to the Virginia State Bar and allegations of Lester’s
perjury were referred to the local prosecutor.
During discovery, defendants sought production of the contents of Lester’s account because they
believed it would help their case on the issue of damages. Defendants were inspired to pursue this
evidence after discovering a photo on Facebook depicting “Lester clutching a beer can, wearing a T-shirt
emblazoned with ‘I [heart] hot moms.’” Specifically, they served discovery requests that sought
information relating to Lester’s Facebook account including screen-prints and attached the
compromising picture.
After reviewing Lester’s Facebook page in conjunction with the discovery requests, Murray’s paralegal
(at Murray’s direction) advised Lester via e-mail to “clean up” the Facebook page because “we do NOT
want blow ups of other pics at trial so please, please clean up your facebook and myspace.” She also
advised there were “other pics that should be deleted.” Thereafter, Murray concocted a scheme to
deactivate Lester’s Facebook account and to advise defense counsel that on the date the answer to the
discovery was signed that Lester had no Facebook page.
Shortly thereafter, Lester deleted 16 photos (which were ultimately available at trial) to ensure he
complied with Murray’s initial “clean up” request, although Murray denied knowledge of deletion of
these photos in discovery responses. Thereafter, Murray and Lester attempted -- unsuccessfully -- to
cover up these communications and related acts. (Defendants’ internet technology expert was ultimately
able to confirm deletion of the photos).
The amount and scope of sanctions in Lester make clear that social media evidence is as important as
http://www.ediscoverylawalert.com/2012/02/articles/legal-decisions-court-rules/lester-v-alli... 4/3/2013
Lester v. Allied Part 2: "Clean Up" of Compromising Social Media Evidence Can Result i ... Page 2 of 2
other electronic discovery, and that the penalties for its spoliation can be as severe as those for
destroying other evidence. While the egregious malfeasance of Lester and Murray (and Murray’s
paralegal, who was under his supervision) did not result in an overturned verdict (though it was reduced
by around $4 million for other reasons), both counsel and client paid a high price for their cavalier
treatment and destruction of Facebook evidence.
Mara E. Zazzali-Hogan is a Director on the Gibbons E -Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2012/02/articles/legal-decisions-court-rules/lester-v-alli... 4/3/2013
Ex-Juror Who "Friended" Defendant Faces Jail for Bragging on Facebook About Dismiss... Page 1 of 2

Posted at 9:02 AM on March 22, 2012 by Jennifer Marino Thibodaux
Ex-Juror Who "Friended" Defendant Faces Jail for Bragging on Facebook About
Dismissal From Jury Duty
By now, attorneys should know to advise their clients to watch out for Friend requests from jurors
during a trial. The latest debacle concerning jurors use of social media involves a juror “friending” a
party and then bragging about his resulting dismissal from the panel. For that juror, his Facebook antics
landed him a three-day jail sentence. Click here and here for additional coverage regarding this incident.
Back in December, a Florida man serving jury duty sent a Friend request to the defendant. The
defendant told her attorney about the request, and Circuit Judge Nancy Donnellan dismissed the juror
and admonished him for his actions. But, the story doesn’t end there.
After his dismissal, the ex-juror took to his Facebook page again. This time, he posted a comment:
“Score . . . I got dismissed!! [A]pparently they frown upon sending a friend request to the defendant . . .
haha.” Unfortunately for the ex-juror, court officials became aware of the post, leading Judge Donnellan
to call the ex-juror back into Court to face criminal contempt charges. At the end of a two-hour hearing,
Judge Donnellan commented, “I cannot think of a more insidious threat to the erosion of democracy
than citizens who do not care.” She sentenced the ex-juror to three days in jail, and he was led from the
courtroom in handcuffs.
This is just another example of social media impacting judicial proceedings. (For recent blog posts on
this issue, click here.) As the attorney for the plaintiff in this case commented, “it was a very big deal” to
his client “who had $48,000 in medical bills on the line,” and if the female defendant had, for example,
responded to the Friend request and went on a date with the juror, perhaps the juror would have found in
her favor.
In light of these missteps, attorneys must be keen to clients’ Facebook page and activity and should
consider monitoring the Facebook pages of the defendant and any jurors -- especially any juror who has
been dismissed for the improper use of social media.
Jennifer Marino Thibodaux is an Associate on the Gibbons E-Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
http://www.ediscoverylawalert.com/2012/03/articles/legal-decisions-court-rules/exjuror-wh... 4/1/2013
Ex-Juror Who "Friended" Defendant Faces Jail for Bragging on Facebook About Dismiss... Page 2 of 2
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
http://www.ediscoverylawalert.com/2012/03/articles/legal-decisions-court-rules/exjuror-wh... 4/1/2013
SOUTHERN DISTRICT OF NEW YORK
-------------------------------------- x
MONIQUE DA SILVA MOORE, et al., :
Plaintiffs, : 11 Civ. 1279 (ALC) (AJP)
-against- : OPINION AND ORDER
PUBLICIS GROUPE & MSL GROUP, :
Defendants. :
-------------------------------------- x
ANDREW J. PECK, United States Magistrate Judge:
In my article Search, Forward: Will manual document review and keyword searches
be replaced by computer-assisted coding?, I wrote:
To my knowledge, no reported case (federal or state) has ruled on the use of

computer-assisted coding. While anecdotally it appears that some lawyers are using
predictive coding technology, it also appears that many lawyers (and their clients)
are waiting for a judicial decision approving of computer-assisted review.
Perhaps they are looking for an opinion concluding that: "It is the opinion
of this court that the use of predictive coding is a proper and acceptable means of
conducting searches under the Federal Rules of Civil Procedure, and furthermore that
the software provided for this purpose by [insert name of your favorite vendor] is the
software of choice in this court." If so, it will be a long wait.
....
Until there is a judicial opinion approving (or even critiquing) the use of
predictive coding, counsel will just have to rely on this article as a sign of judicial
approval. In my opinion, computer-assisted coding should be used in those cases
where it will help "secure the just, speedy, and inexpensive" (Fed. R. Civ. P. 1)
determination of cases in our e-discovery world.
G:\AJP\DA SILVA MOORE - ESI

2
Andrew Peck, Search, Forward, L. Tech. News, Oct. 2011, at 25, 29. This judicial opinion now
recognizes that computer-assisted review is an acceptable way to search for relevant ESI in
appropriate cases.1/
CASE BACKGROUND
In this action, five female named plaintiffs are suing defendant Publicis Groupe, "one
of the world's 'big four' advertising conglomerates," and its United States public relations subsidiary,
defendant MSL Group. (See Dkt. No. 4: Am. Compl. ¶¶ 1, 5, 26-32.) Plaintiffs allege that
defendants have a "glass ceiling" that limits women to entry level positions, and that there is
"systemic, company-wide gender discrimination against female PR employees like Plaintiffs." (Am.
Compl. ¶¶ 4-6, 8.) Plaintiffs allege that the gender discrimination includes
(a) paying Plaintiffs and other female PR employees less than similarly-situated male
employees; (b) failing to promote or advance Plaintiffs and other female PR
employees at the same rate as similarly-situated male employees; and (c) carrying
out discriminatory terminations, demotions and/or job reassignments of female PR
employees when the company reorganized its PR practice beginning in 2008 . . . .
(Am. Compl. ¶ 8.)
Plaintiffs assert claims for gender discrimination under Title VII (and under similar
New York State and New York City laws) (Am. Compl. ¶¶ 204-25), pregnancy discrimination under
Title VII and related violations of the Family and Medical Leave Act (Am. Compl. ¶¶ 239-71), as
1/
To correct the many blogs about this case, initiated by a press release from plaintiffs' vendor
– the Court did not order the parties to use predictive coding. The parties had agreed to
defendants' use of it, but had disputes over the scope and implementation, which the Court
ruled on, thus accepting the use of computer-assisted review in this lawsuit.

3
well as violations of the Equal Pay Act and Fair Labor Standards Act (and the similar New York
Labor Law) (Am. Compl. ¶¶ 226-38).
The complaint seeks to bring the Equal Pay Act/FLSA claims as a "collective action"
(i.e., opt-in) on behalf of all "current, former, and future female PR employees" employed by
defendants in the United States "at any time during the applicable liability period" (Am. Compl.
¶¶ 179-80, 190-203), and as a class action on the gender and pregnancy discrimination claims and
on the New York Labor Law pay claim (Am. Compl. ¶¶ 171-98). Plaintiffs, however, have not yet
moved for collective action or class certification at this time.
Defendant MSL denies the allegations in the complaint and has asserted various
affirmative defenses. (See generally Dkt. No. 19: MSL Answer.) Defendant Publicis is challenging
the Court's jurisdiction over it, and the parties have until March 12, 2012 to conduct jurisdictional
discovery. (See Dkt. No. 44: 10/12/11 Order.)
COMPUTER-ASSISTED REVIEW EXPLAINED
My Search, Forward article explained my understanding of computer-assisted review,
as follows:
By computer-assisted coding, I mean tools (different vendors use different names)

that use sophisticated algorithms to enable the computer to determine relevance,
based on interaction with (i.e., training by) a human reviewer.
Unlike manual review, where the review is done by the most junior staff,
computer-assisted coding involves a senior partner (or [small] team) who review and
code a "seed set" of documents. The computer identifies properties of those
documents that it uses to code other documents. As the senior reviewer continues
to code more sample documents, the computer predicts the reviewer's coding. (Or,
the computer codes some documents and asks the senior reviewer for feedback.)

4
When the system's predictions and the reviewer's coding sufficiently

coincide, the system has learned enough to make confident predictions for the
remaining documents. Typically, the senior lawyer (or team) needs to review only
a few thousand documents to train the computer.
Some systems produce a simple yes/no as to relevance, while others give a

relevance score (say, on a 0 to 100 basis) that counsel can use to prioritize review.
For example, a score above 50 may produce 97% of the relevant documents, but
constitutes only 20% of the entire document set.
Counsel may decide, after sampling and quality control tests, that documents
with a score of below 15 are so highly likely to be irrelevant that no further human
review is necessary. Counsel can also decide the cost-benefit of manual review of
the documents with scores of 15-50.
Andrew Peck, Search, Forward, L. Tech. News, Oct. 2011, at 25, 29.2/
My article further explained my belief that Daubert would not apply to the results of
using predictive coding, but that in any challenge to its use, this Judge would be interested in both
the process used and the results:
[I]f the use of predictive coding is challenged in a case before me, I will want to
know what was done and why that produced defensible results. I may be less
interested in the science behind the "black box" of the vendor's software than in
whether it produced responsive documents with reasonably high recall and high
precision.
That may mean allowing the requesting party to see the documents that were
used to train the computer-assisted coding system. (Counsel would not be required
to explain why they coded documents as responsive or non-responsive, just what the
coding was.) Proof of a valid "process," including quality control testing, also will
be important.
....
2/
From a different perspective, every person who uses email uses predictive coding, even if
they do not realize it. The "spam filter" is an example of predictive coding.

5
Of course, the best approach to the use of computer-assisted coding is to

follow the Sedona Cooperation Proclamation model. Advise opposing counsel that
you plan to use computer-assisted coding and seek agreement; if you cannot,
consider whether to abandon predictive coding for that case or go to the court for
advance approval.
Id.
THE ESI DISPUTES IN THIS CASE AND THEIR RESOLUTION
After several discovery conferences and rulings by Judge Sullivan (the then-assigned
District Judge), he referred the case to me for general pretrial supervision. (Dkt. No. 48: 11/28/11
Referral Order.) At my first discovery conference with the parties, both parties' counsel mentioned
that they had been discussing an "electronic discovery protocol," and MSL's counsel stated that an
open issue was "plaintiff's reluctance to utilize predictive coding to try to cull down the"
approximately three million electronic documents from the agreed-upon custodians. (Dkt. No. 51:
12/2/11 Conf. Tr. at 7-8.)3/ Plaintiffs' counsel clarified that MSL had "over simplified [plaintiffs']
stance on predictive coding," i.e., that it was not opposed but had "multiple concerns . . . on the way
in which [MSL] plan to employ predictive coding" and plaintiffs wanted "clarification." (12/2/11
Conf. Tr. at 21.)
The Court did not rule but offered the parties the following advice:
Now, if you want any more advice, for better or for worse on the ESI plan
and whether predictive coding should be used, . . . I will say right now, what should
3/
When defense counsel mentioned the disagreement about predictive coding, I stated that:
"You must have thought you died and went to Heaven when this was referred to me," to
which MSL's counsel responded: "Yes, your Honor. Well, I'm just thankful that, you know,
we have a person familiar with the predictive coding concept." (12/2/11 Conf. Tr. at 8-9.)

6
not be a surprise, I wrote an article in the October Law Technology News called
Search Forward, which says predictive coding should be used in the appropriate case.
Is this the appropriate case for it? You all talk about it some more. And if
you can't figure it out, you are going to get back in front of me. Key words, certainly
unless they are well done and tested, are not overly useful. Key words along with
predictive coding and other methodology, can be very instructive.
I'm also saying to the defendants who may, from the comment before, have
read my article. If you do predictive coding, you are going to have to give your seed
set, including the seed documents marked as nonresponsive to the plaintiff's counsel
so they can say, well, of course you are not getting any [relevant] documents, you're
not appropriately training the computer.
(12/2/11 Conf. Tr. at 20-21.) The December 2, 2011 conference adjourned with the parties agreeing
to further discuss the ESI protocol. (12/2/11 Conf. Tr. at 34-35.)
The ESI issue was next discussed at a conference on January 4, 2012. (Dkt. No. 71:
1/4/12 Conf. Tr.) Plaintiffs' ESI consultant conceded that plaintiffs "have not taken issue with the
use of predictive coding or, frankly, with the confidence levels that they [MSL] have proposed . . . ."
(1/4/12 Conf. Tr. at 51.) Rather, plaintiffs took issue with MSL's proposal that after the computer
was fully trained and the results generated, MSL wanted to only review and produce the top 40,000
documents, which it estimated would cost $200,000 (at $5 per document). (1/4/12 Conf. Tr. at 47-
48, 51.) The Court rejected MSL's 40,000 documents proposal as a "pig in a poke." (1/4/12 Conf.
Tr. at 51-52.) The Court explained that "where [the] line will be drawn [as to review and
production] is going to depend on what the statistics show for the results," since "[p]roportionality
requires consideration of results as well as costs. And if stopping at 40,000 is going to leave a
tremendous number of likely highly responsive documents unproduced, [MSL's proposed cutoff]
doesn't work." (1/4/12 Conf. Tr. at 51-52; see also id. at 57-58; Dkt. No. 88: 2/8/12 Conf. Tr. at 84.)

7
The parties agreed to further discuss and finalize the ESI protocol by late January 2012, with a
conference held on February 8, 2012. (1/4/12 Conf. Tr. at 60-66; see 2/8/12 Conf. Tr.)
Custodians
The first issue regarding the ESI protocol involved the selection of which custodians'
emails would be searched. MSL agreed to thirty custodians for a "first phase." (Dkt. No. 88: 2/8/12
Conf. Tr. at 23-24.) MSL's custodian list included the president and other members of MSL's
"executive team," most of its HR staff and a number of managing directors. (2/8/12 Conf. Tr. at 24.)
Plaintiffs sought to include as additional custodians seven male "comparators,"
explaining that the comparators' emails were needed in order to find information about their job
duties and how their duties compared to plaintiffs' job duties. (2/8/12 Conf. Tr. at 25-27.) Plaintiffs
gave an example of the men being given greater "client contact" or having better job assignments.
(2/8/12 Conf. Tr. at 28-30.) The Court held that the search of the comparators' emails would be so
different from that of the other custodians that the comparators should not be included in the emails
subjected to predictive coding review. (2/8/12 Conf. Tr. at 28, 30.) As a fallback position, plaintiffs
proposed to "treat the comparators as a separate search," but the Court found that plaintiffs could
not describe in any meaningful way how they would search the comparators' emails, even as a
separate search. (2/8/12 Conf. Tr. at 30-31.) Since the plaintiffs likely could develop the
information needed through depositions of the comparators, the Court ruled that the comparators'
emails would not be included in phase one. (2/8/12 Conf. Tr. at 31.)
Plaintiffs also sought to include MSL's CEO, Olivier Fleuriot, located in France and
whose emails were mostly written in French. (2/8/12 Conf. Tr. at 32-34.) The Court concluded that

8
because his emails with the New York based executive staff would be gathered from those
custodians, and Fleuriot's emails stored in France likely would be covered by the French privacy and
blocking laws,4/ Fleuriot should not be included as a first-phase custodian. (2/8/12 Conf. Tr. at 35.)
Plaintiffs sought to include certain managing directors from MSL offices at which
no named plaintiff worked. (2/8/12 Conf. Tr. at 36-37.) The Court ruled that since plaintiffs had
not yet moved for collective action status or class certification, until the motions were made and
granted, discovery would be limited to offices (and managing directors) where the named plaintiffs
had worked. (2/8/12 Conf. Tr. at 37-39.)
The final issue raised by plaintiffs related to the phasing of custodians and the
discovery cutoff dates. MSL proposed finishing phase-one discovery completely before considering
what to do about a second phase. (See 2/8/12 Conf. Tr. at 36.) Plaintiffs expressed concern that
there would not be time for two separate phases, essentially seeking to move the phase-two
custodians back into phase one. (2/8/12 Conf. Tr. at 35-36.) The Court found MSL's separate phase
approach to be more sensible and noted that if necessary, the Court would extend the discovery
cutoff to allow the parties to pursue discovery in phases. (2/8/12 Conf. Tr. at 36, 50.)
4/
See, e.g., Societe Nationale Industrielle Aerospatiale v. U.S. Dist. Ct. for the S.D. of Iowa,
482 U.S. 522, 107 S. Ct. 2542 (1987); see also The Sedona Conference, International
Principles on Discovery, Disclosure & Data Protection (2011), available at
http://www.thesedonaconference.org/dltForm?did=IntlPrinciples2011.pdf.

9
Sources of ESI
The parties agreed on certain ESI sources, including the "EMC SourceOne [Email]
Archive," the "PeopleSoft" human resources information management system and certain other
sources including certain HR "shared" folders. (See Dkt. No. 88: 2/8/12 Conf. Tr. at 44-45, 50-51.)
As to other "shared" folders, neither side was able to explain whether the folders merely contained
forms and templates or collaborative working documents; the Court therefore left those shared
folders for phase two unless the parties promptly provided information about likely contents.
(2/8/12 Conf. Tr. at 47-48.)
The Court noted that because the named plaintiffs worked for MSL, plaintiffs should
have some idea what additional ESI sources, if any, likely had relevant information; since the Court
needed to consider proportionality pursuant to Rule 26(b)(2)(C), plaintiffs needed to provide more
information to the Court than they were doing if they wanted to add additional data sources into
phase one. (2/8/12 Conf. Tr. at 49-50.) The Court also noted that where plaintiffs were getting
factual information from one source (e.g., pay information, promotions, etc.), "there has to be a limit
to redundancy" to comply with Rule 26(b)(2)(C). (2/8/12 Conf. Tr. at 54.)5/
The Predictive Coding Protocol
The parties agreed to use a 95% confidence level (plus or minus two percent) to
create a random sample of the entire email collection; that sample of 2,399 documents will be
5/
The Court also suggested that the best way to resolve issues about what information might
be found in a certain source is for MSL to show plaintiffs a sample printout from that source.
(2/8/12 Conf. Tr. at 55-56.)

10
reviewed to determine relevant (and not relevant) documents for a "seed set" to use to train the
predictive coding software. (Dkt. No. 88: 2/8/12 Conf. Tr. at 59-61.) An area of disagreement was
that MSL reviewed the 2,399 documents before the parties agreed to add two additional concept
groups (i.e., issue tags). (2/8/12 Conf. Tr. at 62.) MSL suggested that since it had agreed to provide
all 2,399 documents (and MSL's coding of them) to plaintiffs for their review, plaintiffs can code
them for the new issue tags, and MSL will incorporate that coding into the system. (2/8/12 Conf.
Tr. at 64.) Plaintiffs' vendor agreed to that approach. (2/8/12 Conf. Tr. at 64.)
To further create the seed set to train the predictive coding software, MSL coded
certain documents through "judgmental sampling." (2/8/12 Conf. Tr. at 64.) The remainder of the
seed set was created by MSL reviewing "keyword" searches with Boolean connectors (such as
"training and Da Silva Moore," or "promotion and Da Silva Moore") and coding the top fifty hits
from those searches. (2/8/12 Conf. Tr. at 64-66, 72.) MSL agreed to provide all those documents
(except privileged ones) to plaintiffs for plaintiffs to review MSL's relevance coding. (2/8/12 Conf.
Tr. at 66.) In addition, plaintiffs provided MSL with certain other keywords, and MSL used the
same process with plaintiffs' keywords as with the MSL keywords, reviewing and coding an
additional 4,000 documents. (2/8/12 Conf. Tr. at 68-69, 71.) All of this review to create the seed
set was done by senior attorneys (not paralegals, staff attorneys or junior associates). (2/8/12 Conf.
Tr. at 92-93.) MSL reconfirmed that "[a]ll of the documents that are reviewed as a function of the
seed set, whether [they] are ultimately coded relevant or irrelevant, aside from privilege, will be
turned over to" plaintiffs. (2/8/12 Conf. Tr. at 73.)

11
The next area of discussion was the iterative rounds to stabilize the training of the
software. MSL's vendor's predictive coding software ranks documents on a score of 100 to zero,
i.e., from most likely relevant to least likely relevant. (2/8/12 Conf. Tr. at 70.) MSL proposed using
seven iterative rounds; in each round they would review at least 500 documents from different
concept clusters to see if the computer is returning new relevant documents. (2/8/12 Conf. Tr. at
73-74.) After the seventh round, to determine if the computer is well trained and stable, MSL would
review a random sample (of 2,399 documents) from the discards (i.e., documents coded as non-
relevant) to make sure the documents determined by the software to not be relevant do not, in fact,
contain highly-relevant documents. (2/8/12 Conf. Tr. at 74-75.) For each of the seven rounds and
the final quality-check random sample, MSL agreed that it would show plaintiffs all the documents
it looked at including those deemed not relevant (except for privileged documents). (2/8/12 Conf.
Tr. at 76.)
Plaintiffs' vendor noted that "we don't at this point agree that this is going to work.
This is new technology and it has to be proven out." (2/8/12 Conf. Tr. at 75.) Plaintiffs' vendor
agreed, in general, that computer-assisted review works, and works better than most alternatives.
(2/8/12 Conf. Tr. at 76.) Indeed, plaintiffs' vendor noted that "it is fair to say [that] we are big
proponents of it." (2/8/12 Conf. Tr. at 76.) The Court reminded the parties that computer-assisted
review "works better than most of the alternatives, if not all of the [present] alternatives. So the idea
is not to make this perfect, it's not going to be perfect. The idea is to make it significantly better than
the alternatives without nearly as much cost." (2/8/12 Conf. Tr. at 76.)

12
The Court accepted MSL's proposal for the seven iterative reviews, but with the
following caveat:
But if you get to the seventh round and [plaintiffs] are saying that the computer is
still doing weird things, it's not stabilized, etc., we need to do another round or two,
either you will agree to that or you will both come in with the appropriate QC
information and everything else and [may be ordered to] do another round or two or
five or 500 or whatever it takes to stabilize the system.
(2/8/12 Conf. Tr. at 76-77; see also id. at 83-84, 88.)
On February 17, 2012, the parties submitted their "final" ESI Protocol which the
Court "so ordered." (Dkt. No. 92: 2/17/12 ESI Protocol & Order.)6/ Because this is the first Opinion
dealing with predictive coding, the Court annexes hereto as an Exhibit the provisions of the ESI
Protocol dealing with the predictive coding search methodology.
OBSERVATIONS ON PLAINTIFF'S OBJECTIONS

TO THE COURT'S RULINGS
On February 22, 2012, plaintiffs filed objections to the Court's February 8, 2012
rulings. (Dkt. No. 93: Pls. Rule 72(a) Objections; see also Dkt. No. 94: Nurhussein Aff.; Dkt. No.
95: Neale Aff.) While those objections are before District Judge Carter, a few comments are in
order.
6/
Plaintiffs included a paragraph noting its objection to the ESI Protocol, as follows:
Plaintiffs object to this ESI Protocol in its entirety. Plaintiffs submitted their
own proposed ESI Protocol to the Court, but it was largely rejected. The Court then
ordered the parties to submit a joint ESI Protocol reflecting the Court's rulings.
Accordingly, Plaintiffs jointly submit this ESI Protocol with MSL, but reserve the
right to object to its use in this case.
(ESI Protocol ¶ J.1 at p. 22.)

13
Plaintiffs' Reliance on Rule 26(g)(1)(A) is Erroneous
Plaintiffs' objections to my February 8, 2012 rulings assert that my acceptance of
MSL's predictive coding approach "provides unlawful 'cover' for MSL's counsel, who has a duty
under FRCP 26(g) to 'certify' that their client's document production is 'complete' and 'correct' as of
the time it was made. FRCP 26(g)(1)(A)." (Dkt. No. 93: Pls. Rule 72(a) Objections at 8 n.7; accord,
id. at 2.) In large-data cases like this, involving over three million emails, no lawyer using any
search method could honestly certify that its production is "complete" – but more importantly, Rule
26(g)(1) does not require that. Plaintiffs simply misread Rule 26(g)(1). The certification required
by Rule 26(g)(1) applies "with respect to a disclosure." Fed. R. Civ. P. 26(g)(1)(A) (emphasis
added). That is a term of art, referring to the mandatory initial disclosures required by Rule
26(a)(1). Since the Rule 26(a)(1) disclosure is information (witnesses, exhibits) that "the disclosing
party may use to support its claims or defenses," and failure to provide such information leads to
virtually automatic preclusion, see Fed. R Civ. P. 37(c)(1), it is appropriate for the Rule 26(g)(1)(A)
certification to require disclosures be "complete and correct."
Rule 26(g)(1)(B) is the provision that applies to discovery responses. It does not call
for certification that the discovery response is "complete," but rather incorporates the Rule
26(b)(2)(C) proportionality principle. Thus, Rule 26(g)(1)(A) has absolutely nothing to do with
MSL's obligations to respond to plaintiffs' discovery requests. Plaintiffs' argument is based on a
misunderstanding of Rule 26(g)(1).7/
7/
Rule 26(g)(1) provides:
(continued...)

14
Rule 702 and Daubert Are Not Applicable to Discovery Search Methods
Plaintiffs' objections also argue that my acceptance of MSL's predictive coding
protocol "is contrary to Federal Rule of Evidence 702" and "violates the gatekeeping function
underlying Rule 702." (Dkt. No. 93: Pls. Rule 72(a) Objections at 2-3; accord, id. at 10-12.)8/
7/
(...continued)
(g) Signing Disclosures and Discovery Requests, Responses, and Objections.
(1) Signature Required; Effect of Signature. Every disclosure under Rule 26(a)(1)
or (a)(3) and every discovery request, response, or objection must be signed by at
least one attorney of record in the attorney's own name . . . . By signing, an attorney
or party certifies that to the best of the person's knowledge, information, and belief
formed after a reasonable inquiry:
(A) with respect to a disclosure, it is complete and correct as of the time

it is made; and
(B) with respect to a discovery request, response, or objection, it is:
(i) consistent with these rules and warranted by existing law or by

a nonfrivolous argument for extending, modifying, or reversing
existing law, or for establishing new law;
(ii) not interposed for any improper purpose, such as to harass,

cause unnecessary delay, or needlessly increase the cost of litigation;
and
(iii) neither unreasonable nor unduly burdensome or expensive,

considering the needs of the case, prior discovery in the case, the
amount in controversy, and the importance of the issues at stake in
the action.
Fed. R. Civ. P. 26(g)(1) (emphasis added).

8/
As part of this argument, plaintiffs complain that although both parties' experts (i.e.,
vendors) spoke at the discovery conferences, they were not sworn in. (Pls. Rule 72(a)
Objections at 12: "To his credit, the Magistrate [Judge] did ask the parties to bring [to the
(continued...)

15
Federal Rule of Evidence 702 and the Supreme Court's Daubert decision9/ deal with
the trial court's role as gatekeeper to exclude unreliable expert testimony from being submitted to
the jury at trial. See also Advisory Comm. Notes to Fed. R. Evid. 702. It is a rule for admissibility
of evidence at trial.
If MSL sought to have its expert testify at trial and introduce the results of its ESI
protocol into evidence, Daubert and Rule 702 would apply. Here, in contrast, the tens of thousands
of emails that will be produced in discovery are not being offered into evidence at trial as the result
of a scientific process or otherwise. The admissibility of specific emails at trial will depend upon
each email itself (for example, whether it is hearsay, or a business record or party admission), not
how it was found during discovery.
Rule 702 and Daubert simply are not applicable to how documents are searched for
and found in discovery.
Plaintiffs' Reliability Concerns Are, At Best, Premature
Finally, plaintiffs' objections assert that "MSL's method lacks the necessary standards
for assessing whether its results are accurate; in other words, there is no way to be certain if MSL's
method is reliable." (Dkt. No. 93: Pls. Rule 72(a) Objections at 13-18.) Plaintiffs' concerns may
be appropriate for resolution during or after the process (which the Court will be closely
8/
(...continued)
conference] the ESI experts they had hired to advise them regarding the creation of an ESI
protocol. These experts, however, were never sworn in, and thus the statements they made
in court at the hearings were not sworn testimony made under penalty of perjury.") Plaintiffs
never asked the Court to have the experts testify to their qualifications or be sworn in.
9/
Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579, 113 S. Ct. 2786 (1993).

16
supervising), but are premature now. For example, plaintiffs complain that "MSL's method fails to
include an agreed-upon standard of relevance that is transparent and accessible to all parties. . . .
Without this standard, there is a high-likelihood of delay as the parties resolve disputes with regard
to individual documents on a case-by-case basis." (Id. at 14.) Relevance is determined by plaintiffs'
document demands. As statistics show, perhaps only 5% of the disagreement among reviewers
comes from close questions of relevance, as opposed to reviewer error. (See page 18 n.11 below.)
The issue regarding relevance standards might be significant if MSL's proposal was not totally
transparent. Here, however, plaintiffs will see how MSL has coded every email used in the seed set
(both relevant and not relevant), and the Court is available to quickly resolve any issues.
Plaintiffs complain they cannot determine if "MSL's method actually works" because
MSL does not describe how many relevant documents are permitted to be located in the final
random sample of documents the software deemed irrelevant. (Pls. Rule 72(a) Objections at 15-16.)
Plaintiffs argue that "without any decision about this made in advance, the Court is simply kicking
the can down the road." (Id. at 16.) In order to determine proportionality, it is necessary to have
more information than the parties (or the Court) now has, including how many relevant documents
will be produced and at what cost to MSL. Will the case remain limited to the named plaintiffs, or
will plaintiffs seek and obtain collective action and/or class action certification? In the final sample
of documents deemed irrelevant, are any relevant documents found that are "hot," "smoking gun"
documents (i.e., highly relevant)? Or are the only relevant documents more of the same thing? One
hot document may require the software to be re-trained (or some other search method employed),
while several documents that really do not add anything to the case might not matter. These types

17
of questions are better decided "down the road," when real information is available to the parties and
the Court.
FURTHER ANALYSIS AND LESSONS FOR THE FUTURE
The decision to allow computer-assisted review in this case was relatively easy – the
parties agreed to its use (although disagreed about how best to implement such review). The Court
recognizes that computer-assisted review is not a magic, Staples-Easy-Button, solution appropriate
for all cases. The technology exists and should be used where appropriate, but it is not a case of
machine replacing humans: it is the process used and the interaction of man and machine that the
courts needs to examine.
The objective of review in ediscovery is to identify as many relevant documents as
possible, while reviewing as few non-relevant documents as possible. Recall is the fraction of
relevant documents identified during a review; precision is the fraction of identified documents that
are relevant. Thus, recall is a measure of completeness, while precision is a measure of accuracy
or correctness. The goal is for the review method to result in higher recall and higher precision than
another review method, at a cost proportionate to the "value" of the case. See, e.g., Maura R.
Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More
Effective and More Efficient Than Exhaustive Manual Review, Rich. J.L.& Tech., Spring 2011, at
8-9, available at http://jolt.richmond.edu/v17i3/article11.pdf.

18
The slightly more difficult case would be where the producing party wants to use
computer-assisted review and the requesting party objects.10/ The question to ask in that situation
is what methodology would the requesting party suggest instead? Linear manual review is simply
too expensive where, as here, there are over three million emails to review. Moreover, while some
lawyers still consider manual review to be the "gold standard," that is a myth, as statistics clearly
show that computerized searches are at least as accurate, if not more so, than manual review. Herb
Roitblatt, Anne Kershaw, and Patrick Oot of the Electronic Discovery Institute conducted an
empirical assessment to "answer the question of whether there was a benefit to engaging in a
traditional human review or whether computer systems could be relied on to produce comparable
results," and concluded that "[o]n every measure, the performance of the two computer systems was
at least as accurate (measured against the original review) as that of human re-review." Herbert L.
Roitblatt, Anne Kershaw & Patrick Oot, Document Categorization in Legal Electronic Discovery:
Computer Classification vs. Manual Review, 61 J. Am. Soc'y for Info. Sci. & Tech. 70, 79 (2010).11/
10/
The tougher question, raised in Klein Prods. LLC v. Packaging Corp. of Am. before
Magistrate Judge Nan Nolan in Chicago, is whether the Court, at plaintiffs' request, should
order the defendant to use computer-assisted review to respond to plaintiffs' document
requests.
11/
The Roitblatt, Kershaw, Oot article noted that "[t]he level of agreement among human
reviewers is not strikingly high," around 70-75%. They identify two sources for this
variability: fatigue ("A document that they [the reviewers] might have categorized as
responsive when they were more attentive might then be categorized [when the reviewer is
distracted or fatigued] as non-responsive or vice versa."), and differences in "strategic
judgment." Id. at 77-78. Another study found that responsiveness "is fairly well defined,
and that disagreements among assessors are largely attributable to human error," with only
5% of reviewer disagreement attributable to borderline or questionable issues as to
relevance. Maura R. Grossman & Gordon V. Cormack, Inconsistent Assessment of
(continued...)

19
Likewise, Wachtell, Lipton, Rosen & Katz litigation counsel Maura Grossman and
University of Waterloo professor Gordon Cormack, studied data from the Text Retrieval Conference
Legal Track (TREC) and concluded that: "[T]he myth that exhaustive manual review is the most
effective – and therefore the most defensible – approach to document review is strongly refuted.
Technology-assisted review can (and does) yield more accurate results than exhaustive manual
review, with much lower effort." Maura R. Grossman & Gordon V. Cormack, Technology-Assisted
Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual
Review, Rich. J.L.& Tech., Spring 2011, at 48.12/ The technology-assisted reviews in the Grossman-
Cormack article also demonstrated significant cost savings over manual review: "The technology-
assisted reviews require, on average, human review of only 1.9% of the documents, a fifty-fold
savings over exhaustive manual review." Id. at 43.
Because of the volume of ESI, lawyers frequently have turned to keyword searches
to cull email (or other ESI) down to a more manageable volume for further manual review.
Keywords have a place in production of ESI – indeed, the parties here used keyword searches (with
Boolean connectors) to find documents for the expanded seed set to train the predictive coding
software. In too many cases, however, the way lawyers choose keywords is the equivalent of the
11/
(...continued)
Responsiveness in E-Discovery: Difference of Opinion or Human Error? 9 (DESI IV: 2011
ICAIL Workshop on Setting Standards for Searching Elec. Stored Info. in Discovery,
Research Paper), available at http://www.umiacs.umd.edu/~oard/desi4/papers/
grossman3.pdf.
12/
Grossman and Cormack also note that "not all technology-assisted reviews . . . are created
equal" and that future studies will be needed to "address which technology-assisted review
process(es) will improve most on manual review." Id.

20
child's game of "Go Fish."13/ The requesting party guesses which keywords might produce evidence
to support its case without having much, if any, knowledge of the responding party's "cards" (i.e.,
the terminology used by the responding party's custodians). Indeed, the responding party's counsel
often does not know what is in its own client's "cards."
Another problem with keywords is that they often are over-inclusive, that is, they find
responsive documents but also large numbers of irrelevant documents. In this case, for example,
a keyword search for "training" resulted in 165,208 hits; Da Silva Moore's name resulted in 201,179
hits; "bonus" resulted in 40,756 hits; "compensation" resulted in 55,602 hits; and "diversity" resulted
in 38,315 hits. (Dkt. No. 92: 2/17/12 ESI Protocol Ex. A.) If MSL had to manually review all of
the keyword hits, many of which would not be relevant (i.e., would be false positives), it would be
quite costly.
Moreover, keyword searches usually are not very effective. In 1985, scholars David
Blair and M. Maron collected 40,000 documents from a Bay Area Rapid Transit accident, and
instructed experienced attorney and paralegal searchers to use keywords and other review techniques
to retrieve at least 75% of the documents relevant to 51 document requests. David L. Blair &
M. E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System,
28 Comm. ACM 289 (1985). Searchers believed they met the goals, but their average recall was
just 20%. Id. This result has been replicated in the TREC Legal Track studies over the past few
years.
13/
See Ralph C. Losey, "Child's Game of 'Go Fish' is a Poor Model for e-Discovery Search,"
in Adventures in Electronic Discovery 209-10 (2011).

21
Judicial decisions have criticized specific keyword searches. Important early
decisions in this area came from two of the leading judicial scholars in ediscovery, Magistrate
Judges John Facciola (District of Columbia) and Paul Grimm (Maryland). See United States v.
O'Keefe, 37 F. Supp. 2d 14, 24 (D.D.C. 2008) (Facciola, M.J.); Equity Analytics, LLC v. Lundin,
248 F.R.D. 331, 333 (D.D.C. 2008) (Facciola, M.J.); Victor Stanley, Inc. v. Creative Pipe, Inc., 250
F.R.D. 251, 260, 262 (D. Md. 2008) (Grimm, M.J.). I followed their lead with Willaim A. Gross
Construction Associates, Inc., when I wrote:
This Opinion should serve as a wake-up call to the Bar in this District about
the need for careful thought, quality control, testing, and cooperation with opposing
counsel in designing search terms or "keywords" to be used to produce emails or
other electronically stored information ("ESI").
....
Electronic discovery requires cooperation between opposing counsel and

transparency in all aspects of preservation and production of ESI. Moreover, where
counsel are using keyword searches for retrieval of ESI, they at a minimum must
carefully craft the appropriate keywords, with input from the ESI's custodians as to
the words and abbreviations they use, and the proposed methodology must be quality
control tested to assure accuracy in retrieval and elimination of "false positives." It
is time that the Bar – even those lawyers who did not come of age in the computer
era – understand this.
William A. Gross Constr. Assocs., Inc. v. Am. Mfrs. Mut. Ins. Co., 256 F.R.D. 134, 134, 136
(S.D.N.Y. 2009) (Peck, M.J.).
Computer-assisted review appears to be better than the available alternatives, and
thus should be used in appropriate cases. While this Court recognizes that computer-assisted review
is not perfect, the Federal Rules of Civil Procedure do not require perfection. See, e.g., Pension
Comm. of Univ. of Montreal Pension Plan v. Banc of Am. Sec., 685 F. Supp. 2d 456, 461 (S.D.N.Y.

22
2010). Courts and litigants must be cognizant of the aim of Rule 1, to "secure the just, speedy, and
inexpensive determination" of lawsuits. Fed. R. Civ. P. 1. That goal is further reinforced by the
proportionality doctrine set forth in Rule 26(b)(2)(C), which provides that:
On motion or on its own, the court must limit the frequency or extent of discovery
otherwise allowed by these rules or by local rule if it determines that:
(i) the discovery sought is unreasonably cumulative or duplicative,

or can be obtained from some other source that is more convenient, less
burdensome, or less expensive;
(ii) the party seeking discovery has had ample opportunity to obtain
the information by discovery in the action; or
(iii) the burden or expense of the proposed discovery outweighs its

likely benefit, considering the needs of the case, the amount in controversy,
the parties' resources, the importance of the issues at stake in the action, and
the importance of the discovery in resolving the issues.
Fed. R. Civ. P. 26(b)(2)(C).
In this case, the Court determined that the use of predictive coding was appropriate
considering: (1) the parties' agreement, (2) the vast amount of ESI to be reviewed (over three
million documents), (3) the superiority of computer-assisted review to the available alternatives (i.e.,
linear manual review or keyword searches), (4) the need for cost effectiveness and proportionality
under Rule 26(b)(2)(C), and (5) the transparent process proposed by MSL.
This Court was one of the early signatories to The Sedona Conference Cooperation
Proclamation, and has stated that "the best solution in the entire area of electronic discovery is
cooperation among counsel. This Court strongly endorses The Sedona Conference Proclamation
(available at www.TheSedonaConference.org)." William A. Gross Constr. Assocs., Inc. v. Am.

23
Mfrs. Mut. Ins. Co., 256 F.R.D. at 136. An important aspect of cooperation is transparency in the
discovery process. MSL's transparency in its proposed ESI search protocol made it easier for the
Court to approve the use of predictive coding. As discussed above on page 10, MSL confirmed that
"[a]ll of the documents that are reviewed as a function of the seed set, whether [they] are ultimately
coded relevant or irrelevant, aside from privilege, will be turned over to" plaintiffs. (Dkt. No. 88:
2/8/12 Conf. Tr. at 73; see also 2/17/12 ESI Protocol at 14: "MSL will provide Plaintiffs' counsel
with all of the non-privileged documents and will provide, to the extent applicable, the issue tag(s)
coded for each document . . . . If necessary, counsel will meet and confer to attempt to resolve any
disagreements regarding the coding applied to the documents in the seed set.") While not all
experienced ESI counsel believe it necessary to be as transparent as MSL was willing to be, such
transparency allows the opposing counsel (and the Court) to be more comfortable with computer-
assisted review, reducing fears about the so-called "black box" of the technology.14/ This Court
highly recommends that counsel in future cases be willing to at least discuss, if not agree to, such
transparency in the computer-assisted review process.
Several other lessons for the future can be derived from the Court's resolution of the
ESI discovery disputes in this case.
First, it is unlikely that courts will be able to determine or approve a party's proposal
as to when review and production can stop until the computer-assisted review software has been
trained and the results are quality control verified. Only at that point can the parties and the Court
14/
It also avoids the GIGO problem, i.e., garbage in, garbage out.

24
see where there is a clear drop off from highly relevant to marginally relevant to not likely to be
relevant documents. While cost is a factor under Rule 26(b)(2)(C), it cannot be considered in
isolation from the results of the predictive coding process and the amount at issue in the litigation.
Second, staging of discovery by starting with the most likely to be relevant sources
(including custodians), without prejudice to the requesting party seeking more after conclusion of
that first stage review, is a way to control discovery costs. If staging requires a longer discovery
period, most judges should be willing to grant such an extension. (This Judge runs a self-proclaimed
"rocket docket," but informed the parties here of the Court's willingness to extend the discovery
cutoff if necessary to allow the staging of custodians and other ESI sources.)
Third, in many cases requesting counsel's client has knowledge of the producing
party's records, either because of an employment relationship as here or because of other dealings
between the parties (e.g., contractual or other business relationships). It is surprising that in many
cases counsel do not appear to have sought and utilized their client's knowledge about the opposing
party's custodians and document sources. Similarly, counsel for the producing party often is not
sufficiently knowledgeable about their own client's custodians and business terminology. Another
way to phrase cooperation is "strategic proactive disclosure of information," i.e., if you are
knowledgeable about and tell the other side who your key custodians are and how you propose to
search for the requested documents, opposing counsel and the Court are more apt to agree to your
approach (at least as phase one without prejudice).
Fourth, the Court found it very helpful that the parties' ediscovery vendors were
present and spoke at the court hearings where the ESI Protocol was discussed. (At ediscovery

25
programs, this is sometimes jokingly referred to as "bring your geek to court day.") Even where as
here counsel is very familiar with ESI issues, it is very helpful to have the parties' ediscovery
vendors (or in-house IT personnel or in-house ediscovery counsel) present at court conferences
where ESI issues are being discussed. It also is important for the vendors and/or knowledgeable
counsel to be able to explain complicated ediscovery concepts in ways that make it easily
understandable to judges who may not be tech-savvy.
CONCLUSION
This Opinion appears to be the first in which a Court has approved of the use of
computer-assisted review. That does not mean computer-assisted review must be used in all cases,
or that the exact ESI protocol approved here will be appropriate in all future cases that utilize
computer-assisted review. Nor does this Opinion endorse any vendor (the Court was very careful
not to mention the names of the parties' vendors in the body of this Opinion, although it is revealed
in the attached ESI Protocol), nor any particular computer-assisted review tool. What the Bar should
take away from this Opinion is that computer-assisted review is an available tool and should be
seriously considered for use in large-data-volume cases where it may save the producing party (or
both parties) significant amounts of legal fees in document review. Counsel no longer have to worry
about being the "first" or "guinea pig" for judicial acceptance of computer-assisted review. As with
keywords or any other technological solution to ediscovery, counsel must design an appropriate
process, including use of available technology, with appropriate quality control testing, to review

26
and produce relevant ESI while adhering to Rule 1 and Rule 26(b)(2)(C) proportionality. Computer-
assisted review now can be considered judicially-approved for use in appropriate cases.
SO ORDERED.
Dated: New York, New York

February 24,2012
Andrew J. Peck
United States Magistrate Judge
Copies by ECF to: All Counsel
Judge Andrew L. Carter, Jr.
G:WP\DA SILVA MOORE· ESI

EXHIBIT
SOUTHERN DISTRICT OF NEW YORK
MONIQUE DA SILVA MOORE,

MARYELLEN O'DONOHUE, LAURIE
MAYERS, HEATHER PIERCE, and
K.ATHERINE WILKINSON, on behalf of Case No. 11-cv-1279 (ALC) (AJP)
themselves and all others similarly
situated,
Plaintiffs,
PARTIES' PROPOSED PROTOCOL
vs. RELATING TO THE PRODUCTION OF
ELECTRONICALLY STORED
PUBLICIS GROUPE SA and INFORMATION ("ESI") cf'~--R.
MSLGROUP,
Defendants.
A. Scope
1. General. The procedures and protocols outlined herein govern the production of
electronically stored information ("ESI") by MSLGROUP Americas, Inc. ("MSL") during the
pendency of this litigation. The parties to this protocol will take reasonable steps to comply with
this agreed-upon protocol for the production of documents and information existing in electronic
format. Nothing in this protocol will be interpreted to require disclosure of documents or
information protected from disclosure by the attorney-client privilege, work-product product
doctrine or any other applicable privilege or immunity. It is Plaintiffs' position that nothing in
this protocol will be interpreted to waive Plaintiffs' right to object to this protocol as portions of
it were mandated by the Court over Plaintiffs' objections, including Plaintiffs' objections to the
predictive coding methodology proposed by MSL.
2. Limitations and No-Waiver. This protocol provides a general framework for
the production of ESI on a going forward basis. The Parties and their attorneys do not intend by
this protocol to waive their rights to the attorney work-product privilege, except as specifically
required herein, and any such waiver shall be strictly and narrowly construed and shall not
extend to other matters or information not specifically described herein. All Parties preserve
their attorney client privileges and other privileges and there is no intent by the protocol, or the
production of documents pursuant to the protocol, to in any way waive or weaken these
privileges. All documents produced hereunder are fully protected and covered by the Parties'
confidentiality and clawback agreements and orders of the Court effectuating same.
3. Relevant Time Period. January 1, 2008 through February 24, 2011 for all non
email ESI relating to topics besides pay discrimination and for all e-mails. January 1, 2005
through February 24, 2011 for all non-e-mail ESI relating to pay discrimination for New York
Plaintiffs.
B. ESI Preservation
1. MSL has issued litigation notices to designated employees on February 10, 2010,
March 14,2011 and June 9, 2011.
C. Sources
1. The Parties have identified the following sources of potentially discoverable ESI
at MSL. Phase I sources will be addressed first, and Phase II sources will be addressed after
Phase I source searches are complete. Sources marked as "NI A" will not be searched by the
Parties.
Active corporate system that provides e-mail N/A

mail communication and functions.
c GroupWise E- Legacy corporate system that provided e-mail N/A
• mail communication and functions .
d IBM Sametime Lotus Notes Instant Messaging and collaboration N/A
Page 2
dedicated to individual users. (With the exception of 2
horne directories for which MSL will collect and I
analyze the data to determine the level of duplication as
compared to the EMC SourceOne Archive. The parties
will meet and confer regarding the selection of the two
I
custodians. )
f Shared Folders Shared network storage locations on the file server(s) II
that are accessible by individual users, groups of users
I
or entire departments. (With the exception of the
following Human Resources shared folders which will
be in Phase I: Corporate HR, North America HR and
New York HR.)
! g i Database Servers Backend databases (e.g. Oracle, SQL, MySQL) used to N/A
store information for front end applications or other
purposes.
h Halogen Software Performance management program provided by I
Halogen to conduct performance evaluations.
I 1 I Noovoo Corporate Intranet site. . II
j Corporate E-mail addresses that employees may utilize to provide I
Feedback the company with comments, suggestions and overall
feedback.
k Hyperion Oracle application that offers global financial N/A
Financial consolidation, reporting and analysis.
Management
("HFM")
I VurvlTaleo Talent recruitment software. II
m ServiceNow Help Desk application used to track employee computer N/A
related requests.
n PeopleSoft Human resources information management system. I
0 PRISM PeopleSoft component used for time and billing I
management.
p Portal A project based portal provided through Oracle/BEA II
Systems.
q Desktops/Laptops Fixed and portable computers provided to employees to II
perform work related activities. (With the exception of
2 desktop/laptop hard drives for which MSL will collect
and analyze the data to determine the level of
duplication as compared to the EMC SourceOne
Archive. The parties will meet and confer regarding the
selection of the two custodians.)
r Publicis Benefits Web based site that maintains information about II
Connection employee benefits and related information.
s GEARS Employee expense reporting system. II
t MS&LCity Former corporate Intranet. N/A
Adium Appliclltion which aggregates instant messages. N/A
R Pidgin
IBM Lotus
Applicationyvhich aggregates instant message.
Mobile device synchronization and security system.
Page 3
N/A
N/A
i Traveler and
Mobilelron i
i y Mobile Portable PDAs, smart phones, tablets used for N/A

Communication communication.
I
i Devices
z Yammer Social media and collaboration Eortal. N/A
aa SalesForce.com Web-based customer relationship management N/A
application.
bb Removable Portable storage media, external hard drives, thumb N/A
Storage Devices drives, etc. used to sto~e copies of work related ESI.
a. EMC SourceOne - MSL uses SourceOne, an EMC e-mail archiving
system that captures and stores all e-mail messages that pass through the corporate e-mail
system. In addition, if a user chooses to save an instant messaging chat conversation from IBM
Sametime (referenced below), that too would be archived in SourceOne. Defendant MSL also
acknowledges that calendar items are regularly ingested into the SourceOne system. The parties
have agreed that this data source will be handled as outlined in section E below.
b. Lotus Notes E-mail - MSL currently maintains multiple Lotus Notes
Domino servers in various data centers around the world. All e-mail communication and
calendar items are joumaled in real time to the EMC SourceOne archive. The parties have
agreed to not collect any information from this data source at this time.
c. GroupWise E-mail Prior to the implementation of the Lotus Notes
environment, GroupWise was used for all e-mail and calendar functionality. Before the
decommissioning of the GroupWise servers, MSL created backup tapes of all servers that housed
the GroupWise e-mail databases. The parties have agreed to not collect any information from
this data source at this time.
d. IBM Sametime MSL provides custodians with the ability to have real
time chat conversations via the IBM Sametime application that is part ofthe Lotus Notes suite of
products.
Page 4
e. Home Directories Custodians with corporate network access at MSL
also have a dedicated and secured network storage location where they are able to save files.
MSL will collect the home directory data for 2 custodians and analyze the data to determine the
level of duplication of documents in this data source against the data contained in the EMC
SourceOne archive for the same custodians. (The parties will meet and confer regarding the
selection of the two custodians.) The results of the analysis will be provided to Plaintiffs so that
a determination can be made by the parties as to whether MSL will include this data source in its
production of ESI to Plaintiffs. If so, the parties will attempt to reach an agreement as to the
approach used to collect, review and produce responsive and non-privileged documents.
f. Shared Folders - Individual employees, groups of employees and entire
departments at MSL are given access to shared network storage locations to save and share files.
As it relates to the Human Resources related shared folders (i.e., North America HR Drive (10.2
OB), Corporate HR Drive (440 MB), NY HR Drive (1.9 OB), Chicago HR Drive (1.16 OB),
Boston HR Drive (43.3 MB), and Atlanta HR Drive (6.64 OB)), MSL will judgmentally review
and produce responsive and non-privileged documents from the North America HR Drive,
Corporate HR Drive, and NY HR Drive. MSL will produce to Plaintiffs general information
regarding the content of other Shared Folders. The parties will meet and confer regarding the
information gathered concerning the other Shared Folders and discuss whether any additional
Shared Folders should be moved to Phase I.
g. Database Servers -MSL has indicated that it does not utilize any database
servers, other than those that pertain to the sources outlined above in C, which are likely to
contain information relevant to Plaintiffs' claims.
h. Halogen Software MSL utilizes a third party product, Halogen, for
performance management and employee evaluations. The parties will meet and confer in order to
Page 5
exchange additional information and attempt to reach an agreement as to the scope of data and
the approach used to collect, review and produce responsive and non-privileged documents.
1. Noovoo - MSL maintains a corporate Intranet site called "Noovoo" where
employees are able to access Company-related information. MSL will provide Plaintiffs with
any employment-related policies maintained within Noovoo.
J. Corporate Feedback - MSL has maintained various e-mail addresses that
employees may utilize to provide the company with comments, suggestions and overall
feedback. These e-mail addressesinclude ..powerofone@mslworldwide.com...
"poweroftheindividual@mslworldwide.com", "townhall@mslworldwide.com" and
"whatsonyourmind@mslworldwide.com". The parties have agreed that all responsive and non
privileged ESI will be produced from these e-mail accounts and any other e-mail accounts that
fall under this category of information. At present, MSL intends to manually review the contents
of each of these e-mail accounts. However, if after collecting the contents of each of the e-mail
accounts MSL determines that a manual review would be impractical, the parties will meet and
confer as to the approach used to collect, review and produce responsive and non-privileged
documents.
k. Hyperion Financial Management ("HFM") MS L uses an Oracle
application called HFM that offers global financial consolidation, reporting and analysis
capabilities.
1. Vurv/Taleo - Since approximately 2006, MSL used an application known
as Vurv as its talent recruitment software. As of August 31, 2011, as a result of Vurv being
purchased by Taleo, MSL has been using a similar application by Taleo as its talent recruitment
software. The application, which is accessed through MSL's public website, allows users to
search for open positions as well as input information about themselves. To the extent Plaintiffs
Page 6
contend they were denied any specific positions, they will identify same and the Parties will meet
and confer to discuss what, if any, information exists within Vurv/Taleo regarding the identified
position. If information exists in Vurv/Taleo or another source regarding these positions, MSL
will produce this information, to the extent such information is discoverable.
m. ServiceNow MSL utilizes ServiceNow as its Help Desk application.
This system covers a wide variety of requests by employees for computer-related assistance (e.g.,
troubleshoot incidents, install software, etc.).
n. PeopleSoft - MSL utilizes PeopleSoft, an Oracle-based software product,
to record employee data such as date of hire, date of termination, promotions, salary increases,
transfers, etc. MSL has produced data from this source and will consider producing additional
data in response to a specific inquiry from Plaintiffs.
o. PRISM MSL utilizes PRISM for tracking time and billing. It is used
primarily to track an employee's billable time. MSL will consider producing additional data in
response to a specific inquiry from Plaintiffs.
p. Portal - MSL maintains a portal provided through Oracle/BEA Systems.
The portal is web-based and is used for light workflow activities (such as reviewing draft
documents).
q. Desktops/Laptops MSL provided employees with desktop and/or laptop
computers to assist in work related activities. MSL will collect the desktop/laptop hard drive
data for 2 custodians and analyze the data to determine the level of duplication of documents in
this data source against the data contained in the EMC SourceOne archive for the same
custodians. (The parties will meet and confer regarding the selection of the two custodians.)
The results of the analysis will be provided to Plaintiffs so that a determination can be made by
the parties as to whether MSL will include this data source in its production of ESI to Plaintiffs.
Page 7
If so, the Parties will attempt to reach an agreement as to the approach used to collect, review
and produce responsive and non-privileged documents.
r. Publicis Benefits Connection Plaintiffs understand that MSL provides
employees with access to a centralized web based site that provides access to corporate benefits
information and other related content.
s. GEARS - MSL maintains a centralized web-based expense tracking and
reporting system called "GEARS" where users are able to enter expenses and generate reports.
1. MS&L City - MSL maintained a corporate web-based Intranet prior to
migrating to Noovoo.
u. Adium - This is a free and open source instant messaging client for Mac
OS X users.
v. Pidgin - Pidgin is a chat program which lets users log into accounts on
multiple chat networks simultaneously. However, the data resides with a third party messaging
provider (e.g. AIM, Yahoo!, Google Talk, MSN Messenger, etc.).
w. IBM Lotus Traveler and Mobilelron MSL maintains these systems for e
mail device sync and security features for employees' mobile devices, including Blackberry
devices, iPhones, iPads, Android phones, and Android tablets.
x. Mobile Communication Devices - MSL provides mobile devices and/or
connectivity including Blackberry devices, iPhones, iPads, Android phones, and Android tablets
to designated employees.
y. Yammer This is an instant messaging application hosted externally, used
for approximately one year in or around 2008 through 2009.
z. SalesForce.com This is a web-based customer relationship management
application but it was not widely used.
Page 8
aa. Removable Storage Devices - MSL does not restrict authorized
employees from using removable storage devices.
D. Custodians
1. The Parties agree that MSL will search the e-mail accounts of the following
individuals as they exist on MSL's EMC SourceOne archive. (Except where a date range is
noted, the custodian's entire e-mail account was collected from the archive.)
Page 9
30. I Hass, Mark CEO (former)
31. I Morsman, Michael Managing Director, Ann Arbor (former)
E. Search Methodologyl
1. General. The Parties have discussed the methodologies or protocols for the
search and review of ESI collected from the EMC SourceOne archive and the following is a
summary of the Parties' agreement on the use of Predictive Coding. This section relates solely
to the EMC SourceOne data source (hereinafter referred to as the ..e-mail collection").
2. General Overview of Predictive Coding Process. MSL will utilize the Axcelerate
software by Recommind to search and review the e-mail collection for production in this case.
The process begins with Jackson Lewis attorneys developing an understanding of the
entire e-mail collection while identifying a small number of documents, the initial seed set, that
is representative of the categories to be reviewed and coded (relevance, privilege, issue-relation).
It is the step when the first seed sets are generated which is done by use of search and analytical
tools, including keyword, Boolean and concept search, concept grouping, and, as needed, up to
40 other automatically populated filters available within the Axcelerate system. This assists in
the attorneys' identification of probative documents for each category to be reviewed and coded.
Plaintiffs' counsel will be provided with preliminary results of MSL's hit counts using
keyword searches to create a high priority relevant seed set, and will be invited to contribute
their own proposed keywords. Thereafter, Plaintiffs' counsel will be provided with the non-
privileged keyword hits - both from MSL's keyword list and Plaintiffs' keyword list which
were reviewed and coded by MSL. Plaintiffs' counsel will review the documents produced and
promptly provide defense counsel with their own evaluation of the initial coding applied to the
documents, including identification of any documents it believes were incorrectly coded. To the
1 Asnoted in Paragraphs A(l) and J ofthis Protocol, Plaintiffs object to the predictive coding
methodology proposed by MSL.
Page 10
extent the parties disagree regarding the coding of a particular document, they will meet and
confer in an effort to resolve the dispute prior to contacting the Court for resolution. The
irrelevant documents so produced shall be promptly returned after review and analysis by
Plaintiffs' counsel and/or resolution of any disputes by the Court.
The seed sets are then used to begin the Predictive Coding process. Each seed set of
documents is applied to its relevant category and starts the software ''training'' process. The
software uses each seed set to identify and prioritize all substantively similar documents over the
complete corpus of the e-mail collection. The attorneys then review and code a judgmental
sample of at least 500 of the "computer suggested" documents to ensure their proper
categorization and to further calibrate the system by recoding documents into their proper
categories. Axcelerate learns from the new corrected coding and the Predictive Coding process
is repeated.
Attorneys representing MSL will have access to the entire e-mail collection to be
searched and wi11lead the computer training, but they will obtain input from Plaintiffs' counsel
during the iterative seed selection and quality control processes and will share the information
used to craft the search protocol as further described herein. All non-privileged documents
reviewed by MSL during each round of the iterative process (i.e., both documents coded as
relevant and irrelevant) will be produced to Plaintiffs' counsel during the iterative seed set
selection process. Plaintiffs' counsel will review the documents produced and promptly provide
defense counsel with its own evaluation of the initial coding applied to the documents, including
identification of any documents it believes were incorrectly coded. To the extent the Parties
disagree regarding the coding of a particular document, they will meet and confer in an effort to
resolve the dispute prior to contacting the Court for resolution. Again, the irrelevant documents
Page 11
so produced shall be promptly returned after review and analysis by Plaintiffs' counsel andlor
resolution of any disputes by the Court.
At the conclusion of the iterative review process, all document predicted by Axcelerate
to be relevant will be manually reviewed for production. However, depending on the number of
documents returned, the relevancy rating of those documents, and the costs incurred during the
development of the seed set and iterative reviews, MSL reserves the right to seek appropriate
relief from the Court prior to commencing the final manual review.
The accuracy of the search processes, both the systems' functions and the attorney
judgments to train the computer, will be tested and quality controlled by both judgmental and
statistical sampling. In statistical sampling, a small set of documents is randomly selected from
the total corpus of the documents to be tested. The small set is then reviewed and an error rate
calculated therefrom. The error rates can then be reliably projected on the total corpus, having a
margin of error directly related to the sample size.
3. Issue Tags. The parties agree that, to the extent applicable, as part of the
seed set training described above, as well as during the iterative review process, all documents
categorized as relevant and not privileged, to the extent applicable, also shall be coded with one
or more of the following agreed-upon issue tags:
a. Reorganization.
b. Promotion!Assignments.
c. Work/Life Balance.
d. Termination.
e. Compensation.
f. MatemitylPregnancy.
g. ComplaintslHR.
Page 12
h. Publicis Groupe/Jurisdiction.
This issue coding will take place during the initial random sample, creation of the seed set and
initial and iterative training (see paragraphs 4, 5 and 6 below). This input shall be provided to
Plaintiffs' counsel along with the initial document productions. Plaintiffs' counsel shall
promptly report any disagreements on classification, and the parties shall discuss these issues in
good faith, so that the seed set training may be improved accordingly. This issue-tagging and
disclosure shall take place during the described collaborative seed set training process. The
disclosures here made by MSL on its issue coding are not required in the final production set.
4. Initial Random Sample. Using the Axcelerate software to generate a random
sample ofthe entire corpus of documents uploaded to the Axcelerate search and review platform,
MSL's attorneys will conduct a review of the random sample for relevance and to develop a
baseline for calculating recall and precision. To the extent applicable, any relevant documents
also will be coded with one or more of the issue tags referenced in paragraph E.3 above. The
random sample consists of 2,399 documents, which represents a 95% confidence level with a
confidence estimation of plus or minus 2%. The Parties agree to utilize the random sample
generated prior to the finalization of this protocol. However, during Plaintiffs' counsel's review
of the random sample, they may advise as to whether they believe any of the documents should
be coded with one or more of the subsequently added issue codes (i.e., ComplaintslHR and
Publicis Groupe/Jurisdiction) and will, as discussed above, indicate any disagreement with
MSL's classifications.
5. Seed Set.
a. Defendant MSL. To create the initial seed set of documents that will be
used to "train" the Axcelerate software as described generally above, MSL primarily utilized
keywords listed on Exhibits A and B to this protocol, but also utilized other judgmental analysis
Page 13
and search techniques designed to locate highly relevant documents, including the Boolean,
concept search and other features ofAxcelerate. Given the volume of hits for each keyword
(Exhibit A), MSL reviewed a sampling of the hits and coded them for relevance as well as for
the following eight preliminary issues: (i) Reorganization; (ii) Promotion; (iii) Work/Life
Balance; (iv) Termination; (v) Compensation; and (vi) Maternity. Specifically, except for key
words that were proper names, MSL performed several searches within each set of key word hits
and reviewed a sample of the hits. The Axcelerate software ranked the hits in order of relevance
based on the software's analytical capabilities and the documents were reviewed in decreasing
order of relevance (i.e., each review of the sample of supplemental searches started with the
highest ranked documents). Exhibit B identifies the supplemental searches conducted, the
number of hits, the number of documents reviewed, the number of documents coded as
potentially responsive and general comments regarding the results. In addition, to the extent
applicable, documents coded as responsive also were coded with one or more issue tags. MSL
will repeat the process outlined above and will include the newly defined issues and newly added
custodians. MSL will provide Plaintiffs' counsel with all of the non-privileged documents and
will provide, to the extent applicable, the issue tag(s) coded for each document, as described
above. Plaintiffs' counsel shall promptly review and provide notice as to any documents with
which they disagree where they do not understand the coding. If necessary, counsel will meet
and confer to attempt to resolve any disagreements regarding the coding applied to the
documents in this seed set.
b. Plaintiffs. To help create the initial seed set of documents that will be
used to "train" the Axcelerate software, Plaintiffs provided a list of potential key words to MSL.
MSL provided Plaintiffs with a hit list for their proposed key words. This process was repeated
twice with the hit list for Plaintiffs' most recent set of keywords attached as Exhibit C. MSL
Page 14
will review 4,000 randomly sampled documents from Plaintiffs' supplemental list of key words
to be coded for relevance and issue tags. MSL will provide Plaintiffs' counsel with all non
privileged documents and will provide, to the extent applicable, the issue tag(s) coded for each
document. Plaintiffs' counsel shall promptly review and provide notice as to any documents
with which they disagree with or where they do not understand the coding. If necessary, the
Parties' counsel will meet and confer to attempt to resolve any disagreements regarding the
coding applied to the documents in this seed set.
c. Judgmental Sampling. In addition to the above, a number of targeted
searches were conducted by MSL in an effort to locate documents responsive to several of
Plaintiffs' specific discovery requests. Approximately 578 documents have already been coded
as responsive and produced to Plaintiffs. In addition, several judgmental searches were
conducted which resulted in approximately 300 documents initially being coded as responsive
and several thousand additional documents coded as irrelevant. The documents coded as
relevant and non-privileged also will be reviewed by Plaintiffs' counsel and, subject to their
feedback, included in the seed set. An explanation shall be provided by MSL's attorneys for the
basis of the bulk tagging of irrelevant documents (primarily electronic periodicals and
newsletters that were excluded in the same manner as spam junk mail is excluded). The
explanation shall include the types of documents bulk tagged as irrelevant as well as the process
used to identify those types of documents and other similar documents that were bulk tagged as
irrelevant.
6. Initial And Iterative Training. Following the creation of the first seed set, the
Axcelerate software will review the entire data set to identify other potentially relevant
documents. MSL will then review and tag a judgmental based sample, consisting of a minimum
of 500 documents, including all documents ranked as highly relevant or hot, of the new
Page 15
"Computer Suggested" documents, which were suggested by the Axcelerate software. MSL's
attorneys shall act in consultation with the Axcelerate software experts to make a reasonable,
good faith effort to select documents in the judgmental sample that will serve to enhance and
increase the accuracy of the predictive coding functions. The results of this first iteration, both
the documents newly coded as relevant and not relevant for particular issue code or codes, will
be provided to Plaintiffs' counsel for review and comment. (All documents produced by the
parties herein to each other, including, without limitation, these small seed set development
productions, shall be made under the Confidentiality Stipulation in this matter as well as any
clawback agreement that shall be reduced to an order acceptable to the Court. Any documents
marked as irrelevant shall be returned to counsel for MSL at the conclusion of the iterative
training phase, unless the relevancy of any documents are disputed, in which case they may be
submitted to the Court for review.)
Upon completion of the initial review, and any related meet and confer sessions and
agreed upon coding corrections, the Axcelerate software will be run again over the entire data set
for suggestions on other potentially relevant documents following the same procedures as the
first iteration. The purpose of this second and any subsequent iterations of the Predictive Coding
process will be to further refine and improve the accuracy of the predictions on relevance and
various other codes. The results of the second iteration shall be reviewed and new coding shared
with Plaintiffs' counsel as described for the first iteration. This process shall be repeated five
more times, for a total of seven iterations, unless the change in the total number of relevant
documents predicted by the system as a result of a new iteration, as compared to the last
iteration, is less than five percent (5%), and no new documents are found that are predicted to be
hot (aka highly relevant), at which point MSL shall have the discretion to stop the iterative
process and begin the final review as next described. If more than 40,000 documents are
Page 16
returned in the final iteration, then MSL reserves the right to apply to the Court for relief and
limitations in its review obligations hereunder. Plaintiffs reserve the right, at all times, to
challenge the accuracy and reliability of the predictive coding process and the right to apply to
the Court for a review of the process.
7. Final Search and Production. All ofthe documents predicted to be relevant in the
final iteration described in paragraph six above will be reviewed by MSL, unless it applies to the
court for relief hereunder. All documents found by MSL's review to be relevant and non-
privileged documents will be promptly produced to Plaintiffs. If more than 40,000 documents
are included in the final iteration, then MSL reserves its right to seek payment from Plaintiffs for
all reasonable costs and fees MSL incurred related to the attorney review and production of more
40,000 documents. ~is-,:m:rvtsjtm:r;S-Im1:-mterrde{tmnrWR'ivernv-f';lfSi7ttnrlmrsee:lM:lfl-t~11l.i of
all discovery costs at the
conclusion of this action under 28 U.S.C. (4) and Rule 54(d)(l) Federal Rules of Civil
Procedure. [P~ to the inclusion of MSL's proposed cost-shifting language as
prem~~--;~d argumentative, in contravention of the Court's January 4, 2012 and

/
~i Uft", 8, 2012 cn:ders Plailltiffs belieye costs should be subjed to a separate bearing J
8. Quality Control by Random Sample of Irrelevant Documents. In addition, at the
conclusion of this search protocol development process described above, and before the final
search and production described in Paragraph 7 above, MSL will review a random sample of
2,399 documents contained in the remainder of the database that were excluded as irrelevant.
The results of this review, both the documents coded as relevant and not relevant, but not
privileged, will be provided to Plaintiffs' counsel for review. (Any documents initially coded as
"not relevant" will be provided subject to the Confidentiality Stipulation and any clawback
agreements entered in this matter will be returned to counsel for MSL within 60 days of their
Page 17
production.) The purpose for this review is to allow calculation of the approximate degree of
recall and precision of the search and review process used. If Plaintiffs object to the proposed
review based on the random sample quality control results, or any other valid objection, they
shall provide MSL with written notice thereof within five days of the receipt of the random
sample. The parties shall then meet and confer in good faith to resolve any difficulties, and
failing that shall apply to the Court for relief. MSL shall not be required to proceed with the final
search and review described in Paragraph 7 above unless and until objections raised by Plaintiffs
have been adjudicated by the Court or resolved by written agreement of the Parties.
F. Costs
I. MSL proposes to limit the costs of its final review and production of responsive
ESI from the MSL email collection to an additional $200,000, above and beyond the
approximately $350,000 it has already paid or is anticipated to pay in e-discovery related
activities as previously described and disclosed to Plaintiffs. -~;pp;~"!1itv-...,..;H+wTrm1"f'"'"4~I;;!L
coding process described in paragraphs 6 and
right to seek relief from the ourt (e.g., a cost shifting rd and/or ruling that MSL need to
",,/
review more than a specified number 0 ocume~~rsuant to the principles of proportionality.

See Rule 1, Rule 26(b)(2)(C), Ru!y46(b)(2J Rule 26(g), Federal Rules of Civil
/'
/'
/~
Procedure; Commentary o~Pfoportionality in Electronic . covery, 11 SEDONA CONF. J. 289

/'
(201 0); Oot, et a~dating Reasonableness in a Reasonable Inq .
Review, 8J~i:/;22-559 (2010); Also see Rule 403 of the Federal Evidence Co
./'
(inadmissibility
of.cumulative evidence). [Plaintiffs object to the inclusion of MSL's proposed cos ifting
language as premature and argumentative, in contraven Ion
Page 18
and- FelJrual y 8, 201% orders. "Plahdiffs beHeve costs should be subject to a separate
-------~--
2. Plaintiffs agree to bear all of the costs associated with their compliance with the
terms of this protocol and with the receipt and review of ESI produced hereunder including the
costs associated with its ESI experts at DOAR Litigation Consulting who will be involved with
Plaintiffs in all aspects of this ESI protocol. Plaintiffs propose that MSL bear all of the costs
associated with its obligations under the terms of this protocol and do not agree to limit the
amount of information subject to the review and production of ESI by MSL.
G. Format of Production For Documents Produced From Axcelerate
1. TIFFlNative File Format Production. Documents will be produced as single-
page TIFF images with corresponding multi-page text and necessary load files. The load files
will include an image load file as well as a metadata (.DAT) file with the metadata fields
identified on Exhibit D. Defendant MSL will produce spreadsheets (.xls files) and PowerPoint
presentations (.ppt files) in native form as well as any documents that cannot be converted to
TIFF format (e.g., audio or video files, such as mp3s, wavs, megs, etc.). In addition, for any
redacted documents that are produced, the documents' metadata fields will be redacted where
required. For the production of ESI from non-email sources, the parties will meet and confer to
attempt to reach an agreement of the format of production.
2. Appearance. Subject to appropriate redaction, each document's electronic image
will convey the same information and image as the original document. Documents that present
imaging or formatting problems will be promptly identified and the parties will meet and confer
in an attempt to resolve the problems.
3. Document Numbering. Each page of a produced document will have a legible,
unique page identifier "Bates Number" electronically "burned" onto the image at a location
Page 19
that does not obliterate, conceal or interfere with any infonnation from the source document. The
Bates Number for each page of each document will be created so as to identify the producing
party and the document number. In the case of materials redacted in accordance with applicable
law or confidential materials contemplated in any Confidentiality Stipulation entered into by the
parties, a designation may be "burned" onto the document's image at a location that does not
obliterate or obscure any infonnation from the source document.
4. Production Media. The producing party will produce documents on
readily accessible, computer or electronic media as the parties may hereafter agree upon,
including CD-ROM, DVD, external hard drive (with standard PC compatible interface), (the
"Production Media"). Each piece of Production Media will be assigned a production number
or other unique identifying label corresponding to the date of the production of documents on the
Production Media (e.g., "Defendant MSL Production April 1,2012") as well as the sequence of
the material in that production (e.g. "-001", "-002"). For example, if the production comprises
document images on three DVDs, the producing party may label each DVD in the following
manner "Defendant MSL Production April 1,2012", "Defendant MSL Production April 1,2012
002", "Defendant MSL Production April 1, 2012-003." Additional infonnation that will be
identified on the physical Production Media includes: (1) text referencing that it was produced
in da Silva Moore v. Publicis Groupe SA, et al.; and (2) the Bates Number range of the
materials contained on the Production Media. Further, any replacement Production Media will
cross-reference the original Production Media and clearly identify that it is a replacement and
cross-reference the Bates Number range that is being replaced.
5. Write Protection and Preservation. All computer media that is capable of write
protection should be write-protected before production.
Page 20
6. Inadvertent Disclosures. The tenus of the Parties' Clawback Agreement and Court
Order shall apply to this protocoL
7. Duplicate Production Not Required. A party producing data in electronic form
need not produce the same document in paper format.
H. Timing.
1. To the extent a timeframe is not specifically outlined herein, the parties will use
their reasonable efforts to produce ESI in a timely manner consistent with the Court's discovery
schedule.
2. The parties will produce ESI on a rolling basis.
I. General Provisions.
1. Any practice or procedure set forth herein may be varied by agreement of the
parties, and first will be confirmed in writing, where such variance is deemed appropriate to
facilitate the timely and economical exchange of electronic data.
2. Should any party subsequently determine it cannot in good faith proceed as
required by this protocol, the parties will meet and confer to resolve any dispute before seeking
Court intervention.
3. The Parties agree that e-discovery will be conducted in phases and, at the
conclusion of the search process described in Section E above, the Parties will meet and confer
regarding whether further searches of additional custodians and/or the Phase II sources is
warranted and/or reasonable. If agreement cannot be reached, either party may seek relief from
the Court.
Page 21
J. Plaintiffs' Objection
1. Plaintiffs object to this ESI Protocol in its entirety. Plaintiffs submitted their own
proposed ESI Protocol to the Court, but it was largely rejected. The Court then ordered the
parties to submit a joint ESI Protocol reflecting the Court's rulings. Accordingly, Plaintiffs
jointly submit this ESI Protocol with MSL, but reserve the right to object to its use in this case.
This protocol may be executed in counterparts. Each counterpart, when so executed,
will be deemed and original, and will constitute the same instrument.
By: ______________________ By: __________________________

JANETTE WIPPER, ESQ. BRETT M. ANDERS, ESQ.
DEEPIKA BAINS, ESQ. VICTORIA WOODIN CHAVEY, ESQ.
SIHAM NURHUSSEIN, ESQ. JEFFREY W. BRECHER, ESQ.
JACKSON LEWIS LLP
SANFORD WITTELS & HEISLER, LLP Attorneys for Defendant MSLGROUP
Attorneys for Plaintiffs and Class 58 South Service Road, Suite 410
555 Montgomery Street, Ste. 1206 Melville, NY 11747
San Francisco, CA 94111 Telephone: (631) 247-0404
Telephone: (415) 391-6900
Date: ___________________, 2012
Date: _____________:, 2012
Page 22
Taking the Plunge: Judge Peck Issues First Decision Endorsing Computer-Assisted ESI : ... Page 1 of 3

Posted at 12:40 PM on March 2, 2012 by Phillip J. Duffy
Taking the Plunge: Judge Peck Issues First Decision Endorsing Computer-Assisted
ESI
Late last year, Magistrate Judge Andrew Peck of the U.S. District Court for the Southern District of New
York, one of the most prominent judicial thought leaders in e-discovery, wrote an article entitled Search,
Forward in which he opined that computer-assisted ESI review "should be used where it will help
'secure the just, speedy and inexpensive' (Fed. R. Civ. P. 1) determination of cases", but he forecast that
lawyers awaiting a judicial opinion endorsing predictive coding might have "a long wait." As it turns
out, the wait wasn't very long at all; on Friday, February 24, 2012, less than 6 months after the
publication of his article, Judge Peck himself issued the first judicial opinion approving the use of
predictive coding "in appropriate cases."
Judge Peck issued his opinion in Monique Da Silva Moore, et al., v. Publicis Groupe & MSL Group,
Civ. No. 11-1279 (ALC)(AJP) (S.D.N.Y. February 24, 2012), a gender discrimination suit brought by
five female plaintiffs against Publicis Groupe, "one of the world's 'big four advertising conglomerates,'"
and MSL Group, its U.S. public relations subsidiary. Faced with more than 3 million documents to be
reviewed, the parties agreed to use predictive coding -- a process involving senior attorney review and
coding of a "seed set" of documents that are then used to train a computer to search the entire data set
for relevant documents and cull them -- but they disagreed regarding the methodology.
Defendants' ESI proposal contemplated a series of "iterative rounds" to test and refine the searches and
stabilize the training of the search software, with production to plaintiffs of both relevant and irrelevant
documents returned by the refined searches. Plaintiffs would then be invited to provide feedback to
allow further refinement of the searches. Plaintiffs objected to defendants' protocol and submitted their
own, which the Court apparently rejected.
In addressing plaintiffs' objections, Judge Peck underscored the cost-savings rationale expressed in his
article and carried over to his decision: "computer-assisted review 'works better than most of the
alternatives, if not all of the [present] alternatives. So the idea is not to make this perfect, it's not going to
be perfect. The idea is to make it significantly better than the alternatives without nearly as much cost.'"
Consistent with Judge Peck's directive, the parties submitted a Joint ESI Protocol, which Judge Peck "so
ordered " despite plaintiffs' continuing objections.
On February 22, 2012, plaintiffs filed objections to Judge Peck's rulings with District Judge Carter. In
sum, plaintiffs contended Judge Peck's acceptance of defendants' protocol allowed defendants to violate
FRCP 26(g)'s requirement to certify defendants' document production is complete and further is not
sufficiently reliable to pass muster under Federal Rule of Evidence 702 and the U.S. Supreme Court's
http://www.ediscoverylawalert.com/2012/03/articles/legal-decisions-court-rules/taking-the... 3/27/2013
decision in Daubert. In the opinion, Judge Peck addressed and ultimately dismissed plaintiffs'
objections. The Court reasoned that plaintiffs misunderstood Rule 26(g), which does not require such
certification and observed that it would be impossible to certify the completeness of a document
production of this magnitude. Moreover, Judge Peck stated that Rule 702 and Daubert are inapplicable
to the manner in which parties search for documents in discovery. Judge Peck also explained, in detail,
the reasons for his decision to endorse predictive coding in appropriate cases and took the time to offer
some "lessons for the future."
In sum, the Court found the use of predictive coding appropriate in Da Silva Moore based on the parties'
agreement with the concept, the vast data set of more than 3 million documents, the superiority of
computer-assisted review to the alternatives (manual review or keyword searches) the need for cost
effectiveness and proportionality under FRCP 26(b)(2)(C) and the “transparent” process defendants
proposed.
In this regard, Judge Peck stressed the importance of cooperation among counsel and reiterated the
Court's endorsement of The Sedona Conference® Cooperation Proclamation. Moreover, and
notwithstanding plaintiffs' expressed concerns, Judge Peck praised defendants' transparency and credited
their agreement to provide to plaintiffs all of the documents used to create the seed set, both relevant and
non-relevant, as essentially paving the way for the Court's decision.
So, what does all of this mean? Is predictive coding the only way to proceed with large scale document
review in the future? Will it eliminate the need for attorney involvement in the document review
process? Is computer assisted review the panacea that frustrated and cash strapped litigants have been
searching for to solve the incredible cost and manpower burdens of e-discovery. Well, the short answer
to these questions is "no." As Judge Peck made clear in his opinion, the Court did not order the use of
predictive coding, the parties agreed to it (at least some form of it); there is no requirement that
computer-assisted review be used in all cases. Nor, the Court was careful to note, is Da Silva Moore a
pronouncement that the ESI Protocol used in that case (which is attached to the opinion will be
appropriate in other cases that use predictive coding. Finally, the Court avoided endorsing any particular
e-discovery service provider or technology.
As for the continued need for attorney involvement in the review process, as noted in this blog's prior
June 2011 posting, while e-discovery software may make lawyers more efficient, "human knowledge,
reaction and intuition as to facts, issues and nuances of legal theories make the role of the live attorney
indispensable." Indeed, Judge Peck acknowledged as much in his opinion: "[t]he Court recognizes that
computer-assisted review is not a magic, Staples-Easy-Button, solution appropriate for all cases. The
technology exists and should be used where appropriate, but it is not a case of machine replacing
humans: it is the process used and the interaction of man and machine that the courts need to examine."
Da Silva Moore is most significant in that it opens the door for parties to consider the use of computer
assisted review or predictive coding in appropriate cases without the fear that a party or judge will resist
on the basis that no court has accepted the use of this modern technology in the e-discovery process. The
courts have now officially “taken the plunge,” and we predict that the pool will get crowded quickly.
Phillip J. Duffy is a Director on the Gibbons E -Discovery Task Force.
Comments (0)
Gibbons P.C.
gibbonslaw.com
l Newark
l New York
l Philadelphia
l Trenton
l Wilmington
11 Predictive Coding Lessons from
Judge Peck’s Opinion in Da Silva Moore
By Matthew F. Knouff proposed by plaintiffs), Boolean operators, conceptual

searches, and similar methods, to identify documents to fur-
This article was extracted from a four-part series published online on
the Lawyer to Lawyer blog in March 2012. The ideas and opinions ex- ther develop the seed sets.
pressed in this article are solely the opinions of the author and do
not necessarily reflect the position of Magistrate Judge Andrew J. Peck, II. The non-privileged results from this initial phase would be
CDS Legal, or any other person or entity. reviewed by plaintiffs’ counsel with any disputes regarding
coding being subject to conference. The resultant sets would
One would be hard-pressed to find a legal topic hotter than pre- be the initial seed sets used to code the full document uni-
dictive coding at the moment. Whether your buzz term of choice verse.
is “computer-assisted review,” “technology-assisted review,”
“predictive coding,” or “computer-assisted coding,” the debate III. The initial seed sets would be run against the entire corpus to
over the use of this new technology has revolved around miscon- identify similar documents. MSL would then review sample
ceptions of its role in the discovery process and fears about its sets of at least 500 computer-coded documents to evalu-
efficacy in comparison to other methods of search and retrieval. ate coding decisions, make any changes, and further train
In Monique Da Silva Moore, et. al. v. Publicis Groupe & MSL Group the system. At least seven iterations of refinement would be
No. 11 Civ. 1279 (ALC) (AJP) (S.D.N.Y. Feb 24, 2012), the Honorable performed. Plaintiffs would receive all documents reviewed
Andrew J. Peck, United States Magistrate Judge for the Southern during each iteration, allowing them the opportunity to par-
District of New York, gives us the first opinion approving the use ticipate in calibrating the predictive process.
of computer-assisted review, which in his article Search, Forward
IV. The entire set of relevant documents following the final itera-
from the October 2011 edition of Law Technology News he defines
tion would be reviewed by defense counsel. MSL proposed
as, “tools…that use sophisticated algorithms to enable the com-
cost-shifting for the review and production of documents
puter to determine relevance, based on interaction with (i.e. train-
exceeding the 40,000 mark.
ing by) a human reviewer.” Those who looked to Judge Peck’s
commentary in Search, Forward as a “sign of judicial approval” V. MSL would perform quality control by reviewing a random
now have a ruling which can begin to assuage the fears associ- sample of documents excluded during the computer-assisted
ated with breaking from the costly and oft-ineffective traditions of review process as being non-responsive. These documents
exhaustive human review and search methodologies solely built would be provided to plaintiffs for review to determine if the
on keywords. levels of recall and precision are acceptable.
In Da Silva Moore, a gender discrimination case, the parties
discussed an electronic discovery protocol which included the
use of predictive coding. A dispute arose not over the use of the In evaluating the ESI protocol and plaintiffs’ objections to the
actual technology, but rather over the protocol surrounding its Court’s acceptance thereof, Judge Peck debunked myths, reiter-
use. In brief, the ESI protocol proposed by the defendant (MSL), ated some old lessons, and provided counsel with much needed
and objected to in its entirety by the plaintiffs, included the fol- guidance. In an age where increasing volumes of ESI highlight
lowing steps: the shortcomings of traditional search and retrieval methods, the
need to effectuate the “just, speedy, and inexpensive determi-
I. MSL would use technology to generate a random sample nation” requires that legal practitioners adapt accordingly. Com-
set from the document universe (2,399 documents out of puter-assisted review can provide counsel with a means to tackle
approximately three million). This sample would be reviewed the realities of modern litigation discovery head on. Whether you
and coded according to certain categories (e.g. relevance, decide to take the leap into computer assisted review technology
privilege, and issue-relation), beginning the process of cre- now that you “no longer have to worry about being the ‘first’ or
ating the seed sets used to train the computer. In addition, a ‘guinea pig’” as Judge Peck comments, or decide to resist the
defense counsel would use “judgmental sampling” tech- Matrix with all your might, it behooves you to consider eleven key
niques that leveraged the use of keywords (including those lessons that can be gleaned from Peck’s opinion:
1. Technology-assisted review is as effective, if not more so,
than manual review. The long-held belief that manual review
Matthew F. Knouff is the General Counsel & eDiscovery Coun- is the most effective method for finding all the relevant docu-
sel for Complete Discovery Source. For more information visit ments during discovery is a myth. Although humans are very
www.cdslegal.com.
continued next page

smart, and have big brains that rival the reasoning power 4. FRE 702 and Daubert are not applicable to search method-
of any computer, they get fatigued, they make mistakes, ology during discovery. In holding that Defendant’s ESI pro-
and they can produce wildly divergent decisions even when tocol did not run afoul of FRE 702, Judge Peck highlighted
evaluating the same documents. Judge Peck sites to vari- that FRE 702 and Daubert apply to the admissibility of
ous scholarly works to support his opinion that “statistics expert testimony at trial, not to search and retrieval meth-
clearly show that computerized searches are at least as odology during the discovery process.
accurate, if not more so, than manual review.”
5. Humans are not being replaced by technology. A common,
2. The standard for review is not perfection; rather the pro- though cartoonish, concern involves a veritable doomsday
cess should be better than the alternatives at a lower cost. scenario where attorneys are replaced by machines and
A perfect methodology for returning 100% of all the relevant the legal profession as we know it ceases to exist. Counsel
information in a case has never existed, and probably will can rest easy. Predictive coding is meant to help humans
never exist. Judge Peck explicitly concedes that computer- practice law. Actual people are necessary to “train” the
assisted review is not perfect, and reiterates Judge Shira system by selecting a seed set, reviewing resultant sets,
Scheindlin’s Pension Committee decision that parties are and refining the protocol. Keep in mind that counsel still
not expected to meet such a standard. Rather, the aim is to has to use the information to build a case. We are a long
“’secure the just, speedy, and inexpensive determination’ way from robots making courtroom appearances. Think of
of lawsuits” reinforced by the FRCP 26(b)(2)(C) proportion- technology-assisted review technology as another tool in
ality doctrine. In comparing predictive coding to keyword your arsenal; albeit a tool that only works effectively within
searching, Judge Peck illustrates numerous shortcomings a defined process. As Judge Peck stated, “The technology
associated with keywords including the inherent problems exists and should be used where appropriate, but it is not
in selecting terms without having the requisite knowledge a case of machine replacing humans: it is the process used
about a dataset, over-inclusiveness, and their usual lack and the interaction of man and machine that the court
of effectiveness. He goes on to state, “Computer assisted needs to examine.”
review appears to be better than the available alternatives.”
6. FRCP 26(g)(1) does not require counsel to certify that a
3. Proportionality plays a key role in developing a protocol production is “complete.” In objecting to Judge Peck’s
around computer-assisted review, but is evaluated in light acceptance of defendant’s ESI protocol, plaintiff argued
of the results. Judge Peck rejected the part of Defendant’s that the court provided “unlawful cover” for defense coun-
proposed ESI protocol that limited review and production sel who has a duty to certify the “completeness” of docu-
to the top 40,000 documents returned by the ESI protocol. ment productions under FRCP 26(g)(A)(1). First, Judge Peck
He stated, “…[p]roportionality requires consideration of highlighted plaintiffs’ erroneous reliance on this subsec-
results as well as costs. And if stopping at 40,000 is going tion which applies to initial disclosures. FRCP 26(g)(1)(B)
to leave a tremendous number of highly likely responsive applies to discovery and incorporates proportionality under
documents unproduced, [MSL’s proposed cutoff] doesn’t FRCP 26(b)(2)(C) as opposed to a requirement for “com-
work.” The threshold for where a producing party can rely pleteness.”
on technology to corral an unnecessarily high review and
production burden depends on an analysis of the results 7. Cooperation and transparency are important consider-
rather than setting an arbitrary cap. This does not mean that ations for gaining acceptance from the bench. Judge Peck
early agreement on such a cap is not possible or even desir- noted his strong endorsement of the Sedona Conference
able, but costs “cannot be considered in isolation from the Cooperation Proclamation and stated, “the best solution
results of the predictive coding process and the amount at in the entire area of electronic discovery is cooperation
issue in the litigation.” A court’s ability to make proportion- among counsel.” The defendant MSL was extremely trans-
ality determinations is also dependent upon developments parent with its ESI protocol in that “plaintiff [could] see
within the case. Such issues may include the number how [defendant] has coded every email used in the seed
of documents ultimately produced, the costs involved, set (both relevant and non-relevant).” Judge Peck went on
the discovery of additional “hot documents,” which may to state that defendant’s “transparency in its proposed ESI
require refinement of the seed set and additional “training” search protocol made it easier for the Court to approve the
iterations, and the expansion of the scope of discovery. use of predictive coding.”
continued next page

8. Leverage your client’s knowledge to help develop the search several other factors that can weigh on such a decision. How-
protocol. Parties often have the best source of information ever, I submit that it is important to at least evaluate every
about another party’s documents right under their noses: case to determine if it would be an appropriate solution. If it
their client. Judge Peck expressed surprise that “in many can help cut costs, defensibly streamline the discovery pro-
cases counsel do not appear to have sought and utilized cess, reduce burden on all parties (including the court), and
their client’s knowledge about the opposing party’s cus- identify relevant information more effectively than any alter-
todians and document resources.” He went on to make a native method, there would appear to me to be a strong case
comment that, hopefully, will shock most practitioners: “[C] for using computer-assisted review.
ounsel for the producing party often is not sufficiently knowl-
edgeable about their own client’s custodians and business 11. Rely on a trusted vendor to lend their expertise. Judge Peck
terminology.” Putting aside the ethical obligations that may explicitly stated that his opinion is not an endorsement for
be implicated by such a deficiency, when handling eDiscov- any particular vendor or technology. However, he does note
ery in modern litigation you must know your client’s data, key that it was “very helpful that the parties’ eDiscovery vendors
custodians, business terminology, and internal processes. were present and spoke at the court hearings where the ESI
Having this type of knowledge in advance will not only pro- Protocol was discussed.” He went on to say that vendors can
vide many strategic advantages, but when you openly share be helpful even when counsel is knowledgeable about ESI
information about target custodians and your proposed issues. All vendors are not created equal unfortunately, and
search methodology “opposing counsel and the Court are it is important to vet providers of computer-assisted review
more apt to agree to your approach (at least as phase one for their knowledge of the technology as well as their ability
without prejudice).” to implement processes and protocols to ensure its success-
ful deployment. Just because someone has a band saw in
9. Phased discovery can help control costs. The defendant their garage does not mean they can build an armoire that
proposed a phased approach to discovery where 30 custo- will last. Judge Peck states that, it is “important for the ven-
dians would be reviewed during an initial phase, after which dor…to be able to explain complicated ediscovery concepts
a second phase would be considered. The Court agreed with in ways that make it easily understandable to judges who
this approach and addressed plaintiffs’ concern over hav- may not be tech-savvy.” Make sure you ask your vendor lots
ing enough time to complete both phases by extending the of questions so that you are comfortable that they can suc-
timeframe for discovery if necessary. With this approach the cessfully fill this role.
defendant was able to avoid the up-front costs of searching
seven additional custodians. In addition, Judge Peck held
that the Defendant company’s CEO, whose emails were in The Da Silva Moore opinion significantly advances the case for
France and likely protected by French privacy laws should not predictive coding. Judge Peck ruled that its use was warranted
be reviewed initially because those emails could be obtained given: “(1) the parties’ agreement, (2) the vast amount of ESI to
through various custodians located in New York. This also be reviewed (over three million documents), (3) the superiority
helped reduce the costs associated with complex cross- of computer-assisted review to the available alternatives (i.e.,
border eDiscovery efforts. Those of you that just said, “Wait! linear manual review or keyword searches), (4) the need for cost
Phased discovery is not necessarily a ‘lesson learned’ about effectiveness and proportionality under Rule 26(b)(2)(C), and (5)
the use of computer assisted review!” are arguably correct. the transparent process proposed by the [defendant].” While it
However, I present it to you because it can be a vital part of appears that we have a long way to go before computer-assisted
any effective technology-assisted review protocol. review supplants the traditional, and more costly, manual review
methods, the new technology and associated processes pres-
10. Predictive coding is not necessarily appropriate for every ent a means to address the problems associated with expanding
case. Judge Peck stated several times that he approved of amounts of ESI and the associated eDiscovery burdens that may
computer-assisted review in “appropriate cases.” In the arise. The legal industry does not exist in isolation from develop-
same manner that a chainsaw is not the best tool for all types ments in technology, but rather, is intimately intertwined with
of yard work, technology-assisted review is not the ideal tool such advancement. As a result, legal practitioners must adapt.
in all cases. In Da Silva Moore it was deemed appropriate, With his opinion in Da Silva Moore, Judge Peck gives all legal pro-
in part, because of the large document universe. Some may fessionals a means to jut ahead a few positions on the evolution-
argue that every case involving millions of records does not ary chart.
necessarily require computer-assisted review since there are

The “CAR” Keys to Defensibility ( Computer Assisted Review ) | | Complete Discovery Source
Share
20
The “CAR” Keys to Defensibility ( Computer Assisted Review

Industry News
Technology Update: CDS’ Nytrix is Moving

By Dino Medina More Than Review Into the Cloud
Judge Carter Upholds Peck’s Decision in Da
http://www.cdslegal.com
Silva Moore
As published in the New York County Lawyer, September 2012
Cloud Could Cut $12 Billion from US
The purpose of pretrial discovery in the American legal system is to promote the full Government Annual Deficit
disclosure of facts relevant to a given dispute. To this end, the Federal Rules permit
Technology Assisted Review – Is it
a litigating party to seek all information that is reasonably calculated to lead to the defensible?
discovery of admissible evidence.[1] There was a time when discovery requests
would yield only boxes of documents from which to locate relevant material. The Inadvertent Production of Two Privileged
Pages Among Over Two Million May Waive
societal shift from paper documents to electronically stored information (ESI) over
the Attorney-Client Privilege
the past ten years, however, has led to sharp increases in discovery data volume,
review time and cost. The following statistics illustrate this phenomenon: Predictive Coding Based Legal Methods for
Search and Review
Statisticians estimate that only 5% to 7% of all data is created outside of a
New York Appellate Court: Disintegration of
computer system.[2] Business Relationship Requires Preservation
of ESI
73% of all production costs are attributable to review of ESI.[3]
Third Circuit: Winners Still Have to Shoulder
The cost to review one gigabyte of ESI is between $5,000 and $30,000.[4]
Most E-Discovery Costs
As discovery in the legal industry continues to evolve in this electronic age, Allergan, Inc. v. Merz Pharma: Central
defensible methods to manage increased data volume, speed-up review time and District of CA Hammers Merz for Failure to
Timely Produce Documents
curb review cost have become essential.
Recovery of E-Discovery Costs 28 USC
Technology’s latest solution to these discovery issues is computer-assisted review §1920(4)
(CAR). CAR is a process by which analytics technology leverages the subject matter
proficiency of human document reviewers to make conceptually-based coding
decisions on all documents within a particular universe or “corpus.” Attorneys with
subject matter expertise must first generate a “seed” or “training” set of documents
Tools
(e.g., 500 docs), code it consistently using a specified protocol and run an analytics
engine against the “seed” set to extrapolate its coding to the remaining documents in The Basics: What is eDiscovery?
the corpus. Next, the analytics software generates a random sample from the
52 e-Discovery Terms You Should Know
corpus, which is then re-coded by the attorneys with subject matter expertise. Once
re-coded, the analytics tool will again extrapolate the coding to the corpus. Third, an CDS and Relativity Assisted Review Case Study
overturn report is run which, among other things, allows one to see the percentage
Recent e-Discovery Case Law
of computer coding decisions overturned by the attorneys with subject matter
expertise. Several iterations of this process are implemented until the computer’s 10 Tips: Planning for eDiscovery
coding is overturned by the attorneys with subject matter expertise at a rate of less
than 5%. This process is proven to be significantly less costly and time consuming eDiscovery Best Practices Guide
than traditional manual document review.[5]
11 Predictive Coding Lessons from Judge Peck’s
Opinion in Da Silva Moore, March 2011
Given all of the recent hype surrounding CAR and the judicial acceptance of this
“black box” technology in the Da Silva Moore, Global Aerospace and Kleen Products
Articles/Case Studies
cases, one might wonder: “can CAR really be as reliable (and therefore, legally
defensible) as traditional manual review in identifying relevant data?” This question The "CAR" Keys to Defensibility - Computer
has become one of primordial importance to large-scale litigators tasked with sorting Assisted Review
through huge volumes of ESI to find responsive and privileged material in an efficient
Predictive Coding: Rise of the Machines?
and cost-effective manner. As will become evident, CAR technology combined with a
robust audit trail can provide a defensible mechanism to meet one’s e-discovery Managing eDiscovery Strategy and
obligations. Implementation
How does one achieve defensibility in the context of CAR? The producing party must IPM Case Study: CDS Tames Big Data with Big
Storage
build a detailed audit trail that satisfies a two-pronged test. The first prong speaks
to the “reasonableness” of the CAR process; the selected method must accurately UNC JOLT: US Legal Holds Across Borders: A
capture a reasonably sufficient number of the responsive, non-privileged ESI Conundrum?
pursuant to a given document request, and any remaining unproduced ESI is
predominantly not relevant or privileged.[6] An audit trail satisfying the first CAR Richmond JOLT: Admissibility of Non-US
Evidence
defensibility prong entails reaching agreement with one’s adversary as to the degree
of recall and precision the CAR software will utilize in gathering, analyzing and coding Subscribe to our YouTube Channel
discovery documents.
Recall is the number of potentially relevant documents the system retrieves relative
to the number of documents in the entire collection[7] (e.g., as a result of a keyword
search or concept-based search). Precision is the number of relevant documents
identified in relation to the number of potentially relevant documents previously
retrieved. Recall is an indicator of the system’s effectiveness in finding potentially
relevant data, and precision is an indicator of the accuracy of the document coding
process. Recall is a measure of completeness or quantity, whereas precision is a
measure of exactness or quality. With too little recall, potentially relevant
information will be overlooked; with too little precision, one’s “relevant” document set
becomes riddled with non-relevant documents. The key to defensibility is a
memorialized negotiation and agreement on these issues; it would be difficult for
one’s adversary to claim production deficiencies in the face of a written recall and
precision rate agreement.
Additionally important in satisfying the first defensibility prong as it relates to CAR, is

the ability to document the process of coding “seed” documents and subsequent
system “training” documents. It is generally understood that the technology used to
identify relevant data is only as good as those training it. Because most difficulties in
the CAR process stem from human review errors, the “seed” and subsequent
“training” phases must be completed accurately and consistently by attorneys with
expertise in the particular subject matter. Without accurate and consistent human
input, the CAR analytics engine is unable to produce reliable results. Attorneys with
subject matter expertise should draft a protocol detailing the criteria for accurately
and consistently categorizing documents for defensibility purposes.
Finally, the statistical components underlying the CAR process must be scientifically
sound and documented to satisfy the first defensibility prong. These statistical
components include random sampling, confidence levels and confidence intervals.
Random sampling is a means to obtain a truly representative sample of documents,
by enabling all documents within a corpus to have an equal chance of being selected.
Random sampling can be used to select the initial “seed” set and must be used to
draw samples for each of the iterative training rounds.
A confidence level expresses the level of certainty one can expect that a sample set
of documents is drawn randomly from the system and reflective of the types of
documents in a given population. A confidence interval is the margin of error—
expressed as both positive and negative numbers—that a selected sample is truly
random. Most CAR experts use the 95% confidence level, with an accompanying
margin of error of +/-2.5%. The key here for prong one defensibility is that (a) both
the confidence level and confidence interval be agreed to by the parties and (b) a
report is generated at each phase of the CAR analytics process to document the
manner in which samples are drawn and coded by the software.
The second defensibility prong is aimed at satisfaction of Rules 1 and 26 of the

Federal Rules of Civil Procedure. That is, “the burden or expense of [the] discovery
[process must] not outweigh its likely benefit”[8] and assist in obtaining “the just,
speedy and inexpensive” resolution of a case.[9] Document intensive cases—in
which traditional manual review is prohibitively costly, time consuming and error
prone—make prime candidates for passing muster under this second defensibility
prong. Consider the following data from a 2012 study conducted by the RAND
Corporation:
Computer-assisted review enables case teams to identify at least as many

relevant documents as manual review with almost the same level of accuracy.
Computer-assisted review can cut attorney review time by as much as 75%.
Computer-assisted review can achieve an 80% cost savings over traditional

manual review.[10]
While these CAR statistics are impressive, the key to defensibility under this second
prong is compiling detailed CAR vs. manual review cost comparison data, relevancy
effectiveness data and estimated review time data.
In conclusion, the volume, complexity and variety of today’s data is challenging the
legal industry to find new ways to defensibly manage discovery. It is important to
approach CAR defensibility in terms of method reasonableness combined with a
robust audit trail. Such a framework will enable parties to rapidly and accurately
identify relevant documents, completely eliminate irrelevant documents, prioritize the
most relevant documents for review, achieve substantial cost savings and pass
judicial scrutiny.
[1] See Fed. R. Civ. P. 26(b)(1)
[2] Craig Ball, The DNA of Data, Law Technology News, April, 2005.
[3] Nicholas M. Pace & Laura Zakaras. Where the Money Goes: Understanding
Litigant Expenditures for Producing Electronic Discovery. Santa Monica, CA: RAND
Corporation, 2012. http://www.rand.org/pubs/monographs/MG1208.
[4] Dean Gonsowski, E-Discovery Costs: Pay Now or Pay Later, Inside Counsel,
May 23, 2012.
[5] Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-

Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review,
XVII Rich. J.L. & Tech. 11 (2011).
[6] The Sedona Conference, The Sedona Conference Commentary on Achieving

Quality in the E-Discovery Process, 10 SEDONA CONF. J. 299, 320 (2009).
[7] Alex Hoover, Top Ten Points on Computer Assisted Review, Association of
Corporate Counsel, September 1, 2011.
[8] FED. R. CIV. P. 1.
[9] FED. R. CIV. P. 26(b)(2)(C)(iii).
[10] See Pace & Zakaras, supra note 3.
Legal Notice and Privacy Policy - Contact

Richmond Journal of Law and Technology Vol. XVII, Issue 3
TECHNOLOGY-ASSISTED REVIEW IN E-DISCOVERY CAN BE

MORE EFFECTIVE AND MORE EFFICIENT
THAN EXHAUSTIVE MANUAL REVIEW
By Maura R. Grossman* & Gordon V. Cormack† **
Cite as: Maura R. Grossman & Gordon V. Cormack,

Technology-Assisted Review in E-Discovery Can Be More
Effective and More Efficient Than Exhaustive Manual
Review, XVII RICH. J.L. & TECH. 11 (2011),
http://jolt.richmond.edu/v17i3/article11.pdf.
*
Maura R. Grossman is counsel at Wachtell, Lipton, Rosen & Katz. She is co-chair of
the E-Discovery Working Group advising the New York State Unified Court System, and
a member of the Discovery Subcommittee of the Attorney Advisory Group to the Judicial
Improvements Committee of the U.S. District Court for the Southern District of New
York. Ms. Grossman is a coordinator of the Legal Track of the National Institute of
Standards and Technology’s Text Retrieval Conference (“TREC”), and an adjunct faculty
member at Rutgers School of Law–Newark and Pace Law School. Ms. Grossman holds a
J.D. from Georgetown University Law Center, and an M.A. and Ph.D. in Clinical/School
Psychology from Adelphi University. The views expressed herein are solely those of the
Author and should not be attributed to her firm or its clients.
†
Gordon V. Cormack is a Professor at the David R. Cheriton School of Computer
Science, and co-director of the Information Retrieval Group, at the University of
Waterloo. He is a coordinator of the TREC Legal Track, and Program Committee
member of TREC at large. Professor Cormack is the co-author of Information Retrieval:
Implementing and Evaluating Search Engines (MIT Press, 2010), as well as more than
100 scholarly articles. Professor Cormack holds a B.Sc., M.Sc., and Ph.D. in Computer
Science from the University of Manitoba.
**
The Authors would like to thank Ellen Voorhees and Ian Soboroff at NIST for
providing access to the raw TREC 2009 data. The Authors gratefully acknowledge the
helpful comments received from Hon. John M. Facciola (D.D.C.), Hon. Paul W. Grimm
(D. Md.), and Hon. Andrew J. Peck (S.D.N.Y.) on an earlier draft of this paper.
1
ABSTRACT
E-discovery processes that use automated tools to prioritize and

select documents for review are typically regarded as potential cost-savers
– but inferior alternatives – to exhaustive manual review, in which a cadre
of reviewers assesses every document for responsiveness to a production
request, and for privilege. This Article offers evidence that such
technology-assisted processes, while indeed more efficient, can also yield
results superior to those of exhaustive manual review, as measured by
recall and precision, as well as F1, a summary measure combining both
recall and precision. The evidence derives from an analysis of data
collected from the TREC 2009 Legal Track Interactive Task, and shows
that, at TREC 2009, technology-assisted review processes enabled two
participating teams to achieve results superior to those that could have
been achieved through a manual review of the entire document collection
by the official TREC assessors.
2
I. INTRODUCTION
[1] The Sedona Conference Best Practices Commentary on the Use of

Search and Information Retrieval Methods in E-Discovery cautions that:
[T]here appears to be a myth that manual review by

humans of large amounts of information is as accurate and
complete as possible – perhaps even perfect – and
constitutes the gold standard by which all searches should
be measured. Even assuming that the profession had the
time and resources to continue to conduct manual review of
massive sets of electronic data sets (which it does not), the
relative efficacy of that approach versus utilizing newly
developed automated methods of review remains very
much open to debate.1
While the word myth suggests disbelief, literature on the subject contains
little scientific evidence to support or refute the notion that automated
methods, while improving on the efficiency of manual review, yield
inferior results.2 This Article presents evidence supporting the position
that a technology-assisted process, in which humans examine only a small
fraction of the document collection, can yield higher recall and/or
precision than an exhaustive manual review process, in which humans
code and examine the entire document collection.
[2] A technology-assisted review process involves the interplay of

humans and computers to identify the documents in a collection that are
responsive to a production request, or to identify those documents that
should be withheld on the basis of privilege.3 A human examines and
1
The Sedona Conference, The Sedona Conference Best Practices Commentary on the
Use of Search and Information Retrieval Methods in E-Discovery, 8 SEDONA CONF. J.
189, 199 (2007) [hereinafter Sedona Search Commentary].
2
Id. at 194 (“The comparative efficacy of the results of manual review versus the results
of alternative forms of automated methods of review remains very much an open matter
of debate.”).
3
See Douglas W. Oard et al., Evaluation of information retrieval for E-discovery, 18:4
ARTIFICIAL INTELLIGENCE & LAW 347, 365 (2010) (“In some cases . . . the end user will
interact directly with the system, specifying the query, reviewing results, modifying the
3
codes only those documents the computer identifies – a tiny fraction of the
entire collection.4 Using the results of this human review, the computer
codes the remaining documents in the collection for responsiveness (or
privilege).5 A technology-assisted review process may involve, in whole
or in part, the use of one or more approaches including, but not limited to,
keyword search, Boolean search, conceptual search, clustering, machine
learning, relevance ranking, and sampling.6 In contrast, exhaustive
manual review requires one or more humans to examine each and every
document in the collection, and to code them as responsive (or privileged)
or not.7
[3] Relevant literature suggests that manual review is far from

perfect.8 Moreover, recent results from the Text Retrieval Conference
(“TREC”), sponsored by the National Institute of Standards and
Technology (“NIST”), show that technology-assisted processes can
achieve high levels of recall and precision.9 By analyzing data collected
query, and so on. In other cases, the end user’s interaction with the system will be more
indirect. . . .”).
4
See Sedona Search Commentary supra note 1, at 209.
5
See Maura R. Grossman & Terry Sweeney, What Lawyers Need to Know About Search
Tools, THE NAT’L L.J. (Aug. 23, 2010), available at http://www.law.com/jsp/
lawtechnologynews/PubArticleLTN.jsp?id=1202470952987&slreturn=1&hbxlogin=1
(“‘machine learning tools,’ use ‘seed sets’ of documents previously identified as
responsive or unresponsive to rank the remaining documents from most to least likely to
be relevant, or to classify the documents as responsive or nonresponsive.”).
6
See, e.g., Sedona Search Commentary, supra note 1, at 217–23; CORNELIS JOOST VAN
RIJSBERGEN, INFORMATION RETRIEVAL 74-85 (2d ed. 1979). The specific technologies
employed in the processes that are the subjects of this study are detailed infra Parts III.A.
– III.B.
7
See, e.g., Herbert L. Roitblat et al., Document Categorization in Legal Electronic
Discovery: Computer Classification vs. Manual Review, 61 J. AM. SOC’Y. FOR INFO. SCI.
AND TECH. 70, 70 (2010).
8
See, e.g., Sedona Search Commentary, supra note 1.
9
Bruce Hedin et al., Overview of the TREC 2009 Legal Track, in NIST SPECIAL
PUBLICATION: SP 500-278, THE EIGHTEENTH TEXT RETRIEVAL CONFERENCE (TREC
2009) PROCEEDINGS 16 & tbl.5 (2009), available at http://trec-
4
during the course of the TREC 2009 Legal Track Interactive Task,10 the
Authors demonstrate that the levels of performance achieved by two
technology-assisted processes exceed those that would have been achieved
by the official TREC assessors – law students and lawyers employed by
professional document-review companies – had they conducted a manual
review of the entire document collection.
[4] Part II of this Article describes document review and production in

the context of civil litigation, defines commonly used terms in the field of
information retrieval, and provides an overview of recent studies. Part III
details the TREC 2009 Legal Track Interactive Task, including the H5 and
Waterloo efforts, as well as the TREC process for assessment and gold-
standard creation. Part IV uses statistical inference to compare the recall,
precision, and F1 scores that H5 and Waterloo achieved to those the TREC
assessors would have achieved had they reviewed all of the documents in
the collection. Part V presents a qualitative analysis of the nature of
manual review errors. Parts VI, VII, and VIII, respectively, discuss the
results, limitations, and conclusions associated with this study.
Ultimately, this Article addresses a fundamental uncertainty that arises in
determining what is reasonable and proportional: Is it true that if a human
examines every document from a particular source, that human will, as
nearly as possible, correctly identify all and only the documents that
should be produced? That is, does exhaustive manual review guarantee
that production will be as complete and correct as possible? Or can
technology-assisted review, in which a human examines only a fraction of
the documents, do better?
II. CONTEXT
[5] Under Federal Rule of Civil Procedure 26(g)(1) (“Rule 26(g)(1)”),

an attorney of record must certify “to the best of [his or her] knowledge,
legal.umiacs.umd.edu/LegalOverview09.pdf; see also Douglas W. Oard et al., Overview

of the TREC 2008 Legal Track, in NIST SPECIAL PUBLICATION: SP 500-277, THE
SEVENTEENTH TEXT RETRIEVAL CONFERENCE (TREC 2008) PROCEEDINGS 8 (2008),
available at http://trec.nist.gov/pubs/trec17/papers/LEGAL.OVERVIEW08.pdf.
10
See Hedin et al., supra note 9, at 2.
5
information, and belief formed after a reasonable inquiry,” that every

discovery request, response, or objection is
consistent with [the Federal Rules of Civil Procedure] . . .

not interposed for any improper purpose, such as to harass,
cause unnecessary delay, or needlessly increase the cost of
litigation[, and is] neither unreasonable nor unduly
burdensome or expensive, considering the needs of the
case, prior discovery in the case, the amount in controversy,
and the importance of the issues at stake in the action.11
Similarly, Federal Rule of Civil Procedure 26(b)(2)(C)(iii) (“Rule

26(b)(2)(C)(iii)”) requires a court to limit discovery when it determines
that “the burden or expense of the proposed discovery outweighs its likely
benefit, considering the needs of the case, the amount in controversy, the
parties’ resources, the importance of the issues at stake in the action, and
the importance of the discovery in resolving the issues.”12 Thus, Rules
26(g)(1) and 26(b)(2)(C)(iii) require that discovery requests and responses
be proportional.13 However, Federal Rule of Civil Procedure 37(a)(4)
(“Rule 37(a)(4)”) provides that “an evasive or incomplete disclosure,
answer or response must be treated as a failure to disclose, answer, or
respond[,]” and therefore requires that discovery responses be complete.14
Together, Rules 26(g)(1), 26(b)(2)(C)(iii), and 37(a)(4) reflect the tension
– between completeness on one hand, and burden and cost on the other –
that exists in all electronic discovery (“e-discovery”) processes.15 In
11
FED. R. CIV. P. 26(g)(1).
12
FED. R. CIV. P. 26(b)(2)(C)(iii).
13
The Sedona Conference, The Sedona Conference Commentary on Proportionality in
Electronic Discovery, 11 SEDONA CONF. J. 289, 294 (2010) [hereinafter Sedona
Proportionality Commentary].
14
FED. R. CIV. P. 37(a)(4).
15
Typically, a responding party will not only seek to produce all responsive documents,
but to identify only the responsive documents, in order to guard against overproduction or
waiver of privilege. See, e.g., Mt. Hawley Ins. Co. v. Felman Prod., Inc., 271 F.R.D. 125,
136 (S.D.W. Va. 2010) (finding that plaintiff’s over-production of documents by more
than 30% was a factor in waiver of privilege).
6
assessing what is reasonable and proportional with respect to e-discovery,

parties and courts must balance these competing considerations.16
[6] One of the greatest challenges facing legal stakeholders is

determining whether or not the cost and burden of identifying and
producing electronically stored information (“ESI”) is commensurate with
its importance in resolving the issues in dispute.17 In current practice, the
problem of identifying responsive (or privileged) ESI, once it has been
collected, is almost always addressed, at least in part, by a manual review
process, the cost of which dominates the e-discovery process.18 A natural
question to ask, then, is whether this manual review process is the most
effective and efficient one for identifying and producing the ESI most
likely to resolve a dispute.
A. Information Retrieval
[7] The task of finding all, and only, the documents that meet “some
requirement” is one of information retrieval (“IR”), a subject of scholarly
16
See Harkabi v. Sandisk Corp., No. 08 Civ. 8203 (WHP), 2010 WL 3377338, at *1
(S.D.N.Y Aug. 23, 2010) (“Electronic discovery requires litigants to scour disparate data
storage mediums and formats for potentially relevant documents. That undertaking
involves dueling considerations: thoroughness and cost.”).
17
See id. at *8 (“Integral to a court’s inherent power is the power to ensure that the game
is worth the candle—that commercial litigation makes economic sense. Electronic
discovery in this case has already put that principle in jeopardy.”); Hopson v. Mayor of
Balt., 232 F.R.D. 228, 232 (D. Md. 2005) (“This case vividly illustrates one of the most
challenging aspects of discovery of electronically stored information—how properly to
conduct Rule 34 discovery within a reasonable pretrial schedule, while concomitantly
insuring that requesting parties receive appropriate discovery, and that producing parties
are not subjected to production timetables that create unreasonable burden, expense, and
risk of waiver of attorney-client privilege and work product protection”). See generally
Sedona Proportionality Commentary, supra note 13.
18
Marisa Peacock, The True Cost of eDiscovery, CMSWIRE, http://www.cmswire.com/
cms/enterprise-cms/the-true-cost-of-ediscovery-006060.php (2009) (citing Sedona
Search Commentary, supra note 1, at 192); Ashish Prasad et al., Cutting to the
“Document Review” Chase: Managing a Document Review in Litigation and
Investigations, 18 BUS. LAW TODAY, 2, Nov.–Dec. 2008.
7
research for at least a century.19 In IR terms, “some requirement” is

referred to as an information need, and relevance is the property of
whether or not a particular document meets the information need.20 For
e-discovery, the information need is typically specified by a production
request (or by the rules governing privilege), and the definition of
relevance follows.21 Cast in IR terms, the objective of review in
e-discovery is to identify as many relevant documents as possible, while
simultaneously identifying as few nonrelevant documents as possible.22
The fraction of relevant documents identified during a review is known as
recall, while the fraction of identified documents that are relevant is
known as precision.23 That is, recall is a measure of completeness, while
precision is a measure of accuracy, or correctness.24
[8] The notion of relevance, although central to information science,

and the subject of much philosophical and scientific investigation, remains
elusive.25 While it is easy enough to write a document describing an
19
The concepts and terminology outlined in Part II.A may be found in many information
retrieval textbooks. For a historical perspective, see GERARD SALTON & MICHAEL J.
MCGILL, INTRODUCTION TO MODERN INFORMATION RETRIEVAL (1983); VAN
RIJSBERGEN, supra note 6. For a more modern treatment, see STEFAN BÜTTCHER ET AL.,
INFORMATION RETRIEVAL: IMPLEMENTING AND EVALUATING SEARCH ENGINES 33–75
(2010).
20
See BÜTTCHER ET AL., supra note 19, at 5-6, 8.
21
22
See VAN RIJSBERGEN, supra note 6, at 4.
23
See David C. Blair & M. E. Maron, An Evaluation of Retrieval Effectiveness for a Full-
Text Document-Retrieval System, 28 COMMC’NS ACM 289, 290 (1985) (“Recall
measures how well a system retrieves all the relevant documents; and Precision, how
well the system retrieves only the relevant documents.”); VAN RIJSBERGEN, supra note 6,
at 112-13.
24
See VAN RIJSBERGEN, supra note 6, at 113.
25
See Tefko Saracevic, Relevance: A Review of the Literature and a Framework for
Thinking on the Notion in Information Science. Part II: Nature and Manifestations of
Relevance, 58 J. AM. SOC’Y FOR INFO. SCI. & TECH. 1915 (2007); Tefko Saracevic,
Relevance: A Review of the Literature and a Framework for Thinking on the Notion in
8
information need and hence relevance, determining the relevance of any

particular document requires human interpretation.26 It is well established
that human assessors will disagree in a substantial number of cases as to
whether a document is relevant, regardless of the information need or the
assessors’ expertise and diligence.27
[9] A review resulting in higher recall and higher precision than

another review is more nearly complete and correct, and therefore
superior,28 while a review with lower recall and lower precision is
inferior.29 If one result has higher recall while the other has higher
precision, it is not immediately obvious which should be considered
superior. To calculate a review’s effectiveness, researchers often employ
F1 – the harmonic mean of recall and precision30 – a commonly used
summary measure that rewards results achieving both high recall and high
precision, while penalizing those that have either low recall or low
precision.31 The value of F1 is always intermediate between recall and
precision, but is generally closer to the lesser of the two.32 For example, a
result with 40% recall and 60% precision has F1 = 48%. Following
Information Science. Part III: Behavior and Effects of Relevance, 58:13 J. AM. SOC’Y
FOR INFO. SCI. & TECH.2126 (2007).
26
See Peter Bailey et al., Relevance Assessment: Are Judges Exchangeable and Does It
Matter?, in SIGIR ’08 PROCEEDINGS OF THE 31ST ANNUAL INTERNATIONAL ACM SIGIR
CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL 667
(2008); see also VAN RIJSBERGEN, supra note 6, at 112.
27
See Bailey et al., supra note 26, at § 4.3.
28
See Blair & Maron, supra note 23.
29
See id.
30
= .
31
See BÜTTCHER ET AL., supra note 19, at 68.
32
See id.
9
TREC, this Article reports recall and precision, along with F1 as a

summary measure of overall review effectiveness.33
B. Assessor Overlap
[10] The level of agreement between independent assessors may be

quantified by overlap – also known as the Jaccard index – the number of
documents identified as relevant by two independent assessors, divided by
the number identified as relevant by either or both assessors.34 For
example, suppose assessor A identifies documents {W,X,Y,Z} as relevant,
while assessor B identifies documents {V,W,X}. Both assessors have
identified two documents {W,X} as relevant, while either or both have
identified five documents {V,W,X,Y,Z} as relevant. So the overlap is 2/5,
or forty percent. Informally, overlap of less than fifty percent indicates
that the assessors disagree on whether or not a document is relevant more
often than when they agree that a document is relevant.35
[11] In her study, Variations in Relevance Judgments and the

Measurement of Retrieval Effectiveness, Ellen Voorhees measured overlap
between primary, secondary, and tertiary reviewers who each made 14,968
assessments of relevance for 13,435 documents,36 with respect to 49
33
34
Ellen M. Voorhees, Variations in Relevance Judgments and the Measurement of
Retrieval Effectiveness, 36 INFO. PROCESSING & MGMT 697, 700 (2000), available at
http://www.cs.cornell.edu/courses/cs430/2006fa/cache/Trec_8.pdf (“Overlap is defined
as the size of the intersection of the relevant document sets divided by the size of the
union of the relevant document sets.”); see CHRISTOPHER D. MANNING ET AL., AN
INTRODUCTION TO INFORMATION RETRIEVAL 61 (2009) (draft), available at
nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf; see also Raimundo Real & Juan
M. Vargas, The Probabilistic Basis of Jaccard’s Index of Similarity, 45 SYSTEMATIC
BIOLOGY 380, 381 (1996).
35
See Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, in
EVALUATION OF CROSS-LANGUAGE INFORMATION RETRIEVAL SYSTEMS SECOND
WORKSHOP OF THE CROSS-LANGUAGE EVALUATION FORUM, CLEF 2001 DARMSTADT,
GERMANY, SEPTEMBER 3-4, 2001 REVISED PAPERS 355, 364 (Carol Peters et al. eds.,
2002).
36
E-mail from Ellen M. Voorhees to Gordon V. Cormack (Jul. 31, 2019 14:34 EDT) (on
file with authors). The numbers in the text are derived from the file,
10
information needs (or “topics,” in TREC parlance), in connection with Ad

Hoc Task of the Fourth Text Retrieval Conference (“TREC 4”).37 As
illustrated in Table 1, the overlap between primary and secondary
assessors was 42.1%;38 the overlap between primary and tertiary assessors
was 49.4%;39 and the overlap between secondary and tertiary assessors
was 42.6%.40
[12] Perhaps due to the assessors’ expertise,41 Voorhees’ overlap results

are among the highest reported for pairs of human assessors. Her findings
demonstrate that assessors disagree at least as often as they agree that a
document is relevant.42 Voorhees concluded:
The scores for the [secondary and tertiary] judgments imply

a practical upper bound on retrieval system performance is
65% precision at 65% recall since that is the level at which
humans agree with one another.43
“threeWayJudgments,” attached to Voorhees’ e-mail. Some of the documents were

assessed for relevance to more than one topic.
37
Voorhees, supra note 34, at 708; see also Donna Harman, Overview of the Fourth Text
REtrieval Conference (TREC-4), in NIST SPECIAL PUBLICATION 500-236: THE FOURTH
TEXT RETRIEVAL CONFERENCE (TREC-4) 2 (2004), available at
http://trec.nist.gov/pubs/trec4/t4_proceedings.html (follow the first link under
“PAPERS”).
38
See infra Table 1; see also Voorhees, supra note 34, at 701 tbl.1.
39
40
41
All assessors were professional information retrieval experts. Voorhees, supra note 34,
at 701.
42
See id.
43
Id.
11
[13] It is not widely accepted that these findings apply to e-discovery.44

This “legal exceptionalism” appears to arise from common assumptions
within the legal community:
1. that the information need (responsiveness or privilege) is

more precisely defined for e-discovery than for classical
information retrieval;45
2. that lawyers are better able to assess relevance and privilege

than the non-lawyers typically employed for information
retrieval tasks;46 and
3. that the most defensible way to ensure that a production is

accurate is to have a lawyer examine each and every
document.47
44
See Sedona Search Commentary, supra note 1 (noting the widespread perception that
manual review is nearly perfect). If that perception were correct, manual reviewers
would have close to 100% overlap, contrary to Voorhees’ findings. Vorhees, supra note
34, at 701 tbl.1.
45
Oard et al., supra note 3, at 362 (“It is important to recognize that the notion of
relevance that is operative in E-discovery is, naturally, somewhat more focused than what
has been studied in information seeking behavior studies generally . . . .”).
46
Cf. Alejandra P. Perez, Assigning Non-Attorneys to First-Line Document Reviews
Requires Safeguards, THE E-DISCOVERY 4-1-1 (LeClairRyan), Jan. 2011, at 1, available
at http://marketing.leclairryan.com/files/Uploads/Documents/the-e-discovery-4-1-1-01-
21-2011.pdf (opining that non-attorney document reviewers typically require additional
training, particularly regarding the legal concept of privilege).
47
See Sedona Search Commentary, supra note 1, at 203 (“Some litigators continue to
primarily rely upon manual review of information as part of their review process.
Principal rationales [include] . . . the perception that there is a lack of scientific validity of
search technologies necessary to defend against a court challenge . . . .”); see also
Thomas E. Stevens & Wayne C. Matus, A ‘Comparative Advantage’ To Cut E-Discovery
Costs, NAT’L L.J. (Sept. 4, 2008), http://www.law.com/jsp/nlj/PubArticle
NLJ.jsp?id=1202424251053 (describing a “general reluctance by counsel to rely on
anything but what they perceive to be the most defensible positions in electronic
discovery, even if those solutions do not hold up any sort of honest analysis of cost or
quality”).
12
Assumptions (1) and (2) are amenable to scientific evaluation, as is the

overarching question of whether technology-assisted review can improve
upon exhaustive manual review. Assumption (3) – a legal opinion –
should be informed by scientific evaluation of the first two assumptions.
Assessment Primary Secondary Tertiary

Primary 100%
Secondary 42.1% 100%
Tertiary 49.4% 42.6% 100%
Table 1: Overlap in relevance assessments by primary, secondary, and tertiary

48
assessors for the TREC 4 Ad Hoc Task.
[14] Recently, Herbert Roitblat, Anne Kershaw, and Patrick Oot studied
the level of agreement among review teams using data produced to the
Department of Justice (“DOJ”) in response to a Second Request that
stemmed from MCI’s acquisition of Verizon.49 In their study, two
independent teams of professional assessors, Teams A and B, reviewed a
random sample of 5,000 documents.50 Roitblat and his colleagues
reported the level of agreement and disagreement between the original
production, Team A, and Team B, as a contingency matrix,51 from which
the Authors calculated overlap, as shown in Table 2.52 The overlap
between Team A and the original production was 16.3%;53 the overlap
between Team B and the original production was 15.8%;54 and the overlap
between Teams A and B was 28.1%.55 These and other studies of overlap
48
Voorhees, supra note 34, at 701 tbl.1.
49
See Roitblat et al., supra note 7, at 73.
50
See id. at 73-74.
51
Id. at 74 tbl.1.
52
See infra Table 2.
53
Id.
54
Id.
55
Id.
13
indicate that relevance is not a concept that can be applied consistently by

independent assessors, even if the information need is specified by a
production request and the assessors are lawyers.56
Assessment Production Team A Team B

Production 100%
Team A 16.3% 100%
Team B 15.8% 28.1% 100%
Table 2: Overlap in relevance assessments between original production in a

57
Second Request, and two subsequent manual reviews.
C. Assessor Accuracy
[15] Measurements of overlap provide little information regarding the

accuracy of particular assessors because there is no “gold standard”
against which to compare them.58 One way to resolve this problem is to
deem one assessor’s judgments correct by definition, and to use those
judgments as the gold standard for the purpose of evaluating the other
assessor(s).59
[16] In the Voorhees study, the primary assessor composed the

information need specification for each topic.60 It may therefore be
reasonable to take the primary assessor’s coding decisions to be the gold
standard. In the Roitblat, Kershaw, and Oot study, a senior attorney
familiar with the case adjudicated all instances of disagreement between
Teams A and B.61 Although Roitblat and his colleagues sought to
56
See Roitblat et al., supra note 7, at 73; Voorhees, supra note 34.
57
The Authors derived the information in Table 2 from the Roitblat, Kershaw, and Oot
study. Roitblat et al., supra note 7, at 74; see supra para. 13.
58
Roitblat et al., supra note 7, at 77.
59
See Voorhees, supra note 34, at 700.
60
Id.
61
14
measure agreement,62 it may be reasonable to use their “adjudicated

results” as the gold standard. These adjudicated results deemed the senior
attorney’s opinion correct in cases where Teams A and B disagreed, and
deemed the consensus correct in cases where Teams A and B agreed.63
Assuming these gold standards, Table 3 shows the effectiveness of the
various assessors in terms of recall, precision, and F1.64 Note that recall
ranges from 52.8% to 83.6%, while precision ranges from 55.5% to
81.9%, and F1 ranges from 64.0% to 70.4%.65 All in all, these results
appear to be reasonable, but hardly perfect. Can technology-assisted
review improve on them?
D. Technology-Assisted Review Accuracy
[17] In addition to the two manual review groups, Roitblat, Kershaw,

and Oot had two service providers (Teams C and D) use technology-
assisted review processes to classify each document in the dataset as
62
Id. at 72 (“Formally, the present study is intended to examine the hypothesis: The rate
of agreement between two independent reviewers of the same documents will be equal to
or less than the agreement between a computer-aided system and the original review.”).
63
Id. at 74.
The 1,487 documents on which Teams A and B disagreed were

submitted to a senior Verizon litigator (P. Oot), who adjudicated
between the two teams, again without knowledge of the specific
decisions made about each document during the first review. This
reviewer had knowledge of the specifics of the matter under review, but
had not participated in the original review. This authoritative reviewer
was charged with determining which of the two teams had made the
correct decision.
Id.
64
See infra Table 3. Recall and precision for the secondary and tertiary assessors, using
the primary assessor as the gold standard, are provided by Voorhees, supra note 34, at
701 tbl.2; recall and precision for Teams A and B, using the adjudicated results as the
gold standard, were derived from Roitblat et al., supra note 7, at 74 tbl.1; F1 was
calculated from recall and precision using the formula at supra note 30.
65
See infra Table 3.
15
relevant or not.66 Unfortunately, the adjudicated results described in Part

II.C. were made available to one of the two service providers, and
therefore, cannot be used as a gold standard to evaluate the accuracy of the
providers’ efforts.67
Study Review Recall Precision F1

Voorhees Secondary 52.8% 81.3% 64.0%
Voorhees Tertiary 61.8% 81.9% 70.4%
Roitblat et al. Team A 77.1% 60.9% 68.0%
Roitblat et al. Team B 83.6% 55.5% 66.7%
Table 3: Recall, precision, and F1 of manual assessments in studies by

Voorhees, and Roitblat et al. Voorhees evaluated secondary and tertiary
assessors with respect to a primary assessor, who was deemed correct. The
Authors computed recall, precision, and F1 from the results reported by Roitblat
68
et al. for Teams A and B, using their adjudicated results as the gold standard.
[18] Instead, Roitblat and his colleagues reported recall, precision, and
F1 using, as an alternate gold standard, the set of documents originally
produced to, and accepted by, the DOJ.69 There is little reason to believe
that this original production, and hence the alternate gold standard, was
perfect.70 The first two rows of Table 4 show the recall and precision of
manual review Teams A and B when evaluated with respect to this
66
Roitblat et al., supra note 7, at 74-75.
67
Id. at 74 (“One of these systems based its classifications in part on the adjudicated
results of Teams A and B, but without any knowledge of how those teams’ decisions
were related to the decisions made by [the] original review team. As a result, it is not
reasonable to compare the classifications of these two systems to the classifications of the
two re-review teams, but it is reasonable to compare them to the classifications of the
original review.”).
68
Voorhees, supra note 34, at 701 tbl.2; Roitblat et al. supra note 7, at 74 tbl.1.
69
70
Id. at 76 (“The use of precision and recall implies the availability of a stable ground
truth against which to compare the assessments. Given the known variability of human
judgments, we do not believe that we have a solid enough foundation to claim that we
know which documents are truly relevant and which are not.”).
16
alternate gold standard.71 These results are much worse than those in
Table 3.72 Team A achieved 48.8% recall and 19.7% precision, while
Team B achieved 52.9% recall and 18.3% precision.73 The corresponding
F1 scores were 28.1% and 27.2%, respectively – less than half of the F1
scores achieved with respect to the gold standard derived using the senior
attorney’s opinion.74
[19] The recall and precision Roitblat, Kershaw, and Oot reported were
computed using the original production as the gold standard, and are
dramatically different from those shown in Table 3, which were computed
using their adjudicated results as the gold standard.75 Nevertheless, both
sets of results appear to suggest the relative accuracy between Teams A
and B: Team B has higher recall, while Team A has higher precision and
higher F1, regardless of which gold standard is applied.76
[20] The last two rows of Table 4 show the effectiveness of the
technology-assisted reviews conducted by teams C and D, as reported by
Roitblat, Kershaw, and Oot using the original production as the gold
standard.77 The results suggest that technology-assisted review Teams C
and D achieved about the same recall as manual review Teams A and B,
and somewhat better precision and F1.78 However, due to the use of the
alternate gold standard, the result is inconclusive.79 Because the
71
See id. at 76 tbl.2; infra Table 4.
72
Compare supra Table 3, with infra Table 4.
73
See infra Table 4; see also Roitblat et al., supra note 7, at 74-76.
74
Compare supra Table 3, with infra Table 4.
75
Compare supra Table 3, with infra Table 4. See generally Roitblat et al., supra note 7,
at 76 tbl.2.
76
See supra Table 3; infra Table 4; Roitblat et al., supra note 7, at 76 tbl.2.
77
See infra Table 4; see also Roitblat et al., supra note 7, at 74-75.
78
See infra Table 4.
79
See Roitblat et al., supra note 7, at 76 (“The use of precision and recall implies the
availability of a stable ground truth against which to compare the assessments. Given the
17
improvement from using technology-assisted review, as reported by

Roitblat and his colleagues, is small compared to the difference between
the results observed using the two different gold standards, it is difficult to
determine whether the improvement represents a real difference in
effectiveness as compared to manual review.
Study Review Method Recall Precision F1

Roitblat et al. Team A Manual 48.8% 19.7% 28.1%
Roitblat et al. Team B Manual 52.9% 18.3% 27.2%
Roitblat et al. Team C Tech. Asst. 45.8% 27.1% 34.1%
Roitblat et al. Team D Tech. Asst. 52.7% 29.5% 37.8%
Table 4: Recall, precision, and F1 of manual and technology-assisted review

teams, evaluated with respect to the original production to the DOJ. The first
two rows of this table differ from the last two rows of Table 3 only in the gold
80
standard used for evaluation.
[21] In a heavily cited study by David C. Blair and M.E. Maron, skilled
paralegal searchers were instructed to retrieve at least 75% of all
documents relevant to 51 requests for information pertaining to a legal
matter.81 For each request, the searchers composed keyword searches
using an interactive search system, retrieving and printing documents for
further review.82 This process was repeated until the searcher was
satisfied that 75% of the relevant documents had been retrieved.83
Although the searchers believed they had found 75% of the relevant
documents, their average recall was only 20.0%.84 Despite this low rate of
known variability of human judgments, we do not believe that we have a solid enough
foundation to claim that we know which documents are truly relevant and which are
not.”).
80
Id. at 73-76.
81
See Blair & Maron, supra note 23, at 291.
82
Id.
83
Id.
84
Id. at 293; see also Maureen Dostert & Diane Kelly, Users’ Stopping Behaviors and
Estimates of Recall, in SIGIR ’09 PROCEEDINGS OF THE 32ND ANNUAL INTERNATIONAL
ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION
RETRIEVAL 820–21 (2009) (showing that most subjects in an interactive information
18
recall, the searchers achieved a high average precision of 79.0%.85 From

the published data,86 the Authors calculated the average F1 score to be
28.0% – remarkably similar to that observed by Roitblat and his
colleagues for manual review.87
[22] Blair and Maron argue that the searchers would have been unable
to achieve higher recall even if they had known there were many relevant
documents that were not retrieved.88 Researcher Gerald Salton
disagrees.89 He claims that it would have been possible for the searchers
to achieve higher recall at the expense of lower precision, either by
broadening their queries or by taking advantage of the relevance ranking
capability of the search system.90
[23] Overall, the literature offers little reason to believe that manual
review is perfect. But is it as complete and accurate as possible, or can it
be improved upon by technology-assisted approaches invented since Blair
and Maron’s study?
[24] As previously noted, recent results from TREC suggest that

technology-assisted approaches may indeed be able to improve on manual
review.91 In the TREC 2008 Legal Track Interactive Task, H5, a San
retrieval experiment reported they had found about 51-60% of the relevant documents
when, on average, recall was only 7%).
85
See Blair & Maron, supra note 23, at 293.
86
Id.
87
See Roitblat et al., supra note 7 at 76.
88
See Blair & Maron, supra note 23, at 295-96.
89
See Gerard Salton, Another Look at Automatic Text-Retrieval Systems, 29:7 COMMC’NS
ACM 648, 650 (1986).
90
Id. at 648-49.
91
See generally Hedin et al., supra note 9; Oard et al., supra note 9.
19
Francisco-based legal information retrieval firm,92 employed a user-

modeling approach93 to achieve recall, precision, and F1 of 62.4%, 81.0%,
and 70.5%, respectively, in response to a mock request to produce
documents from a 6,910,192-document collection released under the
tobacco Master Settlement Agreement.94 In the course of this effort, H5
examined only 7,992 documents95 – roughly 860 times fewer than the
6,910,192 it would have been necessary to examine in an exhaustive
manual review. Yet the results compare favorably with those previously
reported for manual review or keyword search, exceeding what Voorhees
characterizes as a “practical upper bound” on what may be achieved, given
uncertainties in assessment.96
92
See Contact Us, H5, http://www.h5.com/about/contact.php (last visited Mar. 22, 2011);
Who We Are, H5, http://www.h5.com/about/who_we_are.html (last visited Apr. 11,
2011).
93
Christopher Hogan et al., H5 at TREC 2008 Legal Interactive: User Modeling,
Assessment & Measurement, in NIST SPECIAL PUBLICATION: SP 500-277, THE
SEVENTEENTH TEXT RETRIEVAL CONFERENCE (TREC 2008) PROCEEDINGS (2008),
available at http://trec.nist.gov/pubs/trec17/papers/ h5.legal.rev.pdf (last visited Mar. 23,
2011).
94
Oard et al., supra note 9, at 30 tbl.15; see also Complex Document Image Processing
(CDIP), ILL. INST. TECH., http://ir.iit.edu/projects/ CDIP.html (last visited Apr. 11, 2011);
Master Settlement Agreement, NAT’L ASS’N ATTORNEYS GEN. (Nov. 1998), available at
http://www.naag.org/backpages/naag/tobacco/msa/msa-pdf/MSA%20with%20Sig%20
Pages%20and%20Exhibits.pdf; TREC 2008, Complaint for Violation of the Federal
Securities Laws, Mellon v. Echinoderm Cigarettes, Inc., (2008), available at http://trec-
legal.umiacs.umd.edu/topics/8I.pdf.
95
Hogan et al., supra note 92, at 8.
96
Voorhees, supra note 34, at 701.
20
Topic Production Request

201 All documents or communications that describe, discuss, refer to,
report on, or relate to the Company’s engagement in structured
commodity transactions known as “prepay transactions.”
report on, or relate to the Company’s engagement in transactions that
the Company characterized as compliant with FAS 140 (or its
predecessor FAS 125).
report on, or relate to whether the Company had met, or could, would,
or might meet its financial forecasts, models, projections, or plans at
any time after January 1, 1999.
report on, or relate to any intentions, plans, efforts, or activities
involving the alteration, destruction, retention, lack of retention,
deletion, or shredding of documents or other evidence, whether in
hard-copy or electronic form.
report on, or relate to energy schedules and bids, including but not
limited to, estimates, forecasts, descriptions, characterizations,
analyses, evaluations, projections, plans, and reports on the volume(s)
or geographic location(s) of energy loads.
report on, or relate to any discussion(s), communication(s), or
contact(s) with financial analyst(s), or with the firm(s) that employ
them, regarding (i) the Company’s financial condition, (ii) analysts’
coverage of the Company and/or its financial condition, (iii) analysts’
rating of the Company’s stock, or (iv) the impact of an analyst’s
coverage of the Company on the business relationship between the
Company and the firm that employs the analyst.
report on, or relate to fantasy football, gambling on football, and
related activities, including but not limited to, football teams, football
players, football games, football statistics, and football performance.
Table 5: Mock production requests (“topics”) composed for the TREC 2009
97
Legal Track Interactive Task.
97
TREC 2009, Complaint, Grumby v. Volteron Corp., 14 (2009) available at http://trec-
legal.umiacs.umd.edu/LT09_Complaint _J_final.pdf; see also Hedin et al., supra note 9,
at 5-6.
21
[25] One of the Authors was inspired to try to reproduce these results at
TREC 2009 using an entirely different approach: statistical active
learning, originally developed for e-mail spam filtering.98 At the same
time, H5 reprised its approach for TREC 2009.99 The TREC 2009 Legal
Track Interactive Task used the same design as TREC 2008, but employed
a different complaint100 and seven new mock requests to produce
documents (see Table 5) from a new collection of 836,165 e-mail
messages and attachments captured from Enron at the time of its
collapse.101 Each participating team was permitted to request as many
topics as they wished, however, due to resource constraints, the most
topics that any team was assigned was four of the seven.102
98
See generally Gordon V. Cormack & Mona Mojdeh, Machine Learning for
Information Retrieval: TREC 2009 Web, Relevance Feedback and Legal Tracks, in NIST
SPECIAL PUBLICATION: SP 500-278, THE EIGHTEENTH TEXT RETRIEVAL CONFERENCE
(TREC 2009) PROCEEDINGS (2009), available at http://trec.nist.gov/pubs/
trec18/papers/uwaterloo-cormack.WEB.RF.LEGAL.pdf.
99
Hedin et al., supra note 9, at 6.
100
See generally TREC 2009, Complaint, supra note 97.
101
Hedin et al., supra note 9, at 4; see Information Released in Enron Investigation, FED.
ENERGY REG. COMM’N, http://www.ferc.gov/industries/electric/indus-act/wec/enron/
info-release.asp (last visited Apr. 11, 2011) [hereinafter FERC]; E-mail from Bruce
Hedin to Gordon V. Cormack (Aug. 31, 2009 20:33 EDT) (on file with authors) (“I have
attached full list of the 836,165 document-level IDs . . . .”). The collection is available at
Practice Topic and Assessments for TREC 2010 Legal Learning Task, U. WATERLOO,
http://plg1.uwaterloo.ca/~gvcormac/treclegal09/ (follow “The TREC 2009 dataset”) (last
visited Apr. 18, 2011).
102
Hedin et al., supra note 9, at 7; E-mail from Bruce Hedin to Gordon V. Cormack &
Maura R. Grossman (Mar. 24, 2011 02:46 EDT) (on file with authors).
22
Team Topic Reviewed Produced Recall Precision F1
Waterloo 201 6,145 2,154 77.8% 91.2% 84.0%

Waterloo 202 12,646 8,746 67.3% 88.4% 76.4%
Waterloo 203 4,369 2,719 86.5% 69.2% 76.9%
H5 204 20,000 2,994 76.2% 84.4% 80.1%
Waterloo 207 34,446 23,252 76.1% 90.7% 82.8%
Average: 15,521 7,973 76.7% 84.7% 80.0%
Table 6: Effectiveness of H5 and Waterloo submissions to the TREC 2009 Legal

103
Track Interactive Task.
[26] Together, H5 and Waterloo produced documents for five distinct

TREC 2009 topics;104 the results of their efforts are summarized in Table
6. The five efforts employed technology-assisted processes, with the
number of manually reviewed documents for each topic ranging from
4,369 to 34,446105 (or 0.5% to 4.1% of the collection). That is, the total
human effort for the technology-assisted processes – measured by the
number of documents reviewed – was between 0.5% and 4.1% of that
which would have been necessary for an exhaustive manual review of all
836,165 documents in the collection.106 The number of documents
produced for each topic ranged from 2,154 to 23,252107 (or 0.3% to 2.8%
of the collection; about half the number of documents reviewed). Over the
five efforts, the average recall and precision were 76.7% and 84.7%,
103
See infra, para. 25.
104
105
Cormack & Mojdeh, supra note 98, at 6 tbl.2 (showing that Waterloo reviewed
between 4,369 documents (for Topic 203) and 34,446 documents (for Topic 207); see E-
mail from Dan Brassil to Maura R. Grossman (Dec. 17, 2010 15:21 EST) (on file with
authors) (“[H5] sampled and reviewed 20,000 documents”).
106
See sources cited supra note 101.
107
NIST Special Publication 500-277: The Seventeenth Text REtrieval Conference
Proceedings (TREC 2008) http://trec.nist.gov/pubs/trec17/t17_proceedings.html
Appendix: Per Topic Scores: TREC 2009 Legal Track, Interactive Task, 3 tbl.4, 4 tbl.8, 5
tbl.12, 6 tbl.16, 9 tbl.26 http://trec.nist.gov/pubs/trec18/appendices/ app09int2.pdf.
23
respectively; no recall was lower than 67.3%, and no precision was lower
than 69.2%,108 placing all five efforts above what Voorhees characterized
as a “practical upper bound” on what may be achieved, given uncertainties
in assessment.109
[27] Although it appears that the TREC results are better than those
previously reported in the literature, either for manual or technology-
assisted review, they do not include any direct comparison between
manual and technology-assisted review.110 To draw any firm conclusion
that one is superior to the other, one must compare manual and
technology-assisted review efforts using the same information needs, the
same dataset, and the same evaluation standard.111 The Roitblat, Kershaw,
and Oot study is the only peer-reviewed study known to the Authors
suggesting that technology-assisted review may be superior to manual
review – if only in terms of precision, and only by a small amount – based
on a common information need, a common dataset, and a common gold
standard, albeit one of questionable accuracy.112
[28] This Article shows conclusively that the H5 and Waterloo efforts
are superior to manual reviews conducted contemporaneously by TREC
assessors, using the same topics, the same datasets, and the same gold
standard. The manual reviews considered for this Article were the “First-
Pass Assessments” undertaken at the request of the TREC coordinators for
108
See Hedin et al, supra note 9, at 17.
109
Voorhees, supra note 34, at 701.
110
See e.g., Oard et al., supra note 9, at 1-2.
111
See Voorhees, supra note 35, at 356 (“The [Cranfield] experimental design called for
the same set of documents and same set of information needs to be used for each [search
method], and for the use of both precision and recall to evaluate the effectiveness of the
search.”).
112
See Roitblat et al., supra note 7, at 76 (“The use of precision and recall implies the
availability of a stable ground truth against which to compare the assessments. Given the
known variability of human judgments, we do not believe that we have a solid enough
foundation to claim that we know which documents are truly relevant and which are
not.”).
24
the purpose of evaluating the participating teams’ submissions.113 In

comparing the manual and technology-assisted reviews, the Authors used
exactly the same adjudicated gold standard as TREC.114
III. TREC Legal Track Interactive Task
[29] TREC is an annual event hosted by NIST, with the following

objectives:
 to encourage research in information retrieval based on

large test collections;
 to increase communication among industry, academia,
and government by creating an open forum for the
exchange of research ideas;
 to speed the transfer of technology from research labs
into commercial products by demonstrating substantial
improvements in retrieval methodologies on real-world
problems; and
 to increase the availability of appropriate evaluation
techniques for use by industry and academia, including
development of new evaluation techniques more
applicable to current systems.115
Since its inception in 2006,116 the TREC Legal Track has had the goal “to
develop search technology that meets the needs of lawyers to engage in
effective discovery in digital document collections.”117
113
Hedin et al., supra note 9, at 3 (describing the “First-Pass Assessment” process).
114
See id. at 3-4.
115
Text REtrieval Conference (TREC), Overview, NAT’L INST. STANDARDS & TECH.,
http://trec.nist.gov/overview.html (last updated Aug. 10, 2010).
116
See Jason R. Baron, The TREC Legal Track: Origins and Reflections on the First
Year, 8 SEDONA CONF. J. 251, 253 (2007); see also Jason R. Baron et al., TREC-2006
Legal Track Overview, in NIST SPECIAL PUBLICATION: SP 500-272, THE FIFTEENTH
TEXT RETRIEVAL CONFERENCE (TREC 2006) PROCEEDINGS 1-2 (2006), available at
http://trec.nist.gov/pubs/trec15/papers/LEGAL06.OVERVIEW.pdf.
25
[30] Within the TREC Legal Track, the Interactive Task simulates the
process of review of a large population of documents for responsiveness to
one or more discovery requests in a civil litigation.118 In 2008, the first
year of the Interactive Task,119 the population of documents used was the
“Illinois Institute of Technology Complex Document Information
Processing Test Collection, version 1.0” (“IIT CDIP”),120 consisting of
about seven million documents that were released in connection with
various lawsuits filed against certain U.S. tobacco companies and
affiliated research institutes.121 A mock complaint and three associated
requests for production (or topics) were composed for the purposes of the
Interactive Task.122 Participating teams were required to produce the
responsive documents for one or more of the three requests.123
[31] The population of documents used for TREC 2009 consisted of

e-mail messages and attachments that Enron produced in response to
requests by FERC.124 A mock complaint and seven associated requests for
production were composed for the purposes of TREC 2009.125
Participating teams requested as many topics as they desired to undertake,
but time and cost constraints limited the number of topics that any team
was assigned to a maximum of four.126
117
Text Retrieval Conference (TREC), TREC Tracks, NAT’L INST. STANDARDS & TECH.,
http://trec.nist.gov/tracks.html (last updated Feb. 24, 2011).
118
See Oard et al., supra note 9, at 20.
119
See id. at 2.
120
Id. at 3; see Complex Document Image Processing (CDIP), supra note 94.
121
See Oard et al., supra note 9, at 3; Complex Document Image Processing (CDIP),
supra note 93.
122
See Oard et al., supra note 9 at 3, 24.
123
Id. at 24.
124
See Hedin et al., supra note 9, at 4; see also FERC, supra note 101.
125
See Hedin et al., supra note 9, at 5-6.
126
See id. at 7 tbl.1.
26
[32] Aside from the document collections, the mock complaints, and
the production requests, the conduct of the 2008 and 2009 Interactive
Tasks was identical.127 Participating teams were given the document
collection, the complaint, and the production requests several weeks
before production was due.128 Teams were allowed to use any
combination of technology and human input; the exact combination
differed from team to team.129 However, the size of the document
population, along with time and cost constraints, rendered it infeasible for
any team to conduct an exhaustive review of every document.130 To the
Authors’ knowledge, no team examined more than a small percentage of
the document population; H5 and Waterloo, in particular, used various
combinations of computer search, knowledge engineering, machine
learning, and sampling to select documents for manual review.131
[33] To aid the teams in their efforts, as well as to render an

authoritative interpretation of responsiveness (or relevance, within the
context of TREC), a volunteer Topic Authority (“TA”) – a senior attorney
familiar with the subject matter – was assigned for each topic.132 The TA
played three critical roles:
 to consult with the participating teams to clarify the notion of

relevance, in a manner chosen by the teams;
127
See id. at 1-2.
128
See Text Retrieval Conference (TREC), TREC-2008 Legal Track Interactive Task:
Guidelines, 8, 17 (2008), trec-legal.umiacs.umd.edu/2008InteractiveGuidelines.pdf
[hereinafter TREC-2008 Guidelines]; see also E-mail from Dan Brassil to Maura R.
Grossman, supra note 105.
129
TREC-2008 Guidelines, supra note 128, at 4, 7; see also E-mail from Bruce Hedin to
Gordon V. Cormack (Apr. 07, 2011 00:56 EDT) (confirming that teams were permitted
to use any combination of technology and human input).
130
See TREC-2008 Legal Track Interactive Task: Guidelines, supra note 128, at 8.
131
See Hogan et al., supra note 9, at 5; Cormack & Mojdeh, supra note 98, at 6.
132
27
 to prepare a set of written guidelines used by the human

reviewers to evaluate, after the fact, the relevance of
documents produced by the teams; and
 to act as a final arbiter of relevance in the adjudication

process.133
[34] The TREC coordinators evaluated the various participant efforts

using estimates of recall, precision, and F1 based on a two-pass human
assessment process.134 In the first pass, human reviewers assessed a
stratified sample of about 7,000 documents for relevance.135 For some
topics (Topics 201, 202, 205, and 206), the reviewers were primarily
volunteer law students supervised by the TREC coordinators; for others
(Topics 203, 204, and 207), the reviewers were lawyers employed and
supervised by professional document-review companies, who volunteered
their services.136
[35] The TREC coordinators released the first-pass assessments to

participating teams, which were invited to appeal relevance determinations
with which they disagreed.137 For each topic, the TA adjudicated the
appeals, and the TA’s opinion was deemed to be correct and final.138 The
gold standard of relevance for the documents in each sample was
therefore:
 The same as the first-pass assessment, for any document that

participants did not appeal; or
133
Id. at 2-3; see Oard et al., supra note 9, at 20.
134
Hedin et al., supra note 9, at 3-4.
135
See id. at 12-14.
136
Id. at 8.
137
Id. at 3.
138
Id.
28
 The TA’s opinion, for any document that participants did

appeal.
The TREC coordinators used statistical inference to estimate recall,

precision, and F1 for the results each participating team produced.139
[36] Assuming participants diligently appealed the first-pass

assessments with which they disagreed, it is reasonable to conclude that
TREC’s two-pass assessment process yields a reasonably accurate gold
standard. Moreover, that same gold standard is suitable to evaluate not
only the participants’ submissions, but also the first-pass assessments of
the human reviewers.140
[37] Parts III.A and III.B briefly describe the processes employed by
the two participants whose results this Article compares to manual review.
Notably, the methods the two participants used differ substantially from
those typically described in the industry as “clustering” or “concept
search.”141
A. H5 Participation
[38] At TREC 2009, H5 completed one topic (Topic 204).142

According to Dan Brassil of H5, the H5 process involves three steps: (i)
“definition of relevance,” (ii) “partly-automated design of deterministic
queries,” and (iii) “measurement of precision and recall.”143 “Once
relevance is defined, the two remaining processes of (1) sampling and
query design and (2) measurement of precision and recall are conducted
139
Id. at 3, 11-16.
140
See Hedin et al., supra note 9, at 13 (describing the construction of the gold standard).
141
Sedona Search Commentary, supra note 1, at 202-03.
142
Hedin et al., supra note 9, at 6-7.
143
E-mail from Dan Brassil to Maura R. Grossman, supra note 105.
29
iteratively – ‘allowing for query refinement and correction’ – until the

clients’ accuracy requirements are met.”144
[39] H5 describes how its approach differs from other information

retrieval methods as follows:
It utilizes an iterative issue-focusing and data-

focusing methodology that defines relevancy in detail; most
alternative processes provide a reductionist view of
relevance (e.g.: a traditional coding manual), or assume that
different individuals share a common understanding of
relevance.
[H5’s approach] is deterministic: each document is
assessed against the relevance criteria and a relevant / not
relevant determination is made. . . .
[The approach] is built on precision: whereas many
alternative approaches start with a small number [of]
keywords intended to be broad so as to capture a lot of
relevant data (with the consequence of many false
positives), H5’s approach is focused on developing in an
automated or semi-automated fashion large numbers of
deterministic queries that are very precise: each string may
capture just a few documents, but nearly all documents so
captured will be relevant; and all the strings together will
capture most relevant documents in the collection.145
In the course of its TREC 2009 effort, H5 sampled and reviewed a total of
20,000 documents.146 H5 declined to quantify the number of person-hours
144
Id.
145
Id. (citing Dan Brassil et al., The Centrality of User Modeling to High Recall with
High Precision Search, in 2009 IEEE Int’l Conf. on Systems, Man, and Cybernetics, 91,
91-96.
146
Id.
30
it expended during the seven to eight week time period between the
assignment of the topic and the final submission date.147
B. Waterloo Participation
[40] The University of Waterloo (“Waterloo”) completed four topics

(Topics 201, 202, 203, and 207).148 Waterloo’s approach consisted of
three phases: (i) “interactive search and judging,” (ii) “active learning,”
and (iii) recall estimation.149 The interactive search and judging phase
“used essentially the same tools and approach [Waterloo] used in TREC
6.”150 Waterloo coupled the Wumpus search engine151 to a custom web
interface that provided document excerpts and permitted assessments to be
coded with a single mouse click.152 Over the four topics, roughly 12,500
documents were retrieved and reviewed, at an average rate of about 3
documents per minute (about 22 seconds per document; 76 hours in
147
Id.; E-mail from Dan Brassil to Maura R. Grossman (Feb. 16, 2011 15:58 EST) (on
file with authors).
148
Cormack & Mojdeh, supra 98, at 2.
149
Id. at 1-3.
150
Id. at 2. See generally, Gordon V. Cormack et al., Efficient Construction of Large
Test Collections, in SIGIR ’98 PROCEEDINGS OF THE 21ST ANNUAL INTERNATIONAL
ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION
RETRIEVAL 282, 284 (1998).
151
Welcome to the Wumpus Search Engine!, WUMPUS, http://www.wumpussearch.org/
(last visited Apr. 11, 2011).
152
See Cormack & Mojdeh, supra note 98, at 3 & fig.2; see also infra Figure 1. “We
used the Wumpus search engine and a custom html interface that showed hits-in-context
and radio buttons for adjudication . . . . Available for reference were links to the full text
of the document and to the full email message containing the document, including
attachments in their native format.” Cormack & Mojdeh, supra note 98, at 3.
31
total).153 Waterloo used the resulting assessments to train an on-line active

learning system, previously developed for spam filtering.154
[41] The active learning system “yields an estimate of the [probability]

that each document is relevant.”155 Waterloo developed an “efficient user
interface to review documents selected by this relevance score” (see
Figure 2).156 “The primary approach was to examine unjudged documents
in decreasing order of score, skipping previously adjudicated
documents.”157 The process displayed each document as text and, using a
single keystroke, coded each document as relevant or not relevant.158
Among the four topics, “[a]bout 50,000 documents were reviewed, at an
average rate of 20 documents per minute (3 seconds per document)” or 42
hours in total.159 “From time to time, [Waterloo] revisited the interactive
search and judging system, to augment or correct the relevance
assessments as new information came to light.”160
153
E-mail from Gordon V. Cormack to K. Krasnow Waterman (Feb. 24, 2010 08:25
EST) (on file with authors) (indicating that 12,508 documents were reviewed at a rate of
22 seconds per document, i.e., 76.44 hours in total).
154
Cormack & Mojdeh, supra note 98, at 3.
155
Id. at 3.
156
Id.
157
Id.
158
Id.
159
Cormack & Mojdeh, supra note 98, at 3.
160
Id.
32
Figure 1: Waterloo’s interactive search and judging interface.161
[42] The third and final phase estimated the density of relevant
documents as a function of the score assigned by the active learning
system, based on the assessments rendered during the active learning
phase.162 Waterloo used this estimate to gauge the tradeoff between recall
and precision, and to determine the number of documents to produce so as
to optimize F1, as required by the task guidelines.163
161
Id. at 3 & fig.2.
162
See id. at 6.
163
Id. at 3, 6; see Hedin et al., supra note 9, at 3.
33
[43] For Waterloo’s TREC 2009 effort, the end result was that a human
reviewed every document produced;164 however, the number of documents
reviewed was a small fraction of the entire document population (14,396
of the 836,165 documents were reviewed, on average, per topic).165 Total
review time for all phases was about 118 hours; 30 hours per topic, on
average.166
Figure 2: Waterloo’s minimalist review interface.167
164
See Cormack & Mojdeh supra note 98, at 6 (“the optimal strategy was to include no
unassessed documents”).
165
Id., at 6 tbl.2; E-mail from Bruce Hedin to Gordon V. Cormack, supra note 101 (“I
have attached full list of the 836,165 document-level IDs”).
166
118 hours is the sum of 76 hours for the interactive search and judging phase (supra
para. 39) and 42 hours for the active learning phase (supra para. 41). Since Waterloo did
four topics, the average effort per topic was 29.5 hours.
167
Cormack & Mojdeh, supra note 98, at 4 fig.3.
34
IV. QUANTITATIVE ANALYSIS
[44] This Article’s purpose is to refute the hypothesis that manual

review is the best approach by showing that technology-assisted review
can yield results that are more nearly complete and more accurate than
exhaustive manual review, as measured by recall, precision, and F1. To
compare technology-assisted to manual review, the study required:
1. The results of one or more technology-assisted reviews. For this

purpose, the Authors used the H5 review and the four Waterloo
reviews conducted during the course of their participation in the
TREC 2009 Legal Track Interactive Task.168
2. The results of manual reviews for the same topics and datasets as
the technology-assisted reviews. For this purpose, the Authors
used the manual reviews that TREC conducted on document
samples for the purpose of evaluating the results that the
participating teams submitted.169
3. A gold standard determination of relevance or nonrelevance. For

this purpose, the Authors used the TREC final adjudicated
assessments, for which the TA was the ultimate arbiter.170
[45] The Authors evaluated the results of the technology-assisted

reviews and the manual reviews in exactly the same manner, using the
168
The TREC results are available online, but use, dissemination and publication of the
material is limited. Text REtrieval Conference (TREC), Past Results, NAT’L INST.
STANDARDS & TECH., http://trec.nist.gov/results.html (last visited Apr. 11, 2011)
(“Individuals may request access to the protected area containing the raw results by
contacting the TREC Program Manager. Before receiving access, individuals will be
asked to sign an agreement that acknowledges the limited uses for which the data can be
used.”).
169
Text REtrieval Conference (TREC), Relevance Judgments and Evaluation
Tools for the Interactive Task, NAT’L INST. STANDARDS & TECH.,
http://trec.nist.gov/data/legal/09/evalInt09.zip (last visited Apr. 11, 2011).
170
Id.; see Hedin et al., supra note 9, at 2-3.
35
TREC methodology and the TREC gold standard.171 To compare the

effectiveness of the reviews, this Article reports, for each topic:
1. Recall, precision, and F1 for both the technology-assisted and

manual reviews.172
2. The difference in recall, the difference in precision, and the

difference in F1 between the technology-assisted and manual
reviews.173
3. The significance of the difference for each measure, expressed as

P.174 Traditionally, P < 0.05 is interpreted to mean that the
difference is statistically significant; P > 0.1 is interpreted to mean
that the measured difference is not statistically significant. Smaller
values of P imply stronger significance; P < 0.001 indicates
overwhelming significance.175 The Authors used 100 bootstrap
samples of paired differences to estimate the standard error of
measurement, assuming a two-tailed normal distribution, to
compute P.176
Table 7 shows recall, precision, and F1 for the technology-assisted and

manual reviews for each of the five topics, as well as the overall average
for the five technology-assisted reviews and the five manual reviews. For
brevity, the difference in each measure is not shown, but is easily
171
See Hedin et al., supra note 9, at 2-5.
172
See id. at 3 (reporting recall, precision, and F1 for TREC participants); infra Table 7
(reporting recall, precision, and F1 for the TREC manual reviews).
173
See infra Table 7. A positive difference in some measure indicates that the
technology-assisted review is superior in that measure, while a negative difference
indicates that it is inferior.
174
BÜTTCHER ET AL., supra note 19, at 426.
175
See id.
176
See id. at 412-31. “The bootstrap . . . is a method for simulating an empirical
distribution modeling f (S) by sampling the sample s.”). Id. at 424.
36
computed from the table. For example, for Topic 201, the difference in
recall between Waterloo and TREC is 77.8% – 75.6% = +2.2%.
Topic Team Recall Precision F1

201 Waterloo (†) 77.8% (*) 91.2% (*) 84.0%
TREC (Law Students) 75.6% 5.0% 9.5%
202 Waterloo 67.3% (*) 88.4% (*) 76.4%
TREC (Law Students) (†) 79.9% 26.7% 40.0%
203 Waterloo (*) 86.5% (*) 69.2% (*) 76.9%
TREC (Professionals) 25.2% 12.5% 16.7%
204 H5 (*) 76.2% (*) 84.4% (*) 80.1%
TREC (Professionals) 36.9% 25.5% 30.2%
207 Waterloo 76.1% (†) 90.7% 82.8%
TREC (Professionals) (†) 79.0% 89.0% (†) 83.7%
Avg. H5/Waterloo (†) 76.7% (*) 84.7% (*) 80.0%
TREC 59.3% 31.7% 36.0%
Table 7: Effectiveness of TREC 2009 Legal Track technology-assisted

approaches (H5 and Waterloo) compared to exhaustive manual reviews (TREC).
Results marked (*) are superior and overwhelmingly significant (P < 0.001).
177
Results marked (†) are superior but not statistically significant (P > 0.1).
[46] For each topic and each measure, the larger value is marked with
either (*) or (†); (*) indicates that the measured difference is
overwhelmingly significant (P < 0.001), while (†) indicates that it is not
statistically significant (P > 0.1). As Table 7 illustrates, all of the
measured differences are either overwhelmingly significant or not
statistically significant.178
V. QUALITATIVE ANALYSIS
[47] The quantitative results show that the recall of the manual reviews
varies from about 25% (Topic 203) to about 80% (Topic 202). That is,
human assessors missed between 20% and 75% of all relevant
documents.179 Is this shortfall the result of clerical error, a
177
For the information contained in this table, see Past Results, supra note 168;
Relevance Judgments and Evaluation Tools for the Interactive Task, supra note 169. For
details on the calculation and meaning of P, see supra para. 43.
178
Supra Table 7.
179
See supra Table 7.
37
misinterpretation of relevance, or disagreement over marginal documents

whose responsiveness is debatable? If the missed documents are
marginal, the shortfall may be of little consequence; but if the missed
documents are clearly responsive, production may be inadequate, and
under Rule 37(a)(4), such a production could constitute a failure to
respond.180
[48] To address this question, the Authors examined the documents that
the TREC assessors coded as nonresponsive to Topics 204 and 207, but
H5 or Waterloo coded as responsive, and the TA adjudicated as
responsive. Recall from Table 5 that Topic 204 concerned shredding and
destruction of documents, while Topic 207 concerned football and
gambling. The Authors chose these topics because they were more likely
to be easily accessible to the reader, as opposed to other topics, which
were more technical in nature. In addition, lawyers employed by
professional review companies assessed these two topics using accepted
practices for manual review.181
[49] For Topic 204, 160 of the assessed documents were coded as
nonresponsive by the manual reviewers and responsive by H5 and the
TA;182 Topic 207, 51 documents met these same criteria except that
Waterloo and the TA made the responsiveness determinations.183 From
these numbers, the Authors extrapolated that the manual reviewers would
180
See FED. R. CIV. P. 37(a)(4).
181
See Hedin et al., supra note 9, at 8 (“The review of the samples for three of the seven
Interactive topics (203, 204, and 207) was carried out by two firms that include
professional document-review services among their offerings.”).
182
The Authors identified these documents by comparing the submitted results, see Past
Results, supra note 168 (file input.H52009.gz), the first-pass assessments, see Relevance
Judgments and Evaluation Tools for the Interactive Task, supra note 169 (file
qrels_doc_pre_all.txt), and the final adjudicated results, see id. (file
qrels_doc_post_all.txt).
183
The Authors identified these documents by comparing the submitted results, see Past
Results, supra note 168 (file input.watlint.gz), the first-pass assessments, see Relevance
Judgments and Evaluation Tools for the Interactive Task, supra note 169 (file
qrels_doc_pre_all.txt), and the final adjudicated results, see id. (file
qrels_doc_post_all.txt).
38
have missed 1,918 and 1,273 responsive documents (for Topics 204 and
207, respectively), had they reviewed the entire document collection.
[50] For each of these documents, the Authors used their judgment to
assess whether the document had been miscoded due to:
 Inarguable error: Under any reasonable interpretation of relevance,

the reviewer should have coded the document as responsive, but did
not. Possible reasons for such error include fatigue or inattention,
overlooking part of the document, poor comprehension, or data entry
mistakes in coding the document.184 For example, a document about
“shredding” (see Figure 3) is responsive on its face to Topic 204;
similarly “Fantasy Football” (see Figure 4) is responsive on its face to
Topic 207.
Date: Tuesday, January 22, 2002 11:31:39 GMT

Subject:
I’m in. I’ll be shredding ’till 11am so I should haveplenty of time to make it.
Figure 3: Topic 204 Inarguable error. A professional reviewer coded this document
as nonresponsive, although it clearly pertains to document shredding, as specified in
185
the production request.
184
Cf. Jeremy M. Wolfe et al., Low Target Prevalence Is a Stubborn Source of Errors in
Visual Search Tasks, 136 J. EXPERIMENTAL PSYCH. 623, 623-24 (2007) (showing that in
visual search tasks, humans have much higher error rates when the prevalence of target
items is low).
185
See supra Table 5. Figure 3 is an excerpt from document 0.7.47.1449689 in the
TREC 2009 dataset, supra note 101.
39
From: Bass, Eric

Sent: Thursday, January 17, 2002 11:19 AM
To: Lenhart, Matthew
Subject: FFL Dues
You owe $80 for fantasy football. When can you pay?
Figure 4: Topic 207 Inarguable error. A professional reviewer coded this document
as nonresponsive, although it clearly pertains to fantasy football, as specified in the
186
production request.
 Interpretive error: Under some reasonable interpretation of relevance

– but not the TA’s interpretation as provided in the topic guidelines –
an assessor might consider the document as nonresponsive. For
example, a reviewer might have construed an automated message
stating, “your mailbox is nearly full; please delete unwanted
messages” (see Figure 5) as nonresponsive to Topic 204, although the
TA defined it as responsive. Similarly, an assessor might have
construed a message concerning children’s football (see Figure 6) as
nonresponsive to Topic 207, although the TA defined it as responsive.
186
See supra Table 5. Figure 4 is an excerpt from document 0.7.47.320807 from the
40
WARNING: Your mailbox is approaching the size limit
This warning is sent automatically to inform you that

your mailbox is approaching the maximum size limit.
Your mailbox size is currently 79094 KB.
Mailbox size limits:
When your mailbox reaches 75000 KB you will receive this message.To check the
size of your mailbox:
Right-click the mailbox (Outlook Today),

Select Properties and click the Folder Size button.
This method can be used on individual folders as well.
To make more space available, delete any items that are no longer needed such as
Sent Items and Journal entries.
Figure 5: Topic 204 Interpretive error. A professional reviewer coded this automated
message as nonresponsive, although the TA construed such messages to be
187
responsive to Topic 204.
Subject: RE: Meet w/ Belden
I need to leave at 3:30 today to go to my stepson’s

football game. Unfortunately, I have a 2:00 and 3:00 meeting already. Is this just a
general catch-up discussion?
Figure 6: Topic 207 Interpretive error. The reviewer may have construed a children’s
league football game to be outside of the scope of “gambling on football.” The TA
188
deemed otherwise.
 Arguable error: Reasonable, informed assessors might disagree or find

it difficult to determine whether or not the document met the TA’s
conception of responsiveness (e.g., Figures 7 and 8).
187
188
See supra Table 5. Figure 6 is an excerpt from document 0.7.47.668065 in the TREC
2009 dataset, supra note 101.
41
Subject: Original Guarantees

Just a followup note:
We are still unclear as to whether we should continue to send original incoming and
outgoing guarantees to Global Contracts (which is what we have been doing for about
4 years, since the Corp. Secretary kicked us out of using their vault on 48 for originals
because we had too many documents). I think it would be good practice if Legal and
Credit sent the originals to the same place, so we will be able to find them when we
want them. So my question to y’all is, do you think we should send them to Global
Contracts, to you, or directly the the 48th floor vault (if they let us!).
Figure 7: Topic 204 Arguable error. This message concerns where to store particular
documents, not specifically their destruction or retention. Applying the TA’s
conception of relevance, reasonable, informed assessors might disagree as to its
189
responsiveness.
Subject: RE: How good is Temptation Island 2

They have some cute guy lawyers this year-but I bet you probably watch that manly
Monday night Football.
Figure 8: Topic 207 Arguable error. This message mentions football, but not a
specific football team, player, or game. Reasonable, informed reviewers might
disagree about whether or not it is responsive according to the TA’s conception of
190
relevance.
[51] When rendering assessments for the qualitative analysis, the

Authors considered the mock complaint,191 the topics,192 and the topic-
specific assessment guidelines memorializing the TA’s conception of
relevance, which were given to the human reviewers for reference
189
190
See supra Table 5. Figure 8 shows an excerpt from document 0.7.6.179483 in
the TREC 2009 dataset, supra note 101.
191
See generally Complaint, Grumby v. Volteron Corp., supra note 97.
192
Id. at 14; Hedin et al, supra note 9, at 5-6.
42
purposes.193 Table 8 summarizes the findings: The vast majority of

missed documents are attributable either to inarguable error or to
misinterpretation of the definition of relevance (interpretive error).
Remarkably, the findings identify only 4% of all errors as arguable.
Error Type
Topic Inarguable Interpretive Arguable Total
204 98 56 6 160
207 39 11 1 51
Total 137 67 7 211
Fraction 65% 31% 4% 100%
Table 8: Number of responsive documents that human reviewers missed,

categorized by the nature of the error. 65% of missed documents are relevant on
their face. 31% of missed documents are clearly relevant, when the topic-
specific guidelines are considered. Only 4% of missed documents, in the
opinion of the Authors, have debatable responsiveness, according to the topic-
194
specific guidelines.
VI. RESULTS AND DISCUSSION
[52] Tables 6 and 7 show that, by all measures, the average efficiency
and effectiveness of the five technology-assisted reviews surpasses that of
the five manual reviews. The technology-assisted reviews require, on
average, human review of only 1.9% of the documents, a fifty-fold savings
over exhaustive manual review. For F1 and precision, the measured
difference is overwhelmingly statistically significant (P < 0.001);195 for
recall the measured difference is not significant (P > 0.1).196 These
measurements provide strong evidence that the technology-assisted
193
Text REtrieval Conference (TREC), TREC-2009 Legal Track – Interactive Task,
Topic-Specific Guidelines – Topic 204, U. WATERLOO, http://plg1.cs.uwaterloo.ca/trec-
assess/TopicGuidelines_204.pdf (last updated Oct. 22, 2009); Text REtrieval Conference
(TREC), TREC-2009 Legal Track – Interactive Task, Topic-Specific Guidelines – Topic
207, U. WATERLOO, http://plg1.cs.uwaterloo.ca/trec-assess/TopicGuidelines_207_.pdf
(last updated Oct. 22, 2009).
194
See sources cited supra note 193.
195
See supra Tables 6, 7.
196
Id.
43
processes studied here yield better overall results, and better precision, in
particular, than the TREC manual review process. The measurements also
suggest that the technology-assisted processes may yield better recall, but
the statistical evidence is insufficiently strong to support a firm conclusion
to this effect.
[53] It should be noted that the objective of TREC participants was to

maximize F1, not recall or precision, per se.197 It happens that they
achieved, on average, higher precision.198 Had the participants considered
recall to be more important, they might have traded off precision (and
possibly F1) for recall, by using a broader interpretation of relevance, or
by adjusting a sensitivity parameter in their software.
[54] Table 7 shows that, for four of the five topics, the technology-
assisted processes achieve substantially higher F1 scores, largely due to
their high precision. Nonetheless, for a majority of the topics, the
technology-assisted processes achieve higher recall as well; for two topics,
substantially higher.199 For Topic 207, there is no meaningful difference
in effectiveness between the technology-assisted and manual reviews, for
any of the three measures. There is not one single measure for which
manual review is significantly better than technology-assisted review.
[55] For three of the five topics (Topics 201, 202, and 207) the results
show no significant difference in recall between the technology-assisted
and manual reviews. This result is perhaps not surprising, since the recall
scores are all on the order of 70% – the best that might be reasonably
achieved, given the level of agreement among human assessors. As such,
the results support the conclusion that technology-assisted review can
achieve at least as high recall as manual review, and higher precision, at a
fraction of the review effort, and hence, a fraction of the cost.
197
198
See supra Tables 6, 7.
199
See supra Table 7.
44
VII. LIMITATIONS
[56] The 2009 TREC effort used a mock complaint and production
requests composed by lawyers to be as realistic as possible.200
Furthermore, the role of the TA was intended to simulate that of a senior
attorney overseeing a real document review.201 Finally, the dataset
consisted of real e-mail messages captured within the context of an actual
investigation.202 These components of the study are perhaps as realistic as
might reasonably be achieved outside of an actual legal setting.203 One
possible limitation is that the Enron story, and the Enron dataset, are both
well known, particularly since the Enron documents are frequently used in
vendor product demonstrations.204 Both participants and TAs may have
had prior knowledge of both the story and dataset, affecting their strategies
and assessments. In addition, there is a tremendous body of extrinsic
information that may have influenced participants and assessors alike,
including the results of the actual proceedings, commentaries,205 books,206
200
201
See id.; see also Oard et al., supra note 9, at 20.
202
203
See id.
204
See, e.g., John Markoff, Armies of Expensive Lawyers Replaced by Cheaper
Software, N.Y. TIMES, Mar. 5, 2011, A1, available at
http://www.nytimes.com/2011/03/05/science/05legal.html; see also E-mail from Jonathan
Nystrom to Maura R. Grossman (Apr. 5, 2011 19:12 EDT) (on file with authors)
(confirming use of the Enron data set for product demonstrations); E-mail from Jim
Renehan to Maura R. Grossman (Apr. 5, 2011 20:06 EDT) (on file with authors)
(confirming use of the Enron data set for product demonstrations); E-mail from Lisa
Schofield to Maura R. Grossman (Apr. 5, 2011 18:27 EDT) (on file with authors)
(confirming use of the Enron data set for product demonstrations); E-mail from Edward
Stroz to Maura R. Grossman (Apr. 5, 2011 18:32 EDT) (on file with authors) (confirming
use of the Enron data set for product demonstrations).
205
See, e.g., John C. Coffee Jr., What Caused Enron?: A Capsule Social and Economic
History of the 1990’s, 89 CORNELL L. REV. 269 (2004); Paul M. Healy & Krishna G.
Palepu, The Fall of Enron, 17 J. ECON. PERSP. 3 (2003).
45
and even a popular movie.207 It is unclear what effect, if any, these factors
may have had on the results.
[57] In general, the TREC teams were privy to less detailed guidance
than the manual reviewers, placing the technology-assisted processes at a
disadvantage. For example, Topic 202 required the production of
documents related to “transactions that the Company characterized as
compliant with FAS 140.”208 Participating teams were required to
undertake research to identify the relevant transactions, as well as the
names of the parties, counterparties, and entities involved.209 Manual
reviewers, on the other hand, were given detailed guidelines specifying
these elements.210
[58] Moreover, TREC conducted manual review on a stratified sample

containing a higher proportion of relevant documents than the collection
as a whole,211 and used statistical inference to evaluate the result of
reviewing every document in the collection.212 Beyond the statistical
uncertainty, there also is uncertainty as to whether manual reviewers
would have had the same error rate had they reviewed the entire
collection. It is not unreasonable to think that, because the proportion of
relevant documents would have been lower in the collection than it was in
the sample, reviewer recall and precision might have been even lower,
because reviewers would have tended to miss the needles in the haystacks
due to fatigue, inattention, boredom, and related human factors. This
206
See, e.g., LOREN FOX, ENRON: THE RISE AND FALL (2002); BETHANY MCLEAN AND
PETER ELKIND, THE SMARTEST GUYS IN THE ROOM: THE AMAZING RISE AND
SCANDALOUS FALL OF ENRON (2003).
207
ENRON: THE SMARTEST GUYS IN THE ROOM (Magnolia Pictures 2005).
208
209
See id. at 8.
210
See id. at 3.
211
See id. at 12, tbl.3.
212
See generally id.
46
sampling effect, combined with the greater guidance provided to the

human reviewers, may have resulted in an overestimate of the
effectiveness of manual review, and thus understated the results of the
study.
[59] Of note is the fact that the appeals process involved

reconsideration – and potential reversal – only of manual coding decisions
that one or more participating teams appealed, presumably because their
results disagreed with the manual reviewers’ decisions.213 The appeals
process depended on participants exercising due diligence in identifying
the assessments with which they disagreed.214 And while it appears that
H5 and Waterloo exercised such diligence, it became apparent to the
Authors during the course of their analysis that a few assessor errors were
overlooked.215 These erroneous assessments were deemed correct under
the gold standard, with the net effect of overstating the effectiveness of
manual reviews, while understating the effectiveness of technology-
assisted review.216 It is also likely that the manual review and technology-
assisted processes incorrectly coded some documents that were not
appealed.217 The impact of the resulting errors on the gold standard would
be to overstate both recall and precision for manual review, as well as for
technology-assisted review, with no net advantage to either.
213
See Hedin et al., supra note 9 at 3, 13-14. There is no benefit, and therefore no
incentive, for participating teams to appeal coding decisions with which they agree.
214
See id. If participating teams do not appeal the manual reviewers’ incorrect decisions,
those incorrect decisions will be incorporated into the gold standard, compromising its
accuracy and usefulness.
215
Hedin et al., supra note 9 at 14, tbl.4 (showing that for every topic, H5 and Waterloo
appealed the majority of disagreements between their results and the manual
assessments).
216
See supra note 214. If the manual review is incorrect, and the technology-assisted
review is correct, the results will overstate the effectiveness of manual review at the
expense of technology-assisted review.
217
Given that neither the manual reviewers nor the technology-assisted processes are
infallible, it stands to reason that they may occasionally agree on coding decisions that
are incorrect.
47
[60] In designing this study, the Authors considered only the results of
two of the eleven teams participating in TREC 2009, because they were
considered most likely to demonstrate that technology-assisted review can
improve upon exhaustive manual review. The study considered all
submissions by these two teams, which happened to be the most effective
submissions for five of the seven topics. The study did not consider
Topics 205 and 206, because neither H5 nor Waterloo submitted results
for them. Furthermore, due to a dearth of appeals, there was no reliable
gold standard for Topic 206.218 The Authors were aware before
conducting their analysis that the H5 and Waterloo submissions were the
most effective for their respective topics. To show that the results are
significant in spite of this prior knowledge, the Authors applied
Bonferroni correction,219 which multiplies P by 11, the number of
participating teams. Even under Bonferroni correction, the results are
overwhelmingly significant.
VIII. CONCLUSION
[61] Overall, the myth that exhaustive manual review is the most
effective – and therefore, the most defensible – approach to document
review is strongly refuted. Technology-assisted review can (and does)
yield more accurate results than exhaustive manual review, with much
lower effort. Of course, not all technology-assisted reviews (and not all
manual reviews) are created equal. The particular processes found to be
superior in this study are both interactive, employing a combination of
computer and human input. While these processes require the review of
orders of magnitude fewer documents than exhaustive manual review,
neither entails the naïve application of technology absent human
judgment. Future work may address which technology-assisted review
process(es) will improve most on manual review, not whether technology-
assisted review can improve on manual review.
218
Hedin et al., supra note 9, at 17-18 (“Topic 206 represents the one topic, out of the
seven featured in the 2009 exercise, for which we believe the post-adjudication results are
not reliable. . . . We do not believe, therefore, that any valid conclusions can be drawn
from the scores recorded for this topic . . . .”).
219
See BÜTTCHER ET AL., supra note 19, at 428.
48
FEDERAL COURTS LAW REVIEW
Volume 7, Issue 1 2013
THE GROSSMAN-CORMACK GLOSSARY OF

TECHNOLOGY-ASSISTED REVIEW
with
Foreword by John M. Facciola, U.S. Magistrate Judge
FOREWORD
“When I use a word,” Humpty Dumpty said, in rather a scornful tone,

“it means just what I choose it to mean—neither more nor less.”
“The question is,” said Alice, “whether you can make words mean so many
different things.”
Lewis Carroll, Through the Looking Glass, What Alice Found There 1
“A word is not a crystal, transparent and unchanged, it is the skin of a

living thought and may vary greatly in color and content according to the
circumstances and the time in which it is used.”
Justice Oliver Wendell Holmes Jr., Towne v. Eisner, 245 U.S. 418,
425 (1918)
In the heels of the higgling lawyers, Bob,

Too many slippery ifs and buts and howevers,
Too much hereinbefore provided whereas,
Too many doors to go in and out of.
When the lawyers are through

What is there left, Bob?
Can a mouse nibble at it
And find enough to fasten a tooth in?
Carl Sandburg, The Lawyers Know Too Much 2
1. THE COLLECTED STORIES OF LEWIS CARROLL 238 (Citadel Press 1994).

2. THE COMPLETE POEMS OF CARL SANDBURG 189 (Harcourt, Brace and Co., rev. ed.
1970).
2 FEDERAL COURTS LAW REVIEW [Vol. 7
It is always the words. Humpty Dumpty, Holmes, and Sandburg, who

sensed the power of language instinctively, knew how quickly their
meanings can slip away. What seemed clear when the contract or statute
was drafted is now recondite. When the parties contracted to buy and sell a
horse, did they mean a mare or a stallion? When the statute required that a
lawyer be disbarred upon conviction of a crime of moral turpitude, did the
legislature mean a lawyer who gets convicted of assault after a brawl in a
bar? Lawyers and judges are mocked for their continued use of Latin but
they know that it is so much easier to say “res judicata” and take advantage
of the encrusted meaning of those words than to start fresh and try to
improve on what they convey. Law books come and go but Black’s Law
Dictionary will always be around. The words, as Paul Simon might put it,
keep “slip, sliding away.” 3
It is hard enough when the world in which the words are used remains
static, like the farm on which the horse, be it mare or stallion, lives. But,
what happens when the movement of technology radically transforms what
a word might have once meant? What is the “original” of an e-mail? Is
another e-mail a copy of it when the visible text is the same but the
metadata created in its production by a computer, rather than a human
being, is entirely different? What happens when the meaning of the words
in a statute applied to a process that was in existence when the statute was
enacted but now no longer exists? Some of the definitions in the Stored
Communications Act, 4 enacted in 1986, may drive judges to distraction
since they were premised on technology used in 1986 but is no longer and
must be applied to new processes that no one knew would exist when the
statute was enacted. What kind of words can be used in a statute or a rule
that are capacious enough to hold their meaning despite unknown
technological change but precise enough to convey a definite meaning? It
may be hard to believe, but there was not even a clear indication in the
Federal Rules of Civil Procedure until 2006 that “electronically stored
information” was within the scope of what a party had to produce in
3. PAUL SIMON, Slip Slidin’ Away, on THE ESSENTIAL PAUL SIMON (Warner Bros. 2007).
4. See, e.g., The definition of “remote computing service” in 18 U.S.C. § 2711(2) (“the
provision to the public of computer storage or processing services by means of electronic
communications systems”). As Professor Orin Kerr explains, the statute “freez[es] into law the
understanding of computer network use as of 1986.” Orin S. Kerr, A User’s Guide to the Stored
Communications Act, and a Legislator’s Guide to Amending It, 72 GEO. WASH. L. REV. 1208,
1214 (2004). In 1986, users would use remote computing services to outsource computing tasks
such as storing extra files or processing data when doing so was beyond the capacity of their
computers. Id. Given the storage and processing capacities of new computers and tablets, this
kind of distant processing capacity has disappeared. In its wake, however, are the difficult
questions of the application of the definitions in the Stored Communications Act to “cloud
computing.” See William Jeremy Robison, Free at What Cost? Cloud Computing Privacy under
the Stored Communications Act, 98 GEO. L.J. 1195, 1210, 1212-13 (2010).
2013] Glossary of Technology-Assisted Review 3
discovery. 5 Indeed, until 2006, the word “phonorecords” appeared in

Federal Rules of Civil Procedure 34(a) 6 which must have, at the time, 7
mystified anyone under 35.
The pace of technological change makes the situation worse. That
pace is astonishing. What was thought to be the impossible crossing of the
Atlantic Ocean in a plane and the landing of a man on the moon occurred in
my father’s lifetime. Yet, George Washington’s troops moved no faster
than Caesar’s, and it took thousands of years before human beings
discovered how to transmit messages by the telegraph using electricity. 8
Until then, a messenger, whether from Marathon to Athens, or through the
colonial towns of Massachusetts, had to deliver them by hand.
The legal system deals in words and the pace of technological process
is creating billions of them on a nearly daily basis, 9 creating a set of
problems that were unimaginable a few years ago.
The first problem is that this explosion of words has been matched by
the ever-increasing capacity of machines to capture and preserve them. It is
a simple fact that an iPod has much more memory, i.e., capability to store
information indefinitely, than the first computer10 which took up an entire
room. Additionally, the cost of storage is diminishing. A one-terabyte drive
can be purchased for about $100 11 and can hold what would otherwise be
hundreds of thousands of pages of paper. Indeed, now the information can
be kept on a distant server with the space rented from a vendor for that
purpose and retrieved by use of the Internet (“cloud computing”). For the
first time, it is cheaper for human beings to buy a new file cabinet and keep
more paper than to clean the useless clutter out of the old file cabinet to
make room for the new information.
5. Compare FED. R. CIV. P. 34(a)(1)(A) (West 2006 rev. ed.) (describing “any designated
documents or electronically stored information—including writings, drawings, graphs, charts,
photographs, sound recordings, images, and other data or data compilations—stored in any
medium from which information can be obtained either directly or, if necessary, after translation
by the responding party into a reasonably usable form” as discoverable) with FED. R. CIV. P.
34(a)(1)(A) (West 2006) (mentioning only “any designated documents (including writings,
drawings, graphs, charts, photographs, phonorecords, and other data compilations from which
information can be obtained, translated, if necessary, by the respondent through detection devices
into reasonably usable form”).
6. FED. R. CIV. P. 34(a)(1)(A) (West 2006).
7. Vinyl records are making a comeback; eight track tapes will not. See, e.g., Brian
Passey, Vinyl Records spin back into vogue, USATODAY.COM (Feb. 26, 2011).
8. See TOM STANDAGE, THE VICTORIAN INTERNET (Walker & Co. 1998).
9. See George L. Paul & Jason R. Baron, Information Inflation: Can the Legal System
Adapt?, 13 RICH. J.L. & TECH. 10, 14-23 (2007).
10. See, e.g., What is a Mainframe Operating System?, WISEGEEK, http://www.wisegeek.com/
what-is-a-mainframe-operating-system.htm (last visited Nov. 19, 2012).
11. See AMAZON, http://www.amazon.com/s/ref=nb_sb_ss_i_1_7?url=search-alias%3Daps&field-
keywords=1+terabyte+hard+drive&sprefix=1+terab%2Caps%2C227 (listing numerous options under $100
for a one-terabyte external hard drive).
This phenomenon has led to the consequence that litigants are

confronted with the often horrifying costs of searching through immense
amounts of data to find what they need. The Federal Rules of Civil
Procedure contemplate a system of demand and production; plaintiff asks
for all documents pertaining to the merger of companies A and B and
defendant either objects or produces them. But, when there are now
hundreds of thousands of documents that may meet the definition of
“pertinent,” how can defendant find them and not go bankrupt in the
process? Of course, defendant may be the victim of its own failure to
maintain a responsible record-keeping process in which a principled
decision-making process guides what will be kept and what will be thrown
out. And, as anyone knows who has ever cleaned out a closet or an old
hard drive, keeping everything is no solution. It only increases the expense
and cost of finding what you want or need. Nevertheless, the affordability
of cheap storage has led too many entities in our society to be quickly
overwhelmed by their inability to search for what they need, whether
because they need it to run their business or because they must produce it in
discovery. They may find that the cost of searching and producing is so
great that settling the lawsuit may be the only way out of an otherwise
impossible situation.
It is understandable that, nature and technology abhorring a vacuum, a
new scientific methodology has emerged to aid in the collection and
searching process. “Technology-Assisted Review,” called by its nickname
“Predictive Coding,” describes a process whereby computers are
programmed to search a large amount of data to find quickly and efficiently
the data that meet a particular requirement. Computer science and the
sciences of statistics and psychology inform its use. While it bruises the
human ego, scientists, including the authors of this glossary we are
publishing, have determined that machines are better at the task of making
such discoveries than humans. 12 Lawyers love to think that there is no
substitute for their reviewing each document page by page. Not only is
there a substitute, but an improvement. It is now indubitable that
technology-assisted review is an appreciably better and more accurate
means of searching a set of data. 13 That is hardly surprising news to those
judges and lawyers who have experienced the mind-numbing tedium of
reviewing large data sets only to find that one is seeing the same e-mail
12. See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in

E-Discovery Can Be More Effective and Efficient Than Exhaustive and Manual Review, XVII
RICH. J.L. & TECH. 11 (2011).
13. Id.
chain again and again, or worse, wading through mountains of data and
finding nothing of any pertinence to the case being litigated.
The great benefits of technology-assisted review, however, bring in
new concerns and questions for judges and lawyers. In a paper universe,
the manner in which a party searched through a file cabinet hardly raises
any significant issues. If what was produced appeared to be what was
demanded and there were no inexplicable gaps, that was that and the court
and parties moved on to other things. Now, the methodology of the use of
technology-assisted review may itself be in dispute, with the parties
controverted to each other’s use of a particular method or tool. Those
controversies have already lead to judicial decisions that have to grapple
with a wholly new way of searching and with scientific principles derived
from the science of statistics or other disciplines.14 Lawyers and judges
once again have to learn a whole new vocabulary to resolve the emerging
and inevitable battle of the “experts.”
To aid in the creation of that vocabulary, we publish with pride the
glossary created by Maura R. Grossman and Gordon V. Cormack who are
two of the most respected and acknowledged experts in this field. We agree
with them that the creation of a clear and common vocabulary is essential to
a comprehension of the legal issues at stake.
We are particularly gratified that the authors announced that they
intend the glossary to be interactive so that others can suggest additional
clarifications, revisions, and additions. We are certain that, if the
experience of the courts in the first few years of the information technology
revolution is any guide, the learning curve will be steep and that it must be
climbed quickly if courts are going to be able to resolve promptly the
controversies before them at the least expense. We are equally certain that
the bench and bar will find this glossary useful as this new science develops
and grows.
The Editors of the Federal Courts Law Review
By John M. Facciola, U.S. Magistrate Judge
14. See, e.g., EORHB, Inc. v. HOA Holdings, Civ. Ac. No. 7409-VCL (Del. Ch. Oct. 19,
2012); Kleen Prods. LLC v. Packaging Corp., Civ. No. 10C 5711, 2012 WL 4498465 at *84-85
(N.D. Ill. Sept. 28, 2012); In re Actos (Pioglitazone) Prods. Liab. Litig., MDL No. 6:11-md-2299
(W.D. La. July 27, 2012); Global Aerospace Inc. v. Landow Aviation, L.P., No. CL 61040 (Va.
Cir. Ct. Apr. 23, 2012); Moore v. Publicis Groupe & MSL Group, No. 11 Civ. 1279 (ACL) (AJP),
2012 WL 607412 (S.D.N.Y. Feb. 24, 2012).
PREAMBLE
“Disruptive technology” is a term that was coined by Harvard

Business School Professor Clayton M. Christensen, in his 1997 book, The
Innovator’s Dilemma, to describe a new technology that unexpectedly
displaces an established technology. The term is used in business and
technology literature to describe innovations that improve a product or
service in ways that the market did not expect, typically by designing for a
different set of consumers in the new market and later, by lowering prices in
the existing market. Products based on disruptive technologies are typically
cheaper to produce, simpler, smaller, better performing, more reliable, and
often more convenient to use. Technology-Assisted Review (TAR) is such
a disruptive technology. Because disruptive technologies differ from
sustaining technologies – ones that rely on incremental improvements to
established technologies – they bring with them new features, new
vernaculars, and other challenges.
The introduction of TAR into the legal community has brought with it
much confusion because different terms are being used to refer to the same
thing (e.g., “technology-assisted review,” “computer-assisted review,”
“computer-aided review,” “predictive coding,” and “content-based
advanced analytics,” to name but a few), and the same terms are also being
used to refer to different things (e.g., “seed set” and “control sample”).
Moreover, the introduction of complex statistical concepts and terms of art
from the science of information retrieval have resulted in widespread
misunderstanding and sometimes perversion of their intended meanings.
This glossary is written in an effort to bring order to chaos by
introducing a common framework and set of definitions for use by the
bench, the bar, and service providers. This glossary endeavors to be
comprehensive, but its definitions are necessarily brief. Interested readers
may look elsewhere for detailed information concerning any of these topics.
The terms in this glossary are presented in alphabetical order, with defined
terms in capital letters.
We envision this glossary to be a living, breathing work that will
evolve over time. Towards that end, we invite our colleagues in the
industry to send us comments on our definitions, as well as any additional
terms they would like to see included in the glossary, so that we can reach a
consensus on a consistent, common language relating to TAR. Comments
can be sent to us at mrgrossman@wlrk.com and gvcormac@uwaterloo.ca.
Subsequent versions of this glossary will be available online at
http://cormack.uwaterloo.ca/targlossary/.
The authors would like to acknowledge the helpful comments
provided by Craig Ball, Michael Levine, Ralph Losey, Amir Milo, and
Keith Roland on an earlier draft of this work. We are very grateful to

Magistrate Judge John M. Facciola for his enthusiastic support.
We hope that you will find this glossary useful.
Maura R. Grossman* Gordon V. Cormack

Wachtell, Lipton, Rosen & Katz University of Waterloo
New York, New York Waterloo, Ontario
January 2013
* The views expressed herein are solely those of the author and should not be attributed to her
firm or its clients.
THE GLOSSARY
Accept on Zero Error: A technique in which the training of a Machine

Learning method is gauged by taking a Sample after each training step, and
deeming the training process complete when the learning method codes a
Sample with 0% Error (i.e., 100% Accuracy).
Accuracy: The fraction of Documents that are correctly coded by a search

or review effort. Note that Accuracy + Error = 100%, and that Accuracy =
100% – Error. While high Accuracy is commonly advanced as evidence of
an effective search or review effort, its use can be misleading because it is
heavily influenced by Prevalence. Consider, for example, a Document
Population containing one million Documents, of which ten thousand (or
1%) are Relevant. A search or review effort that identified 100% of the
Documents as Not Relevant, and, therefore, found none of the Relevant
Documents, would have 99% Accuracy, belying the failure of that search or
review effort.
Active Learning: An Iterative Training regimen in which the Training Set

is repeatedly augmented by additional Documents chosen by the Machine
Learning Algorithm, and coded by one or more Subject Matter Expert(s).
Actos: See In Re: Actos.
Agreement: The fraction of all Documents that two reviewers code the
same way. While high Agreement is commonly advanced as evidence of an
effective review effort, its use can be misleading, for the same reason that
the use of Accuracy can be misleading. When the vast majority of
Documents in a Population are Not Relevant, a high level of Agreement
will be achieved when the reviewers agree that these Documents are Not
Relevant, irrespective of whether or not they agree that any of the Relevant
Documents are Relevant.
Algorithm: A formally specified series of computations that, when

executed, accomplishes a particular goal. The Algorithms used in E-
Discovery are implemented as computer software.
Area Under the ROC Curve (AUC): From Signal Detection Theory, a
summary measure used to assess the quality of Prioritization. AUC is the
Probability that a randomly chosen Relevant Document is given a higher
priority than a randomly chosen Non-Relevant Document. An AUC score
of 100% indicates a perfect ranking, in which all Relevant Documents have
higher priority than all Non-Relevant Documents. An AUC score of 50%

means the Prioritization is no better than chance.
Artificial Intelligence: An umbrella term for computer methods that

emulate human judgment. These include Machine Learning and
Knowledge Engineering, as well as Pattern Matching (e.g., voice, face, and
handwriting recognition), robotics, and game playing.
Bag of Words: A Feature Engineering method in which the Features of

each Document comprise the set of words contained in that Document.
Documents are determined to be Relevant or Not Relevant depending on
what words they contain. Elementary Keyword Search and Boolean Search
methods, as well as some Machine Learning methods, use the Bag of
Words model.
Bayes / Bayesian / Bayes’ Theorem: A general term used to describe

Algorithms and other methods that estimate the overall Probability of some
eventuality (e.g., that a Document is Relevant), based on the combination of
evidence gleaned from separate observations. In Electronic Discovery, the
most common evidence that is combined is the occurrence of particular
words in a Document. For example, a Bayesian Algorithm might combine
the evidence gleaned from the fact that a Document contains the words
“credit,” “default,” and “swap” to indicate that there is a 99% Probability
that the Document concerns financial derivatives, but only a 40%
Probability if the words “credit” and “default,” but not “swap,” are present.
The most elementary Bayesian Algorithm is Naïve Bayes; however, most
Algorithms dubbed “Bayesian” are more complex. Bayesian Algorithms
are named after Bayes’ Theorem, coined by the 18th century
mathematician, Thomas Bayes. Bayes’ Theorem derives the Probability of
an outcome, given the evidence, from: (i) the probability of the outcome,
independent of the evidence; (ii) the probability of the evidence, given the
outcome; and (iii) the probability of the evidence, independent of the
outcome.
Bayesian Classifier / Bayesian Filter / Bayesian Learning: A colloquial

term used to describe a Machine Learning Algorithm that uses a Bayesian
Algorithm resembling Naïve Bayes.
Bigram: An N-Gram where N = 2 (i.e., a 2-gram).
Binomial Calculator / Binomial Estimation: A statistical method used to

calculate Confidence Intervals, based on the Binomial Distribution, that
models the random selection of Documents from a large Population.

Binomial Estimation is generally more accurate, but less well known, than
Gaussian Estimation. A Binomial Estimate is substantially better than a
Gaussian Estimate (which, in contrast, relies on the Gaussian or Normal
Distribution) when there are few (or no) Relevant Documents in the
Sample. When there are many Relevant and many Non-Relevant
Documents in the Sample, Binomial and Gaussian Estimates are nearly
identical.
Binomial Distribution: The Probability that a Random Sample from a

large Population will contain any particular number of Relevant
Documents, given the Prevalence of Relevant Documents in the Population.
Used as the basis for Binomial Estimation.
Binomial Estimate: A Statistical Estimate of a Population characteristic

using Binomial Estimation. It is generally expressed as a Point Estimate
accompanied by a Margin of Error and a Confidence Level, or as a
Confidence Interval accompanied by a Confidence Level.
Blair and Maron: Authors of an influential 1985 study (David C. Blair &
M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text
Document-Retrieval System, 28 COMMC’NS ACM 289 (1985)), showing that
attorneys supervising skilled paralegals believed they had found at least
75% of the Relevant Documents from a Document Collection, using search
terms and iterative search, when they had in fact found only 20%. That is,
the searchers believed they had achieved 75% Recall, but had achieved only
20% Recall. In the Blair and Maron study, the attorneys and paralegals used
an iterative approach, examining the retrieved Documents and refining their
search terms until they believed they were done. Many current
commentators incorrectly distinguish the Blair and Maron study from
current iterative approaches, failing to note that the Blair and Maron
searchers did in fact refine their search terms based on their review of the
Documents that were returned in response to their queries.
Boolean Search: A Keyword Search in which the Keywords are combined

using operators such as “AND,” “OR,” and “[BUT] NOT.” The result of a
Boolean Search is precisely determined by the words contained in the
Documents. (See also Bag of Words.)
Bulk Coding: The process of Coding all members of a group of Documents

(identified, for example, by Deduplication, Near-Deduplication, Email
Threading, or Clustering) based on the review of only one or a few

members of the group. Also referred to as Bulk Tagging.
Bulk Tagging: See Bulk Coding.
Classical, Gaussian, or Normal Calculator / Classical, Gaussian, or

Normal Estimation: A method of calculating Confidence Intervals based
on the assumption that the quantities to be measured follow a Gaussian
(Normal) Distribution. This method is most commonly taught in
introductory statistics courses, but yields inaccurate Confidence Intervals
when the Prevalence of items with the characteristic being measured is low.
(Cf. Binomial Calculator / Binomial Estimation.)
Classifier / Classification / Classified / Classify: An Algorithm that

Labels items as to whether or not they have a particular property; the act of
Labeling items as to whether or not they have a particular property. In
Technology-Assisted Review, Classifiers are commonly used to Label
Documents as Responsive or Non-Responsive.
Clustering: An Unsupervised Learning method in which Documents are

segregated into categories or groups so that the Documents in any group are
more similar to one another than to those in other groups. Clustering
involves no human intervention, and the resulting categories may or may
not reflect distinctions that are valuable for the purpose of a search or
review effort.
Code / Coded / Coding: The action of Labeling a Document as Relevant or

Non-Relevant, or the set of Labels resulting from that action. Sometimes
interpreted narrowly to include only the result(s) of a Manual Review
effort; sometimes interpreted more broadly to include automated or semi-
automated Labeling efforts. Coding is generally the term used in the legal
industry; Labeling is the equivalent term in Information Retrieval.
Collection: See Document Collection.
Computer-Aided Review: See Technology-Assisted Review.
Computer-Assisted Review (CAR): See Technology-Assisted Review.
Concept Search: An industry-specific term generally used to describe

Keyword Expansion techniques, which allow search methods to return
Documents beyond those that would be returned by a simple Keyword or
Boolean Search. Methods range from simple techniques such as Stemming,

Thesaurus Expansion, and Ontology search, through statistical Algorithms
such as Latent Semantic Indexing.
Confidence Interval: As part of a Statistical Estimate, a range of values

estimated to contain the true value, with a particular Confidence Level.
Confidence Level: As part of a Statistical Estimate, the chance that a

Confidence Interval derived from a Random Sample will include the true
value. For example, “95% Confidence” means that if one were to draw 100
independent Random Samples of the same size, and compute the
Confidence Interval from each Sample, about 95 of the 100 Confidence
Intervals would contain the true value. It is important to note that the
Confidence Level is not the Probability that the true value is contained in
any particular Confidence Interval; it is the Probability that the method of
estimation will yield a Confidence Interval that contains the true value.
Confusion Matrix: A two-by-two table listing values for the number of

True Negatives (TN), False Negatives (FN), True Positives (TP), and False
Positives (FP) resulting from a search or review effort. As shown below, all
of the standard evaluation measures are algebraic combinations of the four
values in the Confusion Matrix. Also referred to as a Contingency Table.
An example of a Confusion Matrix (or Contingency Table) is provided
immediately below.
Coded Relevant Coded Non-Relevant

Truly Relevant True Positives (TP) False Negatives (FN)
Truly Non-Relevant False Positives (FP) True Negatives (TN)
Accuracy = 100% – Error = (TP + TN) / (TP + TN + FP + FN)

Elusion = 100% – Negative Predictive Value = FN / (FN + TN)
Error = 100% – Accuracy = (FP + FN) / (TP + TN + FP + FN)
Fallout = False Positive Rate = 100% – True Negative Rate = FP / (FP+TN)
False Negative Rate = 100% ‒ True Positive Rate = FN / (FN+TP)
Negative Predictive Value = 100% – Elusion = TN / (TN + FN)
Precision = Positive Predictive Value = TP / (TP + FP)
Prevalence = Yield = Richness = (TP + FN) / (TP + TN + FP + FN)
Recall = True Positive Rate = Sensitivity = TP / (TP+FN)
True Negative Rate = Specificity = TN / (TN + FP)
Content-Based Advanced Analytics (CBAA): See Technology-Assisted

Review.
Contingency Table: See Confusion Matrix.
Control Set: A Random Sample of Documents coded at the outset of a

search or review process that is separate from and independent of the
Training Set. Control Sets are used in some Technology-Assisted Review
processes. They are typically used to measure the effectiveness of the
Machine Learning Algorithm at various stages of training, and to determine
when training may cease.
Crossover Trial: An Experimental Design for comparing two search or

review processes using the same Document Collection and Information
Need, in which one process is applied first, followed by the second, and
then the results of the two efforts are compared. (Cf. Parallel Trial.)
Culling: The practice of narrowing a larger data set to a smaller data set for
the purposes of review, based on objective criteria (such as file types or date
restrictors), or subjective criteria (such as Keyword Search Terms).
Documents that do not match the criteria are excluded from the search and
from further review.
Cutoff: A given score or rank in a Prioritized list, resulting from a

Relevance Ranking search or Machine Learning Algorithm, such that the
Documents above the Cutoff are deemed to be Relevant and Documents
below the Cutoff are deemed to be Non-Relevant. In general, a higher
Cutoff will yield higher Precision and lower Recall, while a lower Cutoff
will yield lower Precision and higher Recall. Also referred to as a
Threshold.
Da Silva Moore: Da Silva Moore v. Publicis Groupe, Case No. 11 Civ.

1279 (ALC) (AJP), 2012 WL 607412 (S.D.N.Y. Feb. 24, 2012), aff’d, 2012
WL 1446534 (S.D.N.Y. Apr. 26, 2012). The first federal case to recognize
Computer-Assisted Review as “an acceptable way to search for relevant
ESI in appropriate cases.” The opinion was written by Magistrate Judge
Andrew J. Peck and affirmed by District Judge Andrew L. Carter.
Decision Tree: A step-by-step method of distinguishing between Relevant

and Non-Relevant Documents, depending on what combination of words
(or other Features) they contain. A Decision Tree to identify Documents
pertaining to financial derivatives might first determine whether or not a
Document contained the word “swap.” If it did, the Decision Tree might
then determine whether or not the Document contained the word “credit,”
and so on. A Decision Tree may be created through Knowledge
Engineering or Machine Learning.
Deduplication: A method of replacing multiple identical copies of a

Document by a single instance of that Document. Deduplication can occur
within the data of a single custodian (also referred to as Vertical
Deduplication), or across all custodians (also referred to as Horizontal
Deduplication).
Dimensionality Reduction: A Feature Engineering method used to reduce

the total number of Features considered by a Machine Learning Algorithm.
Simple Dimensionality Reduction methods include Stemming and Stop
Word elimination. More complex Dimensionality Reduction methods
include Latent Semantic Indexing and Hashing.
Document: In the context of Electronic Discovery, a discrete item of

Electronically Stored Information that may be the subject or result of a
search or review effort.
Document Collection: The process of gathering Electronically Stored

Information for search, review, and production; the set of Documents
resulting from such a process. In many cases, the Document Collection and
Document Population are the same; however, it is important to note that
Document Population refers to the set of Documents over which a particular
Statistical Estimate is calculated, which may be the entire Document
Collection, a subset of the Document Collection (e.g., the documents with a
particular file type or matching particular Search Terms), a superset of the
Document Collection (e.g., the universe from which the Document
Collection was gathered), or any combination thereof.
Document Population: The set of Electronically Stored Information or

Documents about which a Statistical Estimate may be made.
Early Case Assessment (ECA): An industry-specific term generally used

to describe a variety of tools or methods for investigating and quickly
learning about a Document Collection for the purposes of estimating the
risk(s) and cost(s) of pursuing a particular legal course of action.
EDI Study: See JASIST Study.

EDI-Oracle Study: An ongoing initiative (as of January 2013) of the

Electronic Discovery Institute to evaluate participating vendors’ search and
document review efforts using a Document Collection contributed by
Oracle America, Inc.
Electronic Discovery / E-Discovery: The process of identifying,

preserving, collecting, processing, searching, reviewing, and producing
Electronically Stored Information that may be Relevant to a civil, criminal,
or regulatory matter.
Electronically Stored Information (ESI): Used in Federal Rule of Civil

Procedure 34(a)(1)(A) to refer to discoverable information “stored in any
medium from which the information can be obtained either directly or, if
necessary, after translation by the responding party into a reasonably usable
form.” Although Rule 34(a)(1)(A) references “Documents or Electronically
Stored Information,” individual units of review and production are
commonly referred to as Documents, regardless of the medium.
Elusion: The fraction of Documents identified as Non-Relevant by a search

or review effort that are in fact Relevant. Elusion is estimated by taking a
Random Sample from the Null Set and determining how many (or what
Proportion of) Documents are actually Relevant. A low Elusion value has
commonly been advanced as evidence of an effective search or review
effort (see, e.g., Kleen), but that can be misleading because it quantifies
only those Relevant Documents that have been missed by the search or
review effort; it does not quantify the Relevant Documents found by the
search or review effort (i.e., Recall). Consider, for example, a Document
Population containing one million Documents, of which ten thousand (or
1%) are Relevant. A search or review effort that returned 1,000
Documents, none of which were Relevant, would have 1.001% Elusion,
belying the failure of the search. Elusion = 100% – Negative Predictive
Value.
Email Threading: Grouping together email messages that are part of the
same discourse, so that they may be understood, reviewed, and coded
consistently as a unit.
EORHB: EORHB v. HOA Holdings LLC, Civ. Action No. 7409-VCL, tr.
and slip op. (Del. Ch. Oct. 19, 2012). The first case in which a court sua
sponte directed the parties to use Predictive Coding as a replacement for
Manual Review (or to show cause why this was not an appropriate case for
Predictive Coding), absent either party’s request to employ Predictive
Coding. Vice Chancellor J. Travis Laster also ordered the parties to use the
same E-Discovery vendor and to share a Document repository.
Error / Error Rate: The fraction of all Documents that are incorrectly
coded by a search or review effort. Note that Accuracy + Error = 100%,
and that 100% – Accuracy = Error. While a low Error Rate is commonly
advanced as evidence of an effective search or review effort, its use can be
misleading because it is heavily influenced by Prevalence. Consider, for
example, a Document Population containing one million Documents, of
which ten thousand (or 1%) are relevant. A search or review effort that
found none of the relevant Documents would have 1% Error, belying the
failure of the search or review effort.
ESI: See Electronically Stored Information.
Experimental Design: A standard procedure accepted in the scientific

community for the evaluation of competing hypotheses. There are many
valid experimental designs. Some that can be appropriate for evaluating
Technology-Assisted Review processes include Crossover Trials and
Parallel Trials.
F1: The Harmonic Mean of Recall and Precision, often used in Information
Retrieval studies as a measure of the effectiveness of a search or review
effort, which accounts for the tradeoff between Recall and Precision. In
order to achieve a high F1 score, a search or review effort must achieve both
high Recall and high Precision.
Fallout: See False Positive Rate.
False Negative (FN): A Relevant Document that is missed (i.e., incorrectly

identified as Non-Relevant) by a search or review effort. Also known as a
Miss.
False Negative Rate (FNR): The fraction (or Proportion) of Relevant

Documents that are Missed (i.e., incorrectly identified as Non-Relevant) by
a search or review effort. Note that False Negative Rate + Recall = 100%,
and that 100% – Recall = False Negative Rate.
False Positive (FP): A Non-Relevant Document that is incorrectly

identified as Relevant by a search or review effort.
False Positive Rate (FPR): The fraction (or Proportion) of Non-Relevant

Documents that are incorrectly identified as Relevant by a search or review
effort. Note that False Positive Rate + True Negative Rate = 100%, and
that 100% – True Negative Rate = False Positive Rate. In Information
Retrieval, also known as Fallout.
Feature Engineering: The process of identifying Features of a Document

that are used as input to a Machine Learning Algorithm. Typical Features
include words and phrases, as well as metadata such as subjects, dates, and
file types. One of the simplest and most common Feature Engineering
techniques is Bag of Words. More complex Feature Engineering
techniques include the use of Ontologies and Latent Semantic Indexing.
Features: The units of information used by a Machine Learning Algorithm

to Classify or Prioritize Documents. Typical Features include text
fragments, such as words or phrases, and metadata such as sender, recipient,
and sent date. See also Feature Engineering.
Find Similar: A search method that identifies Documents that are similar to
a particular exemplar. Find Similar is commonly misconstrued to be the
mechanism behind Technology-Assisted Review.
Gain Curve: A graph that shows the Recall that would be achieved for a
particular Cutoff. The Gain Curve directly relates the Recall that can be
achieved to the effort that must be expended to achieve it, as measured by
the number of Documents that must be reviewed and Coded.
Gaussian Calculator / Gaussian Estimation: See Classical, Gaussian, or

Normal Calculator / Classical, Gaussian, or Normal Estimation.
Gaussian Distribution: See Normal Distribution.
Gaussian Estimate: A Statistical Estimate of a Population characteristic

using Gaussian Estimation. It is generally expressed as a Point Estimate
Global Aerospace: Global Aerospace Inc. v. Landow Aviation, Consol.

Case No. CL 61040, 2012 WL 1431215 (Va. Cir. Ct. Apr. 23, 2012). The
first State Court Order approving the use of Predictive Coding by the
producing party, over the objection of the requesting party, without
prejudice to the requesting party raising an issue with the Court as to the
completeness or the contents of the production, or the ongoing use of

Predictive Coding. The order was issued by Loudoun County Circuit Court
Judge James H. Chamblin.
Global Deduplication: Deduplication of Documents across multiple

custodians. Also referred to as Horizontal Deduplication. (Cf. Vertical
Deduplication.)
Gold Standard: The best available determination of the Relevance or Non-

Relevance of all (or a sample) of a Document Population, used as
benchmark to evaluate the effectiveness of a search and review effort. Also
referred to as Ground Truth.
Goodhart’s Law: An observation made in 1975 by Charles Goodhart,

Chief Adviser to the Bank of England, that statistical economic indicators,
when used for regulation, become unreliable. Restated and generalized in
1997 by University of Cambridge Professor Marilyn Strathern as “When a
measure becomes a target, it ceases to be a good measure.” Within the
context of Electronic Discovery, Goodhart’s Law suggests that the value of
Information Retrieval measures such as Recall and Precision may be
compromised if they are prescribed as the definition of the reasonableness
of a search or review effort.
Grossman and Cormack: Authors of the JOLT (a.k.a. Richmond Journal)

Study.
Ground Truth: See Gold Standard.
Harmonic Mean: The reciprocal of the average of the reciprocals of two or

more quantities. If the quantities are named a and b, their Harmonic Mean
2
is 1 1 . In Information Retrieval, F1 is the Harmonic Mean of Recall and
+
𝑎 𝑏
Precision. The Harmonic Mean, unlike the more common arithmetic mean
(i.e., average), falls closer to the lower of the two quantities. As a summary
measure, a Harmonic Mean may be preferable to an arithmetic mean
because a high Harmonic Mean depends on both high Recall and high
Precision, whereas a high arithmetic mean can be achieved with high Recall
at the expense of low Precision, or high Precision at the expense of low
Recall.
Hashing / Hash / Hash Value: A statistical method used to reduce the

contents of a Document to a single, fixed-size, alphanumeric value, which
is, for all intents and purposes, unique to a particular Document; the single,
fixed-size alphanumeric value resulting from Hashing a particular
Document. Common Hashing Algorithms include, but are not limited to,
MD5, SHA-1, and SHA-2. Hashing and Hash Values are typically used for
Document identification, Deduplication, or ensuring that Documents have
not been altered.
Horizontal Deduplication: See Global Deduplication. (Cf. Vertical

Deduplication.)
Index: A list of Keywords in which each Keyword is accompanied by a list

of the Documents (and sometimes the positions within the Documents)
where it occurs. Manual indices have been used in books for centuries;
automatic indices are used in Information Retrieval systems to identify the
Documents that contain particular Search Terms.
Indexing: The manual or automatic process of creating an Index. In

Electronic Discovery, Indexing typically refers to the automatic
construction of an electronic Index for use in an Information Retrieval
system.
Information Need: In Information Retrieval, the information being sought

in a search or review effort. In E-Discovery, the Information Need is
typically to identify Documents responsive to a request for production, or to
identify Documents that are subject to privilege or work-product protection.
Information Retrieval: The science of how to find information to meet an

Information Need. While modern Information Retrieval relies heavily on
computers, the discipline predates the invention of computers.
In Re: Actos: In Re: Actos (Pioglitazone) Products Liability Litigation,

MDL No. 6:11-md-2299 (W.D. La. July 27, 2012). A product liability
action with a Case Management Order (CMO) that memorializes the
parties’ agreement on a “search methodology proof of concept to evaluate
the potential utility of advanced analytics as a Document identification
mechanism for the review and production” of Electronically Stored
Information. The search protocol provides for the use of a Technology-
Assisted Review tool on the email of four key custodians. The CMO was
issued by District Judge Rebecca F. Doherty.
Internal Response Curve: From Signal Detection Theory, a tool for

estimating the number of Relevant and Non-Relevant Documents in a
Population, or the number of Documents that fall above and below a
particular Cutoff. The use of Internal Response Curves for this purpose
assumes that the scores yielded by a Machine Learning Algorithm for

Relevant Documents obey a Gaussian Distribution, and the scores for Non-
Relevant Documents obey a different Gaussian Distribution. These
distributions are then used to predict the number of Relevant and Non-
Relevant Documents in any given range of scores.
Interval Sample / Interval Sampling: See Systematic Sample / Systematic

Sampling.
Issue Code(s) / Issue Coding: One or more subcategories of the overall

Information Need to be identified in a search or review effort; the act of
generating such subcategories of the overall Information Need. Examples
include specification of the reason(s) for a determination of Relevance or
Non-Relevance, Coding of particular subcategories of interest, and Coding
of privileged, confidential, or significant (“hot”) Documents.
Iterative Training: The process of repeatedly augmenting the Training Set

with additional examples of Coded Documents until the effectiveness of the
Machine Learning Algorithm reaches an acceptable level. The additional
examples may be identified through Judgmental Sampling, Random
Sampling, or by the Machine Learning Algorithm, as in Active Learning.
Jaccard Index: A measure of the consistency between two sets (e.g.,

Documents Coded as Relevant by two different reviewers). Defined
mathematically as the size of the intersection of the two sets, divided by the
size of the union (e.g., the number of Documents coded as Relevant by both
reviewers, divided by the number of Documents identified as Relevant by
one or the other, or both reviewers). It is typically used as a measure of
consistency among review efforts, but also may be used as a measure of
similarity between two Documents represented as two Bags of Words.
Jaccard Index is also referred to as Overlap or Mutual F1. Empirical studies
have shown that expert reviewers commonly achieve Jaccard Index scores
of about 50%, and that scores exceeding 60% are rare.
JASIST Study: A 2009 study (Herbert L. Roitblat, Anne Kershaw &

Patrick Oot, Document Categorization in Legal Electronic Discovery:
Computer Classification vs. Manual Review, 61 J. AM. SOC’Y. FOR INFO.
SCI. & TECH. 70 (2010)), showing that the Positive Agreement between
each of two Technology-Assisted Review methods, and a prior production
to the Department of Justice, exceeded the Positive Agreement between
each of two Manual Review processes and the same production. Also
referred to as the EDI Study.
JOLT Study: A 2011 study (Maura R. Grossman & Gordon V. Cormack,

Technology-Assisted Review in E-Discovery Can Be More Effective and
More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH.
11 (2011)), available at http://jolt.richmond.edu/v17i3/article11.pdf, that
used data from TREC 2009 to show that two Technology-Assisted Review
processes (one using Machine Learning and one using a Rule Base)
generally achieved better Recall, better Precision, and greater efficiency
than the TREC Manual Review process. Also known as the Richmond
Journal Study, or the Richmond Study.
Judgmental Sample / Judgmental Sampling: A method in which a

Sample of the Document Population is drawn, based at least in part on
subjective factors, so as to include the “most interesting” Documents by
some criterion; the Sample resulting from such method. Unlike a Random
Sample, the statistical properties of a Judgmental Sample may not be
extrapolated to the entire Population. However, an individual (such as a
quality assurance auditor or an adversary) may use Judgmental Sampling to
attempt to uncover defects. The failure to identify defects may be taken as
evidence (albeit not statistical evidence, and certainly not proof) of the
absence of defects.
Keyword: A word (or Search Term) that is used as part of a Query in a

Keyword Search.
Keyword Expansion: See Query Expansion.
Keyword Search: A search in which all Documents that contain one or

more specific Keywords are returned.
Kleen: Kleen Prods. LLC v. Packaging Corp. of Am., Case No. 1:10-cv-
05711, various Pleadings and Tr. (N.D. Ill. 2012). A federal case in which
plaintiffs sought to compel defendants to use Content-Based Advanced
Analytics (CBAA) for their production, after defendants had already
employed complex Boolean Searches to identify Responsive Documents.
Defendants advanced Elusion scores of 5%, based on a Judgmental Sample
of custodians, to defend the reasonableness of the Boolean Search. After
two days of evidentiary hearings before (and many conferences with)
Magistrate Judge Nan R. Nolan, plaintiffs withdrew their request for
CBAA, without prejudice.
Knowledge Engineering: The process of capturing the expertise of a

Subject Matter Expert in a form (typically a Rule Base) that can be
executed by a computer to emulate the human’s judgment.
Label / Labeled / Labeling: See Code / Coded / Coding.
Latent Semantic Analysis (LSA): See Latent Semantic Indexing.
Latent Semantic Indexing (LSI): A Feature Engineering Algorithm that

uses linear algebra to group together correlated Features. For example,
“Windows, Gates, Ballmer” might be one group, while “Windows, Gates,
Doors” might be another. Latent Semantic Indexing underlies many
Concept Search tools. While Latent Semantic Indexing is used for Feature
Engineering in some Technology-Assisted Review tools, it is not, per se, a
Technology-Assisted Review method. Also referred to as Latent Semantic
Analysis.
Linear Review: A Document-by-Document Manual Review in which the

Documents are examined in a prescribed order, typically chronological
order.
Logistic Regression: A state-of-the-art Supervised Learning Algorithm that

estimates the Probability that a Document is Relevant, based on the
Features it contains.
Machine Learning: The use of a computer Algorithm to organize or

Classify Documents by analyzing their Features. In the context of
Technology-Assisted Review, Supervised Learning Algorithms (e.g.,
Support Vector Machines, Logistic Regression, Nearest Neighbor, and
Bayesian Classifiers) are used to infer Relevance or Non-Relevance of
Documents based on the Coding of Documents in a Training Set. In
Electronic Discovery generally, Unsupervised Learning Algorithms are
used for Clustering, Near-Duplicate Detection, and Concept Search.
Manual Review: The practice of having human reviewers individually read

and Code the Documents in a Collection for Responsiveness, particular
issues, privilege, and/or confidentiality.
Margin of Error: The maximum amount by which a Point Estimate might

likely deviate from the true value, typically expressed as “plus or minus” a
percentage, with a particular Confidence Level. For example, one might
express a Statistical Estimate as “30% of the Documents in the Population
are Relevant, plus or minus 3%, with 95% confidence.” This means that
the Point Estimate is 30%, the Margin of Error is 3%, the Confidence
Interval is 27% to 33%, and the Confidence Level is 95%. Using Gaussian
Estimation, the Margin of Error is one-half of the size of the Confidence
Interval. It is important to note that when the Margin of Error is expressed
as a percentage, it refers to a percentage of the Population, not to a
percentage of the Point Estimate. In the current example, if there are one
million Documents in the Document Population, the Statistical Estimate
may be restated as “300,000 Documents in the Population are Relevant,
plus or minus 30,000 Documents, with 95% confidence”; or, alternatively,
“between 270,000 and 330,000 Documents in the Population are Relevant,
with 95% confidence.” The Margin of Error is commonly misconstrued to
be a percentage of the Point Estimate. However, it would be incorrect to
interpret the Confidence Interval in this example to mean that “300,000
Documents in the Population are Relevant, plus or minus 9,000
Documents.” The fact that a Margin of Error of “plus or minus 3%” has
been achieved is not, by itself, evidence of a precise Statistical Estimate
when the Prevalence of Relevant Documents is low.
Miss / Missed: A Relevant Document that is not identified as Relevant by a

search or review effort. Also referred to as a False Negative.
Miss Rate: The fraction (or proportion) of truly Relevant Documents that
are not identified as Relevant by a search or review effort. Miss Rate =
100% – Recall. Also referred to as the False Negative Rate.
Model: See Statistical Model.
Mutual F1: See Jaccard Index.
Naïve Bayes: A Supervised Learning Algorithm in which the relative

frequency of words (or other Features) in Relevant and Non-Relevant
Training Examples is used to estimate the likelihood that a new Document
containing those words (or other Features) is Relevant. Naïve Bayes relies
on the simplistic assumption that the words in a Document occur with
independent Probabilities, with the consequence that it tends to yield
extremely low or extremely high estimates.
NDLON: National Day Laborer Organizing Network v. U.S. Immigration

and Customs Enforcement Agency, Case No. 10-Civ-3488 (SAS), 2012 WL
2878130 (S.D.N.Y. July 13, 2012), a Freedom of Information Act (FOIA)
case in which District Judge Shira A. Scheindlin held that “most custodians
cannot be ‘trusted’ to run effective searches because designing legally
sufficient electronic searches in the discovery or FOIA contexts is not part
of their daily responsibilities,” and stated (in dicta) that “beyond the use of
keyword search, parties can (and frequently should) rely on latent semantic
indexing, statistical probability models, and machine learning to find
responsive documents. Through iterative learning, these methods (known
as ‘computer-assisted’ or ‘predictive’ coding) allow humans to teach
computers what documents are and are not responsive to a particular FOIA
or discovery request and they can significantly increase the effectiveness

and efficiency of searches.”
Near-Duplicate Detection: An industry-specific term generally used to

describe a method of grouping together “nearly identical” Documents.
Near-Duplicate Detection is a variant of Clustering in which the similarity
among Documents in the same group is very strong. It is typically used to
reduce review costs, and to ensure consistent Coding. Also referred to as
Near-Deduplication.
Near-Deduplication: See Near-Duplicate Detection.
Nearest Neighbor: A Supervised Learning Algorithm in which a new

Document is Classified by finding the most similar Document in the
Training Set, and assuming that the correct Coding for the new Document is
the same as the most similar one in the Training Set.
Negative Predictive Value (NPV): The fraction (Proportion) of

Documents that are identified as Non-Relevant by a search or review effort,
that are in fact Non-Relevant. The complement of Precision; that is,
Negative Predictive Value is computed the same way as Precision when the
definitions of Relevant and Non-Relevant are transposed.
N-Gram: N consecutive words or characters treated as a Feature. In the

phrase, “To be or not to be,” a word Bigram (i.e., 2-gram) would be “to be”;
a word Trigram (i.e., 3-gram) would be “to be or”; a Quad-Gram (i.e., 4-
gram) would be “to be or not”; and so on. See also Shingling.
Non-Relevant / Not Relevant: In Information Retrieval, a Document is

considered Non-Relevant (or Not Relevant) if it does not meet the
Information Need of the search or review effort. The synonym “irrelevant”
is rarely used in Information Retrieval.
Normal Distribution: The “bell curve” of classical statistics. The number

of Relevant Documents in a Sample tends to obey a Normal (Gaussian)
Distribution, provided the Sample size is large enough to capture a
substantial number of Relevant and Non-Relevant Documents. In this
situation, Gaussian Estimation is reasonably accurate. If the Sample size is
insufficiently large to capture a substantial number of both Relevant and
Non-Relevant Documents (as a rule of thumb, at least 12 of each), the
Binomial Distribution better characterizes the number of Relevant
Documents in the Sample, and Binomial Estimation is more appropriate.
Also referred to as a Gaussian Distribution.
Null Set: The set of Documents that are not returned by a search process, or
that are identified as Not Relevant by a review process.
Ontology: A representation of the relationships among words and their

meanings that is richer than a Taxonomy. For example, an Ontology can
represent the fact that a wheel is a part of a bicycle, that gold is yellow, and
so on.
Overlap: See Jaccard Index.
Parallel Trial: An Experimental Design for comparing two search or

review processes using the same Document Collection and Information
Need, in which both processes are applied concurrently but independently,
and then the results of the two efforts are compared. (Cf. Crossover Trial.)
Pattern Matching: The science of designing computer Algorithms to

recognize natural phenomena like parts of speech, faces, or spoken words.
Point Estimate: The most likely value for a Population characteristic.

When combined with a Margin of Error (or Confidence Interval) and a
Confidence Level, it reflects a Statistical Estimate.
Population: See Document Population.
Positive Agreement: The Probability that, if one reviewer Codes a

Document as Relevant, a second independent reviewer will also Code the
Document as Relevant. Empirical studies show that Positive Agreement
rates of 70% are typical, and Positive Agreement rates of 80% are rare.
Positive Agreement should not be confused with Agreement (which is a less
informative measure) or Overlap (which is a numerically smaller measure
that conveys similar information). Under the assumption that the two
reviewers are equally likely to err, Overlap is roughly equal to the square of
Positive Agreement. That is, if Positive Agreement is 70%, Overlap is
roughly 70% × 70% = 49%.
Positive Predictive Value (PPV): See Precision. Positive Predictive Value

is a term used in Signal Detection Theory; Precision is the equivalent term
in Information Retrieval.
Precision: The fraction of Documents identified as Relevant by a search or

review effort, that are in fact Relevant. Also referred to as Positive
Predictive Value.
Precision-Recall Curve: The curve representing the tradeoff between

Precision and Recall for a given search or review effort, depending on the
chosen Cutoff value. See Recall-Precision Curve.
Precision-Recall Tradeoff: The notion that most search strategies can be

adjusted to increase Precision at the expense of Recall, or vice versa. At
one extreme, 100% Recall could be achieved by a search that returned the
entire Document Population, but Precision would be low (equal to
Prevalence). At the other extreme, 100% Precision could be achieved by a
search that returned a single Relevant Document, but Recall would be low
(equal to 1/N, where N is the number of Relevant Documents in the
Document Population). More generally, a broader search returning many
Documents will have higher Recall and lower Precision, while a narrower
search returning fewer Documents will have lower Recall and higher
Precision. A Precision-Recall Curve illustrates the Precision-Recall
Tradeoff for a particular search method.
Predictive Coding: An industry-specific term generally used to describe a

Technology-Assisted Review process involving the use of a Machine
Learning Algorithm to distinguish Relevant from Non-Relevant
Documents, based on Subject Matter Expert(s)’ Coding of a Training Set of
Documents. See Supervised Learning and Active Learning.
Prevalence: The fraction of Documents in a Population that are Relevant to

an Information Need. Also referred to as Richness or Yield.
Prioritization / Prioritized: See Relevance Ranking.
Probabilistic Latent Semantic Analysis: A variant of Latent Semantic

Analysis based on conditional Probability rather than on correlation.
Probability: The fraction (proportion) of times that a particular outcome

would occur, should the same action be repeated under the same conditions
an infinite number of times. For example, if one were to flip a fair coin, the
Probability of it landing “heads” is one-half, or 50%; as one repeats this
action indefinitely, the fraction of times that the coin lands “heads” will
become indistinguishable from 50%. If one were to flip two fair coins, the
Probability of both landing “heads” is one-quarter, or 25%.
Proportion: The fraction of a set of Documents having some particular

property (typically Relevance).
Proportionality: Pursuant to Federal Rules of Civil Procedure 26(b)(2)(B),

26(b)(2)(C), 26(g)(1)(B)(iii), and other federal and state procedural rules,
the legal doctrine that Electronically Stored Information may be withheld

from production if the cost and burden of producing it exceeds its potential
value to the resolution of the matter. Proportionality has been interpreted in
the case law to apply to preservation as well as production.
Quad-Gram: An N-Gram where N = 4 (i.e., a 4-gram).
Quality Assurance: A method to ensure, after the fact, that a search or

review effort has achieved reasonable results.
Quality Control: Ongoing methods to ensure, during a search or review

effort, that reasonable results are being achieved.
Query: A formal search command provided as input to a search tool.
Query Expansion: The process of adding Search Terms to a Query to

improve Recall, often at the expense of decreased Precision.
Random Sample / Random Sampling: A subset of the Document

Population selected by a method that is equally likely to select any
Document from the Document Population for inclusion in the Sample; the
Sample resulting from such action. Random Sampling is the basis of
Statistical Estimation.
RAND Study: A 2012 study (Nicholas M. Pace & Laura Zakaras, Where
the Money Goes: Understanding Litigant Expenditures for Producing
Electronic Discovery, RAND Institute for Civil Justice (2012)), indicating
that Document review accounts for 73% of Electronic Discovery costs, and
concluding that “[t]he exponential growth in digital information, which
shows no signs of slowing, makes a computer-categorized review strategy,
such as predictive coding, not only a cost-effective choice but perhaps the
only reasonable way to handle many large-scale productions.”
Ranking: See Relevance Ranking.
Recall: The fraction of Relevant Documents that are identified as Relevant

by a search or review effort.
Recall-Precision Curve: See Precision-Recall Curve.
Recall-Precision Tradeoff: See Precision-Recall Tradeoff.

Receiver Operating Characteristic Curve (ROC): In Signal Detection

Theory, a graph of the tradeoff between True Positive Rate and False
Positive Rate, as the Cutoff is varied.
Relevance Feedback: An Active Learning process in which the Documents

with the highest likelihood of Relevance are coded by a human, and added
to the Training Set.
Relevance Ranking: A search method in which the results are ranked from
the most likely to the least likely to be Relevant to an Information Need; the
result of such ranking. Google Web Search is an example of Relevance
Ranking.
Relevance / Relevant: In Information Retrieval, a Document is considered

Relevant if it meets the Information Need of the search or review effort.
Responsiveness: A Document that is Relevant to an Information Need

expressed by a particular request for production or subpoena in a civil,
criminal, or regulatory matter.
Richness: See Prevalence or Yield.
Richmond Journal Study / Richmond Study: See JOLT Study.
Roitblat, Kershaw, and Oot: Authors of the JASIST (a.k.a. EDI) Study.
Rolling Collection / Rolling Ingestion: A process in which the Document

Collection is periodically augmented as new, potentially Relevant
Documents are identified and gathered. Whenever the Document
Collection is augmented, the results of prior search or review efforts must
be supplemented to account for the new Documents.
Rolling Production: A process in which Responsive Documents are

delivered incrementally to a requesting party to provide timely, partial
satisfaction of a Document request.
Rule: A formal statement of one or more criteria used to determine a

particular outcome, e.g., whether to code a Document as Relevant or Non-
Relevant.
Rule Base: A set of Rules created by an expert to emulate the human

decision-making process for the purposes of Classifying Documents in the
context of Electronic Discovery.
Sample / Sampling: A subset of the Document Population used to assess

some characteristic of the Population; the act of generating such a subset of
the Document Population. See Interval Sample, Judgmental Sample,
Random Sample, Statistical Sample, or Systematic Sample.
Sample Size: The number of Documents drawn at random that are used to
calculate a Statistical Estimate.
Search Term: See Keyword.
Sedona / Sedona Conference: The Sedona Conference® (https://thesedona

conference.org) is a nonprofit, 501(c)(3) research and educational institute,
founded in 1997 by Richard G. Braman, dedicated to the advanced study of
law and policy in the areas of antitrust, complex litigation, and intellectual
property rights. Sedona sponsors a preeminent think-tank in the area of
Electronic Discovery known as Working Group 1 on Electronic Document
Retention and Production. Sedona is well known for its thoughtful,
balanced, and free publications, such as The Sedona Conference®
Glossary: E-Discovery & Digital Information Management (Third Edition,
Sept. 2010), The Sedona Principles Addressing Electronic Document
Production, Second Edition (June 2007), and The Sedona Conference®
Cooperation Proclamation (July 2008).
Seed Set: The initial Training Set provided to the learning Algorithm in an
Active Learning process. The Documents in the Seed Set may be selected
based on Random Sampling or Judgmental Sampling. Some commentators
use the term more restrictively to refer only to Documents chosen using
Judgmental Sampling. Other commentators use the term generally to mean
any Training Set, including the final Training Set in Iterative Training, or
the only Training Set in non-Iterative Training.
Sensitivity: See True Positive Rate.
Shingling: A Feature Engineering method in which the Features consist of

all N-Grams in a text, for some number N. For example, the Trigram
Shingling of the text “To be or not to be” consists of the features “to be or”;
“be or not”; “or not to”; and “not to be.” Note that the Features overlap one
another in the text, suggesting the metaphor of roof shingles.
Signal Detection Theory: Invented at the same time and in conjunction

with radar, the science of distinguishing true observations from spurious
ones. Signal Detection Theory is widely used in radio engineering and
medical diagnostic testing. The terms True Positive, True Negative, False
Positive, False Negative, Sensitivity, Specificity, Receiver Operating
Characteristic Curve, Area Under the ROC Curve, and Internal Response
Curve, all arise from Signal Detection Theory.
Significance / Significant: The confirmation, with a given Confidence

Level, of a prior hypothesis, using a Statistical Estimate. The result is said
to be Statistically Significant if all values within the Confidence Interval for
the desired Confidence Level (typically 95%) are consistent with the
hypothesis being true, and inconsistent with it being false. For example, if
the hypothesis is that fewer than 300,000 Documents are Relevant, and a
Statistical Estimate shows that 290,000 Documents are Relevant, plus or
minus 5,000 Documents, we say that the result is Significant. On the other
hand, if the Statistical Estimate shows that 290,000 Documents are
Relevant, plus or minus 15,000 Documents, we say that the result is not
Significant, because the Confidence Interval includes values (i.e., the values
between 300,000 and 305,000) that contradict the hypothesis.
Specificity: See True Negative Rate.
Statistical Estimate: A quantitative estimate of a Population characteristic

using Statistical Estimation. It is generally expressed as a Point Estimate
Statistical Estimation: The act of estimating the Proportion of a Document

Population that has a particular characteristic, based on the Proportion of a
Random Sample that has the same characteristic. Methods of Statistical
Estimation include Binomial Estimation and Gaussian Estimation.
Statistically Significant / Statistical Significance: See Significance.
Statistical Model: A mathematical abstraction of the Document Population

that removes irrelevant characteristics while largely preserving those of
interest for a particular purpose. For the purpose of computing Recall, a
Statistical Model need only consider whether or not the Documents are
Relevant, and whether or not the Documents are Coded Relevant, not any
other characteristics of the Documents.
Statistical Sample / Statistical Sampling: A method in which a Sample of

the Document Population is drawn at random, so that statistical properties
of the Sample may be extrapolated to the entire Document Population; the
Sample resulting from such action.
Stemming: In Keyword or Boolean Search, or Feature Engineering, the

process of equating all forms of the same root word. For example, the
words “stem,” “stemming,” “stemmed,” and “stemmable” would all be

treated as equivalent, and would each yield the same result when used as
Search Terms in a Query. In some search systems, stemming is implicit,
and in others, it must be made explicit through particular Query syntax.
Stop Word: A common word that is eliminated from Indexing.

Eliminating Stop Words from Indexing dramatically reduces the size of the
Index, while only marginally affecting the search process in most
circumstances. Examples of Stop Words include “a,” “the,” “of,” “but,”
and “not.” Because phrases and names such as “To be or not to be,” and
“The Who,” contain exclusively Stop Words that would not be Indexed,
they would not be identified (or identifiable) through a Keyword Search.
Subject Matter Expert(s): One or more individuals (typically, but not

necessarily, attorneys) who are familiar with the Information Need and can
render an authoritative determination as to whether a Document is Relevant
or not.
Supervised Learning: A Machine Learning method in which the learning

Algorithm infers how to distinguish between Relevant and Non-Relevant
Documents using a Training Set. Supervised Learning can be a stand-alone
process, or used repeatedly in an Active Learning process.
Support Vector Machine: A state-of-the-art Supervised Learning

Algorithm that separates Relevant from Non-Relevant Documents using
geometric methods (i.e., geometry). Each Document is considered to be a
point in [hyper]space, whose coordinates are determined from the Features
contained in the Document. The Support Vector Machine finds a
[hyper]plane that best separates Relevant from Non-Relevant Training
Examples. Documents outside the Training Set (i.e., uncoded Documents
from the Document Collection) are then Classified as Relevant or not,
depending on which side of the [hyper]plane they fall on. Although a
Support Vector Machine does not calculate a Probability of Relevance, one
may infer that the Classification of Documents closer to the [hyper]plane is
less certain than for those that are far from the [hyper]plane.
Synthetic Document: An industry-specific term generally used to describe

an artificial Document created by either the requesting party or the
producing party, as part of a Technology-Assisted Review process, for use
as a Training Example for a Machine Learning Algorithm. Synthetic
Documents are contrived Documents in which one party imagines what the
evidence might look like and relies on the Machine Learning Algorithm to
find actual Documents that are similar to the artificial Document.
Systematic Sample / Systematic Sampling: A Sampling method in which

every Nth Document (for some fixed number N) is selected, when the
Documents are considered in some prescribed order; the Sample resulting
from such action. A Systematic Sample is random (and hence a true
Statistical Sample) only when the prescribed order is itself random.
Sometimes referred to as an Interval Sample / Interval Sampling.
TAR: See Technology-Assisted Review.
Taxonomy: A hierarchical organizational scheme that arranges the

meanings of words into classes and subclasses. For example, vehicles,
aircraft, and ships are modes of transportation; cars, trucks, and bicycles are
vehicles, and Fords and Chryslers are cars.
Technology-Assisted Review (TAR): A process for Prioritizing or Coding

a Collection of Documents using a computerized system that harnesses
human judgments of one or more Subject Matter Expert(s) on a smaller set
of Documents and then extrapolates those judgments to the remaining
Document Collection. Some TAR methods use Machine Learning
Algorithms to distinguish Relevant from Non-Relevant Documents, based
on Training Examples Coded as Relevant or Non-Relevant by the Subject
Matter Experts(s), while other TAR methods derive systematic Rules that
emulate the expert(s)’ decision-making process. TAR processes generally
incorporate Statistical Models and/or Sampling techniques to guide the
process and to measure overall system effectiveness.
Term Expansion: See Query Expansion.
Term Frequency and Inverse Document Frequency (TF-IDF): An

enhancement to the Bag of Words method in which each word has a weight
based on Term Frequency – the number of times the word appears in the
Document – and Inverse Document Frequency – the reciprocal of the
number of Documents in which the word occurs.
Thesaurus Expansion: In Keyword or Boolean Search, replacing a single

Search Term by a list of its synonyms, as listed in a thesaurus.
Threshold: See Cutoff.
Training Example: One Document from a Training Set.
Training Set: A Sample of Documents coded by one or more Subject

Matter Expert(s) as Relevant or Non-Relevant, from which a Machine
Learning Algorithm then infers how to distinguish between Relevant and

Non-Relevant Documents beyond those in the Training Set.
TREC: The Text REtrieval Conference, sponsored by the National Institute

of Standards and Technology (NIST), which has run since 1992, to “support
research within the information retrieval community by providing the
infrastructure necessary for large-scale evaluation of text retrieval
methodologies. In particular, the TREC workshop series has the following
goals: to encourage research in information retrieval based on large test
Collections; to increase communication among industry, academia, and
government by creating an open forum for the exchange of research ideas;
to speed the transfer of technology from research labs into commercial
products by demonstrating substantial improvements in retrieval
methodologies on real-world problems; and to increase the availability of
appropriate evaluation techniques for use by industry and academia,
including development of new evaluation techniques more applicable to
current systems.”
TREC Legal Track: From 2006 through 2011, TREC included a Legal
Track, which sought “to assess the ability of information retrieval
techniques to meet the needs of the legal profession for tools and methods
capable of helping with the retrieval of electronic business records,
principally for use as evidence in civil litigation.”
Trigram: An N-Gram where N = 3 (i.e., a 3-gram).
True Negative (TN): A Non-Relevant Document that is correctly identified

as Non-Relevant by a search or review effort.
True Negative Rate (TNR): The fraction (or Proportion) of Non-Relevant

Documents that are correctly identified as Non-Relevant by a search or
review effort.
True Positive (TP): A Relevant Document that is correctly identified as

Relevant by a search or review effort.
True Positive Rate (TPR): The fraction (or Proportion) of Relevant

Documents that are correctly identified as Relevant by a search or review
effort. True Positive Rate is a term used in Signal Detection Theory; Recall
is the equivalent term in Information Retrieval.
Uncertainty Sampling: An Active Learning approach in which the

Machine Learning Algorithm selects the Documents as to which it is least
certain about Relevance, for Coding by the Subject Matter Expert(s), and
addition to the Training Set.
Unsupervised Learning: A Machine Learning method in which the

learning Algorithm infers categories of similar Documents without any
training by a Subject Matter Expert. Examples of Unsupervised Learning
methods include Clustering and Near-Duplicate Detection.
Validation: The act of confirming that a process has achieved its intended
purpose. Validation may involve Statistical or Judgmental Sampling.
Vertical Deduplication: Deduplication within a custodian; identical copies

of a Document held by different custodians are not Deduplicated. (Cf.
Horizontal Deduplication.)
Yield: See Prevalence or Richness.

Technology Assisted Review
Chief Judge Brown joins the JAMS panel after a distinguished judicial and legal career. Most recently, he served
twenty-six years as United States District Judge for the District of New Jersey, the last six as Chief Judge, where he
led the court-wide effort to provide prompt, efficient justice to civil litigants, and to implement new local patent rules.
During this period, he was designated by the Judicial Panel on Multi-District Litigation to manage countless MDL
cases, which resulted in numerous settlements. Prior to appointment to the federal judiciary, he served as Chief
Counsel and Acting Administrator of the U.S. Maritime Administration and as General Counsel of the U.S.
Government Printing Office. He was previously in private practice, where he concentrated on federal commercial
litigation, including antitrust, employment, construction, environmental, banking, intellectual property, and securities
matters.
Judge Brown is highly regarded by counsel for his exceptional ability to expeditiously handle and resolve complex
litigation. He is perceived by the parties as being extremely well prepared, knowledgeable, and experienced.
ADR Experience and Qualifications

 During his 26 years on the federal bench, Chief Judge Brown has presided over and resolved by settlement,
trial, or judgment some 9,000 cases involving a wide range of issues, including:
 Admiralty
 Antitrust
 Banking
 Bankruptcy
 Civil rights
 Class actions
 Complex commercial
 Construction
 Employment/Labor
 Environmental
 Government agencies
 Healthcare
 Insurance
 Intellectual property: patents, trademarks, copyrights, unfair competition, and licensing
 Pharmaceuticals and medical devices/mass torts
 Securities
Representative Matters
During his service as Chief Judge, in addition to leading a large metropolitan court of 38 judicial officers, Judge Brown
maintained a full civil caseload, assisting other judges in complex cases, trying about 11 cases a year and settling
many, many more. He was also designated by the Judicial Panel on Multi-District Litigation to manage numerous
MDL cases during this period, which resulted in numerous settlements. Since many of the most challenging matters
Judge Brown has handled and resolved are subject to confidentiality orders, the names and details of representative
matters are not set forth.
 Admiralty
 Adjudicated numerous admiralty/maritime cases involving: (1) Jones Act-based personal injury claims,
including complex toxic torts actions; (2) maritime collision and Limitation of Liability Act issues; (3)
charter, cargo, and marine insurance
 Antitrust
 Presided over a consolidated matter involving a putative class of title insurance purchasers that alleged
injuries as a result of a price-fixing conspiracy by insurance providers
 Presided over matters involving claims of monopolization of software and services in violation of federal
antitrust laws, in several industries, including sports and telecommunications
 Bankruptcy
 Approved the sale of former automobile executive’s 434-acre property to a golf course developer, for
$15.25 million and resolved a contentious, multi-district five-year legal battle. The proceeds from the sale
were earmarked for creditors, who thereby received about half of the total outstanding debt.
 Actively managed and approved the final Plan and Settlement in the Chapter 11 bankruptcy case, a
roofing materials manufacturer, the successor-in-interest to an entity named in 5 million asbestos actions.
Under the Joint Plan of Reorganization, a settlement trust was created to process, liquidate, and pay valid
asbestos personal injury claims as well as address asbestos contaminated sites.
 Actively managed and ultimately approved the plan of reorganization resolving almost 140 wrongful death
and personal injury lawsuits in case involving 390 ephedra-related personal injury and/or wrongful death
claims, two consumer class actions, four enforcement actions by governmental agencies, and multiple
adversary proceedings related to the Chapter 11 case of the seller of the popular ephedra-containing diet
pill and other supplements. Ephedra was linked to health risks and banned by the U.S. Food and Drug
Administration in April 2004. The herbal stimulant had been associated with heart palpitations, strokes,
seizures, high blood pressure, and heart attacks.
 Class Actions
 ERISA: Presided over a class action dispute brought by participants in a Long Term Savings Plan for
Management Employees for compensation for losses related to breach of ERISA-mandated duties in the
operation and administration of the Plan and granted preliminary and final approval of the $29 million
settlement
 Securities Fraud
 Presided over a trial in a securities fraud class action, which resulted in settlement prior to verdict at
Judge Brown’s recommendation; settlement upheld on appeal to the Third Circuit
 Presided over and approved the final settlement of the class action alleging violations of the
Securities Act against former officers and directors of a now defunct generic drug manufacturer
 Employment/Labor
 Presided over a number of trials involving claims of employment discrimination and settled many more
 Intellectual Property
 Patent Litigation
 Presided over a month-long jury trial, a case involving patent infringement claims related to high
speed DSL modem technology. The jury found that all claims of the patents in the suit were valid and
infringed, and awarded $112 million in economic damages.
 Presided over a month-long jury trial, a case involving patent infringement claims relating to
semiconductor technology. The jury found noninfringement and rendered two of the patents invalid.
 Presided over a jury trial, a case involving patent infringement claims relating to communications and
control within defendant’s mailing machines. The jury found that plaintiff infringed the patents, but
found that those patents were invalid on the grounds of anticipation. Judge Brown upheld the jury
verdict in a post-trial memorandum, which was summarily affirmed by the Federal Circuit.
 Presided over a two-week jury trial, a case involving a nursing garment with a built-in nursing bra, in
which the jury found three patents invalid, declined to find infringement by one defendant and found
one patent unenforceable due to inequitable conduct. Judge Brown sustained the jury's verdict on all
counts, denying motions for judgment as a matter of law and awarded costs to defendants.
 Presided over a bench trial, a patent infringement case involving the pharmaceutical tizanidine. After
reviewing post-trial briefing, Judge Brown issued a bench opinion, finding noninfringement in favor of
defendant’s proposed marketing of its generic product and ordered plaintiff’s patent invalid because it
lacked enablement.
 Presided over a one-week jury trial of a patent infringement case regarding interference suppression
technology for ethernet jacks. After the jury was unable to reach a verdict, Judge Brown was able to
assist the parties to reach a settlement.
 Presided over all aspects of litigation, a consolidated patent case involving Sanofi's patents relating to
the drug Allegra, and assisted in settlements
 Presided over a patent infringement case involving a slow-release morphine technology. After trying
the case, the parties amicably settled the action.
 Presided over a two-week jury trial in a patent infringement case involving metalized plastic cutlery.
Judge Brown ordered a new trial. Thereafter, the parties settled the case on amicable terms.
 Presided over a jury trial, a case involving patent infringement claims regarding technology related to
methods of use for a drug where the jury found that all asserted claims of the patent were invalid and
not infringed
 Trademark
 Presided over a trial disallowing the use of a toy manufacturer’s false and misleading advertising but
upholding its right to continue manufacturing the product at issue without any changes in design
 Presided over a case involving trademark, trade dress, and unfair competition between competing
fast food restaurants
 Presided over many Lanham Act cases involving healthcare and household and consumer products
 Multi-District Litigation (MDL)
 Presided over MDL No. 1443 – In re: Insurance Brokerage Antitrust Litigation, involving allegations that
certain insurance brokers solicited rigged bids from insurers and received kickbacks for steering
customers to those companies. One group of insurers settled for $121 million and a group of brokers
settled for $28 million. Judge Brown approved both class action settlements and awarded $29.5 million in
attorney fees for the larger settlement. The Third Circuit Court of Appeals affirmed the settlement in its
entirety.
 Presided over MDL 1471– In Re: Compensation of Managerial, Professional and Technical Employees
Antitrust Litigation, an antitrust matter involving claims that major U.S. oil companies exchanged detailed
salary information in an effort to suppress the growth of oil industry salary levels. Judge Brown ultimately
granted defendants’ motion for summary judgment on the basis that plaintiffs failed to present evidence to
sufficiently define the product market.
 Presided over the ongoing MDL No. 2243 – In re: Fosamax (Alendronate Sodium) Products Liability
Litigation (No. II), which involved hundreds of cases against Merck & Co., Inc. alleging products liability
claims for femur fracture injuries as a result of taking the osteoporosis drug commonly at issue in that
MDL
 Presided over the ongoing MDL No. 1687 – In re Ford Motor Co. E-350 Van Products Liability Litigation
(No. II), a products liability, warranty, and consumer fraud action involving claims that design defects led
to Ford’s 15-passenger vans’ stability problems and a poor reputation that decreased their actual value to
customers. Through motion practice, Judge Brown resolved more than 60 of the original 80 claims initially
brought by 20 named plaintiffs from 11 states, and he also denied the plaintiffs’ initial motion for class
certification.
 International Legal Experience
 Judge Brown has taught International Admiralty and Maritime Law courses as an Adjunct Professor at
Seton Hall University School of Law programs in Egypt (2006 and 2009) and Tanzania (2010 and 2011).
He has hosted judges from Japan, Russia, and the Ukraine as part of judicial exchange programs, and
represented the U.S. Maritime Administration in diplomatic negotiations with Iceland, the USSR, and
Peoples Republic of China and on various NATO matters. He has lectured abroad for the U.S.
Department of State and U.S. Information Agency on various aspects of American law in Japan, the
Philippines, Indonesia, Malaysia, Singapore, Australia, and New Zealand. He represented various
international clients while in private practice and while on the bench, he has addressed various
international issues including the Foreign Corrupt Practices Act, the Foreign Sovereignty Immunity Act,
and various Hague Conventions and treaties.
Honors, Memberships, and Professional Activities

 Recipient, William J. Brennan, Jr. Award for distinguished service, Association of the Federal Bar of New
Jersey
 Recipient, distinguished service awards from the Attorney General and from the Public Printer of the United
States
 Received designation of Honorary Rear Admiral, U.S. Maritime Service
 Founding member and active participant, John C. Lifland Intellectual Property Inn of Court
 Fellow, American Bar Foundation
 Member, Third Circuit Judicial Council, 2005-2012
 Judicial Conference of the United States: Third Circuit District Court Member, 2005-2008; committee
member, 1997-2004
 Board of Visitors member, Duke University School of Law, 2010-present
 Member, District Judge Education Advisory Committee, Federal Judicial Center, 1999-2004 (lecturer at
various CLE programs)
 Continuing Legal Education Lecturer: In recent years, Chief Judge Brown has taught over 20 CLE courses to
practicing attorneys around the country on such subjects as patent law and litigation, class action, securities,
commercial finance and complex litigation, drug and medical device litigation, and negotiation and alternative
dispute resolution. The organizations he has lectured before include PLI, ALI-ABA, NYIPLA, ACI, IQPC, West
Legal Works, ICLE, NJSBA, and AFBNJ
 Chief Justice Weintraub Lecturer, Rutgers University School of Law, Newark, NJ
 Adjunct Professor, Seton Hall School of Law, Newark, NJ, 1995-2012 (taught over 20 courses on federal
courts and jurisdiction, admiralty and maritime law, professional responsibility, civil trial persuasion and
advocacy, legal negotiation, and dispute resolution)
Background and Education

 Judge, United States District Court, District of New Jersey, 1986-2012 (Chief Judge, 2005-2012)
 Chief Counsel (1983-1985) and Acting Administrator (1985), U.S. Maritime Administration
 General Counsel, U.S. Government Printing Office, 1981-1983
 Private Practice, Newark, NJ, 1973-1981
 Assistant U.S. Attorney, Deputy Chief, Criminal Division, and Executive Assistant, District of NJ, 1969-1973
 Law Clerk, Hon. Vincent S. Haneman, Justice, Supreme Court of New Jersey
 J.D., Duke University School of Law, 1968
 B.A., Lafayette College, 1965
 Member of the New Jersey, New York, and District of Columbia bars
Honorable Tonianne J. Bongiovanni, USMJ has been a United States Magistrate Judge in
Trenton, New Jersey, since 2003. Prior to her appointment to the bench she was First Assistant
Public Defender in the Federal Public Defender’s Office in Newark.
Judge Bongiovanni has been an adjunct Professor at Seton Hall University School of Law since
1998 and in 2002-03 was recognized as “Adjunct Professor of the Year.” She was awarded the
Alumnus of the Year Award in 2000 by the Peter W. Rodino Jr. Law Society and in 2008 was
the recipient of the Chief Justice Richard J. Hughes Award from the law school’s public interest
organizations. Judge Bongiovanni is the recipient of several other awards, including the
Honorable Lawrence A. Whipple Memorial Award presented by the Association of Criminal
Defense Lawyers and the Woman of Achievement Award from the Woman Lawyers of
Monmouth County. She has been affiliated with the John C. Lifland American Inn of Court since
2003 and has been a panelist at conferences sponsored by the American Conference Institute.
Judge Bongiovanni received her B.S., magna cum laude, from Jersey City State College and
her J.D. from Seton Hall University School of Law.
JENNIFER MARINO THIBODAUX
Jennifer Marino Thibodaux devotes a majority of her practice to

complex commercial litigation matters. She has defended class
actions involving international wartime disputes and claims
under the Fair and Accurate Credit Transactions Act. She has
handled professional malpractice actions, shareholder disputes,
fraud actions, patent royalty disputes, and matters arising under
Associate
federal statute, such as ERISA and the Fair Credit Reporting
Business & Commercial
Act. In addition, Ms. Thibodaux has represented clients in a wide
Litigation
variety of white-collar criminal matters. Her role in these matters
has included analyzing the United States Sentencing Guidelines,
Practice Areas
drafting sentencing memoranda, managing documentary
Business & Commercial
responses to grand jury subpoenas, preparing witnesses for
Litigation
grand jury testimony, and conducting internal corporate
Class Action Defense
investigations.
Electronic Discovery &
Information Management
Education
Counseling
Seton Hall University School of Law (J.D., magna cum laude,
Contact Information
2005)
One Gateway Center
Order of the Coif
Newark, NJ 07102-5310
Member, Interscholastic Moot Court Board
Semi-finalist and Best Oral Advocate, Eugene Gressman Moot
Direct: 973-596-4839
Court Competition
Fax: 973-639-6254
Member, Journal of Sports and Entertainment Law
JMarino@gibbonslaw.com
Bucknell University (B.A., cum laude, 2002)
President's Award, Excellence in Spanish
Member, Phi Sigma Alpha Honor Society
Member, Alpha Lambda Delta Honor Society
Member, Phi Eta Sigma Honor Society
Professional Admissions
State of New Jersey, 2005
State of New York, 2006
United States District Court for the District of New Jersey, 2005
United States District Court for the Eastern District of New York,
2007
United States District Court for the Southern District of New

York, 2007
United States District Court for the Eastern District of Wisconsin,

2009
Newark New York Trenton Philadelphia Wilmington

Professional Activities
Member, New Jersey State Bar Association
Member, New York State Bar Association
Member, National Association of Women Lawyers
Executive Committee Member of the Federal Practice

Committee, Essex County Bar Association
Barrister, Lifland Inn of Court
Member, New Jersey Chapter of Women in E-Discovery
Executive Committee Member, Young Lawyers' Division, New

Jersey State Bar Association, 2009-2010
Publications
"Ford Can’t Halt All Claims in Alleged Defective Fuel Tank

Putative Class Action," Business Litigation Alert, February 14,
2013
"Third Circuit Deflates Run-Flat Tire Class Action Against BMW

and Bridgestone," Business Litigation Alert, September 21, 2012
"Changing the "Games": The First Social Media Olympics," E-

Discovery Law Alert, August 28, 2012
"Dancer's Facebook Messages With Opt-In Class Members are

Protected Work Product," E-Discovery Law Alert, August 3, 2012
"Ex-Juror Who "Friended" Defendant Faces Jail for Bragging on

Facebook About Dismissal From Jury Duty," E-Discovery Law
Alert, March 22, 2012
"Third Circuit, En Banc, Approves Settlement Class Containing

Members Who Lack “Viable Claim”," Business Litigation Alert,
February 14, 2012
"Not So Fast: 95 Million Reasons to Carefully Select and Limit

Search Terms," E-Discovery Law Alert, January 24, 2012
"Ineffective Privilege Review Leads to Inadvertent Waiver in

Rolling Document Production," E-Discovery Law Alert,
December 9, 2011
"Fee Awards Under New Jersey Consumer Fraud Act Permitted

Despite Failure to Show Ascertainable Loss at Trial," Business
Litigation Alert, November 8, 2011
"So You Want to Be "Friends?" Why Attorneys Should Think

Twice About "Friending" Represented Parties or Witnesses on
Facebook," E-Discovery Law Alert, August 25, 2011
"Motion for Sanctions Denied Due to DuPont's Reasonable,

Professional Efforts to Implement and Update Litigation Hold
Notices," E-Discovery Law Alert, July 5, 2011
"The Death Knell for Consumer Class Actions?

The Supreme Court Rules That Class Waivers in Consumer
Arbitration Agreements are Enforceable and May Not Be
Invalidated on the Grounds of State Law Unconscionability,"
Business & Commercial Litigation Newsletter, June 21, 2011
(Damian V. Santomauro, Jennifer Marino Thibodaux)
"Davis v. Grant Park Holds That Sanctions Motions for Breach of

Duty to Preserve Electronic Communications are Premature
Until the Close of Discovery," E-Discovery Law Alert, February
22, 2011
"N.J. Judge Dismisses Potential Juror for Attempting to "Friend"

Defendant on Facebook," E-Discovery Law Alert, December 6,
2010
"Create Your Own Arbitration Provision: Two Recent Supreme

Court Decisions Emphasize That Parties Have the Freedom to
Define the Nature and Scope of Their Agreement to Arbitrate,"
The National Law Review, November 29, 2010 (Damian V.
Santomauro, Jennifer Marino Thibodaux)
"Evidence Preservation Lessons in US V. Suarez," Law360,

November 3, 2010 (Jennifer Marino Thibodaux, Mara E. Zazzali-
Hogan)
"Federal Judge Rules Government Failed to Preserve Text

Messages and Orders Adverse Inference Instruction in Criminal
Case," E-Discovery Law Alert, November 2, 2010
"Cyberspace “Friend” or Foe?: Ethics Jurisprudence From

Various Jurisdictions Affects Whether Lawyers Should “Friend”
Witnesses Online," E-Discovery Alert, October 8, 2009
"Friend or Foe: Ethical Issues for Lawyers to Consider When

'Friending' Witnesses Online," New Jersey Law Journal, August
31, 2009 (Jennifer Marino Thibodaux, Mara E. Zazzali-Hogan)
"New identity theft laws taking effect Nov. 1," New Jersey

Lawyer, October 20, 2008 (Frederick W. Alworth, Jennifer
Marino Thibodaux)
"More 'Filthy Words' But No 'Free Passes' for the 'Cost of Doing
Business': New Legislation is the Best Regulation for Broadcast
Indecency," 15 Seton Hall J. Sports & Ent. L. 135, 2005
Speaking Engagements
Speaker, Gibbons P.C. and Women in E-Discovery, "A Social

Media Discussion Among 'Friends,'" Newark, NJ. March 19,
2013
Panelist, Gibbons P.C., "E-Discovery: What's New in NJ, NY,

and PA?," Newark, NJ. January 9, 2013
Speaker, Gibbons E-Discovery Task Force , "Sixth Annual E-

Discovery Conference, " Whippany, NJ. December 13, 2012
Panelist, Gibbons Academy, "A Look at Issues Affecting

Litigators Today " Newark, NJ. September 20, 2012
Panelist, Gibbons Academy, "Is the Law Anti-Social?," Newark,

NJ. May 9, 2012
Panelist, Gibbons E-Discovery Task Force & Women in E-

Discovery, "The Internet and Social Media in the Courtroom,"
Newark, NJ. June 21, 2011
Speaker, Gibbons' Third Annual E-Discovery Conference,

"Tweeting, Poking, and Linking: Risks and Opportunities Social
Media Present to Your Company," Newark, NJ. October 29,
2009
Honors/Awards*
Listed as Super Lawyers Rising Star, Business Litigation, 2010-

2013
Article written for the New Jersey Law Journal on "Ethical Issues
to Consider When 'Friending' Witnesses Online" named one of
the 20 Must-Read Articles on Social Media Ethics for Lawyers
and Judges in The Legal Intelligencer, November 2, 2009
Listed as a "Future Star" by Benchmark Litigation
Judicial Clerkship(s)
Law clerk to the Honorable Edwin R. Alley, New Jersey Superior

Court, Appellate Division

Representative Cases
Center for Special Procedures v. Connecticut General Life Ins.

Co., et al., 2010 U.S. Dist. LEXIS 128289 (D.N.J. Dec. 6, 2010)
Brunswick Surgical Center, LLC v. CIGNA Healthcare, et al.,

2010 U.S. Dist. LEXIS 85043 (D.N.J. Aug. 18, 2010)
Shelton v. Restaurant.com Inc., 2010 U.S. Dist. LEXIS 59111

(D.N.J. June 15, 2010)
Languages
Literate in Spanish
*No aspect of this communication has been approved by the

Supreme Court of New Jersey. Further information about
methodologies for rating or selecting attorneys is available on
our website’s Award Methodology page.

Matthew F. Knouff is General Counsel and eDiscovery Counsel with Complete Discovery
Source, Inc., an award-winning, global electronic discovery services provider and technology
developer. Matthew advises law firms and corporations worldwide on eDiscovery and
information governance policies and processes, cost & risk reduction strategies, and defensible
deployment of technology during legal proceedings. He has led numerous successful discovery
efforts in 26 countries across five continents. He is an active member of The Sedona Conference
Working Groups 1 & 6 and the EDRM’s IGRM Section, Chairman of the New York County
Lawyers’ Association’s eDiscovery Sub-Committee, and an avid participant with the New York
State Bar Association and several other professional organizations. Matthew has developed
numerous CLE programs and is a frequent lecturer, blogger, writer, and national speaker on
issues related to eDiscovery, cross-border litigation, and information governance. In 2012, he
was published in both the North Carolina School of Law Journal of Law and Technology and the
Richmond Journal of Law and Technology on issues related to cross-border eDiscovery, and was
a panelist at the Advanced eDiscovery Institute at Georgetown Law. He is a graduate of the
University of North Carolina at Chapel Hill and the University of North Carolina School of Law.
732.547.4918
Bio: www.CyberJurist.com
Email: Fernando@CyberJurist.com
fernando.pinguelo
Fernando M. Pinguelo, a Partner and Chair of the law firm's Cyber Security & Data Protection group,
is a trial lawyer who devotes his practice to complex business disputes with an emphasis on cyber,
media, intellectual property, and employment law matters in U.S. Federal and state courts. He is the
author of numerous publications on how technology impacts lawsuits, including New Jersey Federal
Civil Procedure (Chapter 13) (New Jersey Law Journal); and lectures internationally on those topics,
including for the Judicial College. He created the ABA Journal award-winning eDiscovery blog
eLessons Learned, eLLblog.com, and the cybersecurity blog eWhiteHouse Watch, eWHWblog.com.
Mr. Pinguelo is experienced in all facets of litigation (trial, mediation, arbitration, and appellate) in
both federal and state courts. He handles a broad spectrum of disputes including business torts, data
breaches, misappropriation of trade secrets, copyright infringement, fraud, breach of non-compete
covenants, and discrimination. Representative matters include:
º Led efforts in the U.S. in conjunction with Brazilian legal efforts concerning a significant data
security breach; filed federal action that compelled website to produce user information.
º Led and coordinated efforts to contain company's data security breach and address data
security obligations under relevant federal and state laws.
º Assessed client's lost data event, determined state data breach notification requirements, and
implemented action plan.
º Successfully addressed breach of security issues for client under state’s Identity Theft
Prevention Act.
Mr. Pinguelo is a member of the New Jersey Supreme Court Committee on the Rules of Evidence. He
is also an Adjunct Professor of Law at Seton Hall Law School and received a Fulbright Specialist
appointment for his work in electronic discovery. Mr. Pinguelo clerked for The Honorable Edwin H.
Stern, Chief Judge of the New Jersey Appeals Court (ret.). He sits on the Boards of The Masters
Conference and Organization of Legal Professionals.
Mr. Pinguelo has received Martindale-Hubbell's AV Preeminent rating since 2011, and has been
recognized by business publications that identify professionals who have made an impact in helping
clients, including The New York Enterprise Report who awarded him its Industry Practice-Technology
Lawyer of the Year award, and Law Technology News (LTN) who recognized him in its October 2012
magazine issue as one of only a few U.S.-based lawyers with notable experience in handling data
security and privacy matters. He earned his B.A., magna cum laude, from Boston College and his J.D.
from Boston College Law School; and is admitted to practice in New Jersey, New York, and
Washington, DC.

Honorable Garrett E. Brown, U.S.D.J. (Ret.)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Honorable Garrett E. Brown, U.S.D.J. (Ret.)

Uploaded by

Copyright:

Available Formats

Civil Litigation Track

Recent Legal and Technological Developments in E-Discovery:

Moderator: Honorable Garrett E. Brown, U.S.D.J. (ret.)

Speakers: Honorable Tonianne J. Bongiovanni, U.S.M.J

Honorable Tonianne J. Bongiovanni, U.S.M.J., Matthew Knouff, Esq.,

Honorable Garrett E. Brown, U.S.D.J. (ret.)

1. 52 eDiscovery Terms That You Should Know, Complete Discovery Source

2. Peace of Mind for Serious eDiscovery, Complete Discovery Source

3. Computer Systems Checklist (Administrator), Complete Discovery Source

7. Local Civil Rule 26.1

8. The Sedona Conference® “Jumpstart Outline,” March 2011 Version. Copyright

9. Meet and Confer Checklist, Complete Discovery Source

Preservation & Production

10. Computer Systems Checklist (Employee), Complete Discovery Source

11. Chain of Custody Short Form, Complete Discovery Source

16. Bull v. UPS, Inc., 665 F.3d 68 (3d Cir. 2012)

Social Media & Sanctions

35. Mara E. Zazzali-Hogan, Esq., “Lester v. Allied Part 2: ‘Clean Up’ of

41. Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-

42. The Grossman-Cormack Glossary of Technology-Assisted Review, Federal Courts

43. Technology Assisted Review Workflow Graphic, Complete Discovery Source

Peace of Mind for Serious eDiscovery™

Peace of Mind for Serious eDiscovery™

eDiscovery is a critical part of your case. From proactive litigation-readiness to

Litigation Readiness Processing and Production

eM@S, CDS’ proprietary data analysis system, • High-speed multi-thread processing

NEW YORK • WASHINGTON, DC • CHICAGO • SAN FRANCISCO • ATLANTA

Review Hosting and Global Coverage and Trans-

— E-DISCOVERY PROJECT MANAGER, TOP 50 LAW FIRM

We self-certify compliance with

The Cloud, Server Consolidation, Ephemeral Data, and Information

T range of technological issues: cloud storage, server

COPYRIGHT 姝 2012 BY THE BUREAU OF NATIONAL AFFAIRS, INC. ISSN 1941-3882

DIGITAL DISCOVERY & E-EVIDENCE REPORT ISSN 1941-3882 BNA 12-6-12

Two recent cases address the issue of ephemeral data

DIGITAL DISCOVERY & E-EVIDENCE REPORT ISSN 1941-3882 BNA 12-6-12

Guideline 10: Compliance with a legal hold should be regularly monitored.

Copyright © 2010, The Sedona Conference®

UNITED STATES DISTRICT COURT

ROXANE LABORATORIES, INC.

BONGIOVANNI, Magistrate Judge

GRANTED in part and DENIED in part.

I. Procedural History and Background

of ordinary skill in the art.

A. The Destruction of Documents

Medeva claims that Roxane impermissibly destroyed relevant documents after it

litigation hold) until July 2007.

involving Roxane’s prototype formulations as well as correspondence regarding the design of

information was destroyed.

Roxane’s evaluation of U.K. Asacol as well as Roxane’s representations to foreign regulatory

Roxane is liable for spoliation.

and Roxane’s claims of non-infringement.

In addition, Medeva claims that Roxane’s supplemental answers to Medeva’s

preclusion is justified even if such a sanction is tantamount to dismissal of Roxane’s claim. As

of documents already being preserved under Roxane’s document retention policies.

a single document concerning mesalamine.

Roxane was not involved in its production of documents.

generic product or Asacol in the human body.

either of the requested adverse inferences on it.

door for additional discovery motions.

containing attorney work product and attorney-client communications. (Roxane’s Sur-Reply at

wholesale production of all of Roxane’s protected information.

future misconduct by imposing fees and costs.

B. The Concealment of Lot No. 089226