Eli Lilly and Quintiles Case Study:

Using MCC ECG Performance Metrics to Improve Sponsor, Site,

and Core Lab Clinical Trial Operations
 Metrics Champion Consortium - Overview
Today’s drug development industry is under increased pressure to improve its research and
development performance/strategies by reducing drug development times and costs, while at
the same time dramatically increasing productivity and maintaining quality.1 Biotechnology and
pharmaceutical organizations that are currently achieving efficient clinical trial cycle times attribute
their success to the following “best practices”1:
• Maintaining focus on core competencies
• Prioritizing the utilization of resources
• Outsourcing services
The utilization of standardized performance metrics by the drug development industry to reduce
clinical development times and to improve clinical trial deliverables is essential to its success.
Sponsors and ECG Core Labs need to effectively manage and track their resources, including
people, time, and money, as each has an impact on productivity and efficiency over the course
of a clinical trial.

Building Partnerships Around Standardized Performance Metrics

Biotechnology, pharmaceutical, and service provider organizations have joined together to form a
not-for-profit organization, the Metrics Champion Consortium (MCC), where member organizations
work collaboratively to develop and implement a set of standardized performance metrics which aim
to improve the efficiency and effectiveness of clinical trials. Implementing standardized performance
metrics benefits sponsor and service provider organizations by enabling them to compare clinical trial
performance across all of a sponsor’s studies, even if they utilize multiple service providers. Together,
sponsors and service providers can review performance with the aim to identify both best practice
scenarios and opportunities for improvement. Once opportunities for improvement are identified,
sponsor/service provider partners can review the specific procedures that determine the process in
question and determine appropriate action steps to improve performance.
Throughout the metrics development and implementation process, the MCC provides a safe harbor for
biotechnology, pharmaceutical, and service provider organizations to share “best practices” and learn
from each other through participation in MCC learning forums and collaborative work groups.

MCC ECG Metrics – Mission Statement

To develop and support a baseline set of ECG performance metrics provided by service providers
within the biotechnology and pharmaceutical industry with the intent to jointly encourage the
sponsors, service providers, investigative sites, and equipment manufacturers in gaining performance
improvement, effectiveness, efficiency, and appropriate levels of controls in support of the drug
development process.

MCC ECG Metrics Development Process

In 2006, a group of Sponsors, ECG Core Labs, and other ECG-related Service Providers began
developing a set of standardized MCC ECG Performance Metrics. The group worked together on
a Steering Committee and in three Working Groups to develop a draft set of standardized ECG
performance metrics [Figure 1]. In October 2007, after reviewing industry feedback, the MCC released
the ECG Performance Metrics version 1.0.

1. Kaitin KI. “Pushing the Innovation Envelope: Drug Development Metrics and the Changing Dynamics of Pharmaceutical R&D.” Presented at the 6th Annual Pharmaceutical
Metrics Event: Driving Quality, Cost, & Time; October 16-18, 2007; Cambridge, MA.

2 Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations
 Figure 1. MCC Metrics Development Process Model

Web Cast Kickoff

MCC Board
of Directors
Industry Survey

Metrics Prioritization Survey

Work Groups
A - Clinical Steering
B - Operations Committee
C - Outsourcing Metrics (Beta Draft)

Metrics
Industry-wide Feedback
Members Only

Open Access Performance Metrics v 1.0

Why Did Lilly and Quintiles Decide to Collaborate in Utilizing the

MCC ECG Performance Metrics?
Premise: If performance metrics are the questions organizations ask of their businesses and their
partnerships, then effective collaboration ensures they are asking the right questions.

From the sponsor perspective, asking the right questions leads to

• Better understanding of the services their core lab provides
• Enhanced understanding of their core lab’s processes
• Improved monitoring of their core lab’s ability to meet critical timelines
• Ability to compare results across all studies (even across multiple core labs)
• Ability to pinpoint areas of interest & utilize both sponsor and core lab knowledge bases
to improve processes
• Enhanced partnership relationships through improved communication among all parties

From the service provider perspective, asking the right questions leads to
• Identification of their client’s business requirements
• Enhanced understanding of their client’s processes
• Improved monitoring of their client’s critical deliverables
• Greater focus on achieving meaningful process improvement
• Higher customer satisfaction
Meaningful conversations around performance metrics will create a better partnership. This
partnership will allow the companies to share goals, understand the mutual needs of each company
and learn how processes can be streamlined which will, hopefully, result in efficiency gains and cost
savings by reducing redundant steps between or within each company. Opening this communication
pathway is expected to increase the visibility of each organization (both within the organization and
between the organizations), thus leading to a better understanding of what each company does and
what each company needs to effectively work.

Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations 3
 Results: Quality Improvements Through Review of Metrics
Example 1: Percentage of ECGs Reported within Agreed Turnaround Time

Unit of Reporting
Definition Formula/Example Target
Measure Frequency

Minimum:
The percentage of ECGs that have met the agreed
upon turnaround time from ECG receipt to successful Formula:
notification of the results to the site. The TAT is defined
and agreed upon between the sponsor and core lab and (Total N of ECGs that Total N and
may be different for every protocol. met the expected Percentage Monthly >95%
turnaround time ÷ (%)
Additional analysis on a “for cause” basis: Total N of ECGs
received) x 100
A listing of ECGs that did not meet the expected
turnaround time and the rationale for missing this target,
broken out by protocol and/or site.

Study Study
ABOVE 4 9

TARGET JUNE 2007

BELOW Study Study Study Study Study Study Study Study Average
1 2 3 5 6 7 8 10

Review Steps
• Examined the “below target” studies and determined that many of the studies missed their
turnaround times by less than 1 hour of agreed turnaround time. Understanding this was a good
indication that the core lab cannot ‘fudge’ data to allow ECG turnaround times that are ‘in the
neighborhood’ to be shown as meeting the expected turnaround time. Receiving data such as
this is actually a very good indicator of the openness of the core labs
• Determined that hand-offs between Quintiles Operations and Clinical departments were not
completed efficiently enough to allow ECGs to meet the expected turnaround time
• Compared the Lilly protocols vs Quintiles process and discovered that the Lilly approach to
replicate ECGs and the Quintiles system were not compatible
• Lilly collects triplicates of ECGs but only requires that only 1 of the 3 copies is reported to
Investigative Sites
• Quintiles’ autofax system is not set up to fax only 1 of 3 reports – it is an “all or none” system –
Quintiles has to use a manual faxing process to accommodate Lilly’s protocol

Action Steps
1. Quintiles automated the handoffs between departments within Quintiles to allow their ECG
processing to flow more efficiently
2. Quintiles updated the “autofax” system to accommodate Lilly’s protocols

4 Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations
 Results

Pecentage ECGs Reported within TAT

100% An automated approach
was implemented and the
turnaround time issues
95%
have been addressed.

90%

85%

80%
07 l-0
7 07 07 t-0
7 07 07 08 08 r-0
8
r-0
8 08 08
n- Ju g- p- v- c- n- b- y- n-
Ju Au Se Oc No De Ja Fe Ma Ap Ma Ju

The manual faxing issues due to Lilly’s replicate The manual processes implemented previously
ECGs were addressed through manual were not addressing the needs of the core lab.
processes. Additionally, the core lab set up The core lab implemented new manual processes
manual processes for hand-offs within the core by lowering the number of individuals involved
lab’s processes to assist in addressing these to ensure the process continued to meet the
issues. The manual processes implemented at expectations of the sponsor until the automated
this time were incorporated as a temporary fix ‘fix’ was accomplished. This refinement of the
until these processes could be automated. As manual process ensured that the turnaround
can be seen, these manual ‘fixes’ addressed the times were met until the automated processes
issues for several months. could be implemented.

Non-
Study Study Study Study Study Study Study Study Study Lilly Lilly
ABOVE 1 2 3 5 6 7 8 9 10 Avg Avg
TARGET JUNE 2008
BELOW Study
4

Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations 5
 Results: Quality Improvements Through Review of Metrics
Example 2: Percentage of On-time ECG Equipment Shipments to Sites

Unit of Reporting
Definition Formula/Example Target
Measure Frequency

Formula:

Minimum: (Total N of protocols

with first equipment Total N and
The percentage of sites who received their ECG equipment Percentage Quarterly >95%
shipped date met ÷
by the agreed upon receipt date based on defined (%)
Total N of protocols
expectations between the sponsor and core lab.
with first equipment
required) x 100

Study Study Study Study Study Study

ABOVE 2 3 4 5 9 10

TARGET JUNE 2007

BELOW Study Study Study Study Average
1 6 7 8

Review Steps
• Studies 1 & 6: Reviewed the protocols and determined that these studies had their own
equipment so no equipment was being shipped to sites, demonstrating the need to fully
understand the studies that are being reported and understanding the story behind the metric
• Studies 7 & 8: Reviewed data and found a problem in the process by which the “on-time”
date was established
• Lilly set optimistic start dates early in protocol development which were revised prior to the
launching the study
• Quintiles was using the initial date provided on the initial form submitted by Lilly, not the
updated start dates

Action Plan
1. Quintiles will indicate “not applicable” for studies where site-owned equipment is used
2. Lilly will establish realistic start dates and communicate them appropriately to Quintiles. Both
companies gained an understanding of the information that is shared between the sponsor
and the core lab and how this information should be used. Through this understanding and
clarification of processes, the core lab and the sponsor know which information is to be used
for projections and which information should be used for shipping equipment to the sites.

6 Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations
 Results

Pecentage On-time ECG

Equipment Shipments to Sites

100% This slight drop was due to a

customs issue. This is important
because by identifying these
80% country-specific issues (such as
customs), we can forecast shipping
difficulties in future studies.
60%

40%
This is the point where the alignment
20% on the meaning and usefulness of the
dates provided by the sponsor to the
core lab were established.
0%
07 l-0
7 07 07 t-0
7 07 07 08 08 08 r-0
8
-0
8 08
n- Ju g- p- v- c- n- b- ar- ay n-
Ju Au Se Oc No De Ja Fe M Ap M Ju

Non-
Study Study Study Lilly Lilly
ABOVE 5 12 14 Avg Avg
TARGET JUNE 2008
BELOW

Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations 7
 Results: Quality Improvements Through Review of Metrics
Example 3: Percentage of ECG Queries from Vendor to Site

Unit of Reporting
Definition Formula/Example Target
Measure Frequency

Formula:

Minimum: (Total N of queries

generated between Total N and
The number of queries generated between the core lab Percentage Quarterly <20%
the core laboratory
and the site, compared to the number of ECGs received for (%)
and the site ÷ Total
a sponsor and core laboratory
N of ECGs received)
x 100

Study Study
ABOVE 7 8

TARGET JUNE 2007

BELOW Study Study Study Study Study Study Study Study Average
1 2 3 4 5 6 9 10

Review Steps
• Studies 7 & 8: Both studies are Oncology studies; Traditionally, oncology study sites have not
previously focused on the ECG/Cardiac safety component of clinical studies
• High number of queries related to the system generating multiple queries when a single ECG has
missing, inconsistent, or unexpected information. We need to better define how to handle this
metric as some vendors will report multiple queries on a single ECG while others would indicate
that if an ECG has multiple queries on it, it is only counted as one

Action Plan
• Lilly and Quintiles will review the study data by site to determine which sites need to be retrained
in order to submit ECGs with complete information – thus reducing the query workload for the
sites and Quintiles
• Lilly will review results to identify how effective different site training methodologies (in person
vs web conference vs taped training) are to determine the best option. This analysis was
completed and it showed that with over 1.5 years of data the data does not suggest that the
type of site training had any impact on the reliability of the site to complete the demographic
information appropriately

8 Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations
 Results

Pecentage Site Queries

30%

25%

20%

15%

10%

5%

0%

07 l-0
7 07 07 t-0
7 07 07 08 08 08 r-0
8
-0
8 08
n- Ju g- p- v- c- n- b- ar- ay n-
Ju Au Se Oc No De Ja Fe M Ap M Ju

Study Study Study Study Study Study Study Lilly

ABOVE 1 2 4 5 6 8 12 Avg

TARGET JUNE 2008

BELOW Non-
Study Study Study Study Study Study Study
Lilly
3 7 9 10 11 13 14
Avg

Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations 9
 Results: Quality Improvements Through Review of Metrics
Example 3: Percentage of ECG Queries from Vendor to Site

Lessons Learned
The Initial Action Plan does not always solve the problem. The team is conducting “for cause” analysis to try to
determine the underlying cause of the queries, including:
• What is most common issue?
Queries raised for the sites collecting the incorrect number of ECGs/timepoint are the most common issue.
However, using metrics such as this you can identify machine-specific nuances that can assist with site training.
Certain data entry fields have more queries raised for them than others (such as the last field that is to be entered
and the site not hitting <ENTER> prior to moving on, site not selecting a entry due to confusion over how the
selection works, etc.) and understanding these nuances can lead to more effective training
• Bad site selection?
This still needs to be investigated. However, certain sites always have more queries than the others. Using the
metrics we can target those sites for re-training in an effort to reduce this query load. Lower queries result in faster
ECG reports back to the site, less work for the core lab, less work for the sites and less work for the sponsor
• Specific study indication(s)?
Yes.....some indications have far more queries than others (i.e. oncology). This is probably due to investigative
sites who do not have as much experience with ECG equipment or investigative sites that don’t believe ECG
findings should jeopardize a compound. This can also assist in identifying sites that need to be re-trained
• Poor training?
This analysis was completed and it showed that with over 1.5 years of data the data does not suggest
that the type of site training had any impact on the reliability of the site to complete the demographic
information appropriately
• Quintiles ECG Training
• Lilly Training
• CRO Training
• Web Training
• CD Training

• Individual sites vs. across multiple sites?

Individual sites within a study tend to drive this number in the wrong direction and be worse than the overall study.
Identifying these sites can lead to targeted re-training of those sites

As of June 2008 the Lilly average for queries is higher than that of the other sponsors this core lab works with.
It is important that we keep an eye on this comparison and determine why this would be. Is it because Lilly asks
the sites to enter information into the ECG machine that is much more difficult than what other sponsors ask
of their sites? Is Lilly selecting sites that are not as detail oriented as the sites other sponsors select? Are the
indications that Lilly is studying that much different than other sponsors? Is the Lilly training model leading to these
issues? These are all aspects that should be reviewed when considering how our query metrics stack up against
other sponsors.
We would expect that early in a study the number and percentage of site queries will be relatively high. However,
we would also expect that these numbers would go down after the first few months. The sites that are the most
concerning are those that have a high query percentage, have the highest amount of outstanding queries and take
the longest to resolve those queries with the core lab. Using these three indicators can identify sites that have
issues that need to be addressed sooner rather than later.

10 Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations
 How Key Players’ Perceptions Changed During the MCC
Performance Metrics Development and Implementation Process
Initial Expectations
• Sponsors: Interested in utilizing standardized performance metrics as a standardized
measurement tool for evaluating ECG core labs across the industry
• Service providers: Concerned that the MCC performance metrics would be used as
a “punitive” tool

Changes in Expectations
• Sponsors: While using these across the industry is still appealing, it was recognized that this
may not be the best use for the metrics; However, developing the metrics and reviewing the
results led to an enormous opportunity to learn about ECG core lab procedures, as well as
performance at the core labs, investigative sites and sponsor. Sponsors must understand the
study the core labs are managing and how the companies work together to draw meaningful
conclusions from these metrics. Even those close to the metrics cannot take the results at face
value without fully understanding the study, what the core lab does for each study, and what the
sponsor is requesting for each study. Because of this, care should be taken when sharing this
information with those not close to this service
• Service providers: Participation in the MCC forum allowed the free exchange of views before
the metric are finalized, reassured vendors that their views on the utility of the metrics were
considered; Through this participation and the discussions that have ensued it is being seen
that they are not being used punitively but rather leading to a better understanding of how
companies can work together

Key Requirements for Successfully Developing and Implementing

Performance Metrics
• Open, honest communication (A Neutral Forum)
• A structured approach (ie The MCC)
• The recognition that both Vendor and Sponsor have areas for improvement and a willingness to
consider all proposed changes for process improvement
• A recognition of both partners positions
• The willingness to seek mutual process improvement
• Acceptance that process improvement can apply to all parties

Eli Lilly and Quintiles Case Study: Using MCC ECG Performance Metrics to Improve Sponsor, Site, and Core Lab Clinical Trial Operations 11
 MCC ECG Performance Metrics v 1.0

Metric Metric Title

Average number of days from ECG study award to contract signature
1
Note: For stand-alone ECG projects

Average number of days from signed ECG technical specifications document (TSD)
2
signature to vendor ready to receive ECGs

3 Percentage of on-time ECG equipment shipments to sites

Percentage of sites who conduct a successful test ECG transmission prior to

4
1st subject visit

5 Percentage of ECGs reported to Investigator Sites within agreed turnaround time

6 Percentage of ECG data queries from vendor to site

7 Turnaround time on resolution of ECG site queries from central vendor

Percentage of ECG alerts successfully communicated to sites within defined

8
turnaround time

9 Percentage of ECGs received from one study that were interpretable by the core lab

Percentage of manual adjustments of automated QT annotations from one study

10 (semi-automatic “computer assisted” method with visual inspection and manual
adjustment whenever necessary)

11 Percentage of ECG equipment failure

12 Average turnaround time on replacing faulty ECG equipment

13 Percentage of on-time, accepted ECG file transfers

14 Key ECG core lab personnel turnover during protocol

15 Percentage of ECG core lab audit/assessment findings closed within agreed timelines

16 Average Percentage of variance maintained in the ECG budget

© 2008 Metrics Champion Consortium, Inc. All rights reserved

Metrics Champion Consortium

706 Pro-Med Lane, Suite 100
Carmel, IN 46032
[office] 317.848.2908
[fax] 317.848.8861
www.metricschampion.org

04/09