1

SCDM
EMEA CONFERENCE

23 - 25
OCTOBER 2019

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 2
Risk-based Quality Management in Clinical
Data Management
Informal CDM Leadership Network
in collaboration with

Medidata
Represented by Peter Stokman (Bayer) & Lisa Ensign (Medidata)

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 3
RBQM in CDM
Informal CDM Leadership Network
Currently 20 companies; 44 participants
Participating companies:

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 4
RBQM in CDM
• Background
• Methodology
• Preliminary Results
• Preliminary Conclusions

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 6
RBQM in Clinical Development
Guidances to encourage better results by using smarter approaches
• ICH Q9 Quality Risk Management (January 2006)
‘… provides principles (…) for QRM that can be applied.’
• ISO 31000: Risk Management Principles Overview 2009
• FDA Guidance: Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring (Aug 2013)
• Transcelerate: Position Paper: Risk-Based Monitoring Methodology (May 2013; + Updates)
• EMA Reflection Paper on risk based quality management in clinical trials (November 2013)
• ICH E6 (R2) Addendum (June 2017)
• ICH E8 (R1) Designing Quality in Clinical Trials (May 2019)

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 8
RBQM in Clinical Development
• Heart of ICH E6 R2: focus on efficient and effective, identifying quality
standards (Quality Tolerance Limits)
• Movement towards understanding what each aspect of the quality
cycle (Monitoring (central, onsite, offsite, statistical), Data Management)
will – and will not – add to quality
• As a consequence: quantify the effectiveness of each part & evaluate
whether it should be used - even on critical data.

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 9
RBQM in Clinical Development
• So what can we
do? 3.7% 1.1%
1.2%
• Work harder!
1.4%

In Summary, all TAs combined:

Queries Q from SDV Qs SDV/ Qs SDVCritical/
(N) (N) Qs All (%) Qs All (%)
349,081 30,680 7.8 (0.5-22.1) 2.3 (0.1-6.6)
Evaluating Source Data Verification as a Quality Control Measure in Clinical Trials
Nicole Sheetz, Therapeutic Innovation & Regulatory Science, 2014, Vol. 48(6) 671-680

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 10
RBQM in Clinical Development
Focus where it matters most  Identify where it matters most
For Monitoring:
• Source Data Verification - Source Data Review
• Trial sites with characteristics correlated with poor performance or noncompliance
• Try to limit time wasters: SDV
For Clinical Data Management – particularly Data Cleaning
• Data Changes
• Critical Data
• Time wasters: queries in non-critical domains that rarely lead to data changes
• 28-71 $ ($50) https://www.linkedin.com/pulse/clinical-data-quality-query-resolution-costs-steven-law/

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 11
RBQM in Clinical Data Management
Seven companies teamed up with Medidata to answer the following
questions:
• Which clinical domains are queried most
• What percentage of these queries are actually leading to data
changes
• In which data fields
• How is requerying related to data changes (future work)

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 12
Study Methods
• Select recently completed Phase III studies
• representing a variety of sponsors and TAs
• primary endpoint within last 5 years
• Extract query data and standardize using Medidata’s FFC ML modeling
approach with supervised review
• Analyze data
• Primary: by TA, Form Domain, Field, Direct/Indirect Change, Requeried
• Secondary: Query Initiator, Time to Query Resolution

No sponsor-identifiable data will be disclosed. No patient-level data

utilized.
SCDM EMEA CONFERENCE | 23 – 25 OCTOBER
 13
Results – Study Summary
Range: 200 – 4200 subjects
Median: 600

Range: 2 – 54 mos
Median: 29.3

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER PULM/RESP

 14
Query File Extraction

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 15
Query Summary
• 2,055,808
Queries
Vaccine/Infection
Pulm/Resp (20%)
(13%)
Pain/Anaesthia
(6%)
Oncology (44%)
Automatic
CV (17%)
Automatic (68%)

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 16
Who is Initiating Manual Queries?

N = 611,265
e.g., Includes
queries by CRA 30%
& Site Monitor

70%

e.g., Includes queries

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

by DM, Data Monitor,
Safety MM, Medical
Monitor, ME
 17
Query Standardization
Use ML to Standardize Data across Multiple Trials

SCDM
Standardized Forms & Model Applied to RBQM
23 – 25
EMEA CONFERENCE | Fields OCTOBER Queries
 18
Query Standardization Form Domains
• 2,055,808 Queries standardized to
• 27 standardized form domains
• 196 standardized field names
Datapage Name Dataset Variable Control Type Field Pretext AI Predicted AI Predicted
Name Name Form Domain Field
death dd_pv_all ddseccs LongText Secondary Causes of Death Survival DEATH_CAUSE
complementary Status/Death
information Add Example Details
death dd_onc ddorres RadioButton Cause of death SSDD DEATH_CAUSE
(Vertical)
death dt dtcause LongText Primary cause of death SSDD DEATH_CAUSE
death dd_pv_all ddprcs LongText Primary Cause of Death SSDD DEATH_CAUSE
complementary
information
death data deth decu DropDownList Primary cause of death SSDD DEATH_CAUSE
death form death dthreas LongText Primary cause of death SSDD DEATH_CAUSE
death form death deathreas Text Cause of death SSDD DEATH_CAUSE
death report dd prcdth DropDownList Cause of Death (indicate most probable): SSDD DEATH_CAUSE
form review
SCDM EMEA CONFERENCE | 23 – 25 OCTOBER
 Results – Manual Query Counts by 19

Standardized Domain
More
queries in
domains that
are collected
throughout
the study:
CM, LB & AE

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 20
Results – Manual Queries by Data Change
• Data Change
• “Direct”: change in queried field
• “Indirect”: change in other field (based on verbiage used)
• 70% of manual queries DID result in a data change
• 58% “Direct” change (Answer Data ≠ Query Data)

• 12% “Indirect” change (Answer Data = Query Data)

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 21
Field-Specific Result Examples

Mean 70%
Changed

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 22
Results – Example Fields with Lower Query Efficiency
• AE Serious
• DS Death Date, IC Date, Treatment Completion/Discontinuation Date
• EX Treatment Name, Dose Frequency, Dose Units
• CM Regimen Number, Start Time, Discontinuation Reason, Response/Result
• LB Name
• MH Body System Consider Query
Efficiency AND overall /
• SV Visit Date TA-specific Criticality

• VS Systolic BP

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 23
Results – Example Fields with Lower Query Efficiency
• AE Serious
• DS Death Date, IC Date, Treatment Completion/Discontinuation Date
• EX Treatment Name, Dose Frequency, Dose Units
• CM Regimen Number, Start Time, Discontinuation Reason, Response/Result
• LB Name
• MH Body System Consider Query
Efficiency AND overall / Domain Criticality
• SV Visit Date TA-specific Criticality (to be determined by study team)
High
• VS Systolic BP Medium
Low

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 24
Comparison to Previous Work
Prior Work RBQM Analysis
Transcelerate RBM Position SDV Queries = 7.8% of All Queries SDV = 8.9% of total queries
Paper
Sheetz et al Paper Focus on all Manual queries
Focus on SDV process
1% eCRF data corrected by manual
1.1% eCRF data* corrected by SDV queries
*Removed
No data removed
• Batch-loaded data
• Non SDV data Included changes where missing data
updated to a populated value
Excluded changes where missing
data updated to a populated value Included “indirect” changes
Did not include “indirect” changes

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 25
Preliminary Conclusions
• Manual queries only lead to a correction in 1% all data
Yet: 70% of manual queries are effective - i.e.: lead to a correction of the data
• Domains and fields can be identified where manual queries lead to relatively
low & relatively high percentages of data changes – impact on quality not clear
• By using this data as historical estimates of data cleaning efficacy, and by
combining them with data criticality assessment, these findings can be used to help
focus on where it matters most
• By looking at the efficacy of queries on an individual trial basis, the efficiency of the
query process can be improved incrementally in the Plan/Do/Study/Act quality
circle

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 26
Future Analyses
• Evaluation of re-query efficacy
• Further dataset mining
• Comparison to ML data quality detection algorithms
• Use as historical data to help specify TA specific tolerance limits
• Publish the results in more detail:
• SCDM’s Data Basics, DIA’s TIRS, ACT, Perspectives in Clinical Research,
International Journal of Big Data Management
• 2020 Annual DIA Conference – abstract submitted
• 2020 Japan DIA Clinical Data Management Workshop - Invited
• Medidata NEXT

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER

 27

SCDM
EMEA CONFERENCE

23 - 25
OCTOBER 2019 Thank you!

SCDM EMEA CONFERENCE | 23 – 25 OCTOBER