Data Quality

Article

Development and validation of reporting

guidelines for studies involving data linkage

Abstract Megan A. Bohensky, Damien Jolley

Centre for Research Excellence in Patient Safety, Department of Epidemiology
Objective: Data or record linkage is
& Preventive Medicine, Monash University, Victoria
commonly used to combine existing
data sets for the purpose of creating
more comprehensive information to Vijaya Sundararajan
conduct research. Linked data may create Department of Human Services, Victoria
additional concerns about error if cases
are not linked accurately. It is important that Sue Evans, Joseph Ibrahim, Caroline Brand
factors compromising the quality of studies Centre for Research Excellence in Patient Safety, Department of Epidemiology
using linked data be reported in a clear & Preventive Medicine, Monash University, Victoria
and consistent way that allows readers
and researchers to accurately appraise

the results. The aim of this study was to
develop and test reporting guidelines for
evaluating the methodological quality of
studies using linked data.
Method: The development process
included a literature review, a Delphi
process and a validation process.
Participants in the process were all
Australian and included biostatisticians,
epidemiologists, registry administrators,
academic clinicians and a peer-reviewed
journal editor.
Results: The final guidelines included
four domains and 14 reporting items.
These included: data sources (six items),
research selected variables (four items),
linkage technology and data analysis
(three items), and ethics, privacy and data
security (one item).
Conclusion: This study is the first to
develop guidelines for appraising the
quality of reported data linkage studies.
Implications: These guidelines will
assist authors to report their results in
a consistent, high-quality manner. They
will also assist readers to interpret the
quality of results derived from data linkage
studies.
Key words: Data collection, medical
record linkage, guideline, research design,
peer review, research
Aust NZ J Public Health. 2011; 35:486-9
doi: 10.1111/j.1753-6405.2011.00741.x
D
ata or record linkage is “a
process of pairing records from
two files and trying to select the
pairs that belong to the same entity.”1 It
is commonly used to combine existing
data sets to create more comprehensive
information to conduct research. Studies
involving data linkage are becoming more
common. The Australian Government
through the National Collaborative Research
Infrastructure Strategy awarded $20 million
to the Population Health Research Network
(PHRN) to establish national capacity for data
linkage in Australia.2 The PHRN also received
more than $30 million in direct and indirect
support from each of the states and territories.
Data linkage will be conducted within
nodes operating in each individual State3-5
(see www.phrn.org.au), as in the model of
Western Australia, with the Centre for Data
Linkage, a national network, to co-ordinate
the activities of the state-based groups.
Although the linkage of data sources
generates valuable information and improves
data quality, it may create additional biases
and methodological concerns.6 To link data
sets accurately requires stable and sufficiently
unique identifiers. However, unique identifiers
are not always available due to ethical and
privacy constraints.
Different methods are used for standardising
data and linking data sets and the choice of
these may influence the quality of results.7,8
A systematic review of the accuracy of
probabilistic linkage processes found the
sensitivity (i.e. the proportion of truly linked
records detected) ranged from 74% to 98%.9
The authors noted that this variation was likely
to be due to the number and quality of fields
available for linkage.
Where there is low sensitivity of linkage
processes, differential inaccuracies in the
data may result in systemic bias. Linkage
rates vary by participants’ age,10 gender,11,12
ethnicity,13 health status, regional location14,15
and socio-economic status.13 These variations
may affect the conclusions of research
studies.16
Factors compromising the quality of
studies using linked data should be reported
in a systematic way to allow readers and
researchers to accurately appraise different
studies. Reporting guidelines have improved
the quality of information in other areas of
research by highlighting the shortcomings
and prompting improvements in the quality
of published studies.17 Currently, there is no
tool available to appraise the quality of studies
using data linkage.

Submitted: September 2010 Revision requested: March 2011 Accepted: April 2011
Correspondence to:
Dr Megan Bohensky, Centre for Research Excellence in Patient Safety, Department of
Epidemiology, Monash University, 99 Commercial Road, Level 6 Alfred Centre, Prahran,
Victoria 3181; e-mail: megan.bohensky@monash.edu

486 AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH 2011 vol. 35 no. 5
© 2011 The Authors. ANZJPH © 2011 Public Health Association of Australia
Data Quality Reporting guidelines for studies involving data linkage
Because the different data linkage nodes in Australia are subject
to different privacy legislation and organisational structures, there
is the potential that they may need to utilise differing methods and
identifiers to link data in each jurisdiction. The development of
standardised guidelines for reporting and appraising the quality
of linkages within each node is timely and can help achieve
greater national consistency in linkage results, especially where
jurisdictional data will be aggregated and considered at a national
scale. The Centre for Data Linkage is making concerted efforts to
harmonise these data for national analyses.
This study aimed to develop and validate reporting guidelines for
evaluating the methodological quality of studies using linked data.
Methods
A modified Delphi process was used to gain agreement about
the contents of reporting guidelines.18 This method has been
employed in the development of reporting guidelines for other
types of research studies, including the CONSORT statement for
the reporting of randomised controlled trials.19
There were three stages to the reporting guideline development
process: 1) a literature review; 2) a Delphi process, incorporating
an informant consultation process and two Delphi voting rounds;
and 3) a tool validation process.
Literature review
The literature review6 summarised articles that identified quality
issues with data linkage studies published from 1991 to 2007.
Thirty-three articles met the inclusion criteria from which four
domains and 26 items were identified that addressed issues of data
linkage reporting quality. The reasons for and the nature of bias
that arose from unlinked records were summarised and forwarded
to the participants in the consultation process.
Consultation process
The key informant consultation process was conducted with 10
experts selected through purposive sampling of Australian experts
in a range of fields related to data linkage. Participants were asked
to review the domains summarised from the literature review and
advise if additional domains and items, not previously identified,
should be included. Participants suggested a fifth domain focusing
solely on the variables to be used within the research study and an
additional 21 items to be added to the preliminary list. The final
list of domains and items was pilot tested by three independent
researchers for face and content validity.
Delphi process
Two Delphi voting rounds were undertaken and participants’
identities were kept anonymous.
Before the Delphi voting rounds, all participants were given a
background summary report of the project and literature review.
The Delphi survey process included participants who had data
linkage experience as researchers, technicians or users of data
2011 vol. 35 no. 5 AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH 487
© 2011 The Authors. ANZJPH © 2011 Public Health Association of Australia
linkage studies. Twelve (60%) of the 20 invited experts agreed to
take part in the Delphi process. The disciplinary backgrounds of
participants were: 17% biostatisticians, 33% epidemiologists, 17%
registry administrators, 25% clinician academics, 8% computer
scientists and 8% journal editors.
The ratings from the first round were summarised quantitatively
and qualitatively. The median group score, range and proportion
of participants rating the item at eight or above were calculated for
presentation in the second round. An a priori decision was made to
include items with a panel median score of eight or higher and with
a high level of agreement based on the Rand/UCLA Appropriateness
Method User’s Manual for strict agreement (A7R).18 Inter-percentile
Range Adjusted for Symmetry (IPRAS) scores, which are a measure
of score dispersion adjusted for panel symmetry, were used to
determine the level of agreement for each item.
After round one, 20 of the 47 items were ranked within the
‘included’ range according to Delphi criteria. In the second round,
participants were asked to re-rate items taking into consideration
the findings from the first round. The final list of ‘accepted’ items
and ‘threshold’ items (where nine or more people rated eight or
above and there was a high level of agreement) were circulated to
participants for review and comment after the end of the second
round. Following round two, 14 items remained.
Validation
The validation process randomly selected a sample of 25 from the
75 eligible articles (impact factors ranging from 0 to 15.7 grouped
into impact factor quintiles). The majority of articles were from
Australia, North America, United Kingdom and Scandinavian
countries. Two researchers (MB and CB) applied the guidelines to
the de-identified articles to rate how well each item was reported
within the article (not applicable, poorly addressed, adequately
addressed, well addressed).
The median number of items rated as ‘well addressed’ by at least
one reviewer in each article was six (range: 1-12). There was not
strong evidence of a relationship between impact factor and the
summary rating of items for each study (r=0.20). The proportion
agreement of the validation process was 71% and the kappa score
was 0.6. Domain-specific kappa scores were as follows: existing data
sources k=0.4; researcher selected variables and data preparation
k=0.5; technology and analysis of linked data k= 0.8; and ethical
review k=0.9.
Ethics approval for this study was received from the Monash
University Standing Committee on Ethical Research in Humans.
Results
The Delphi consensus process identified and validated reporting
guidelines including four domains and 14 reporting items (presented
in Table 1).
The final list of items incorporated six (43%) items from the
domain on data sources, demonstrating the importance of having
high-quality existing data systems to conduct high-quality linkage
research. Of the 14 items, four (29%) items were from the domain on
Bohensky et al. Article

researcher-selected variables related to the researcher’s consideration
of the quality of specific variables to be used. This domain also
demonstrates the importance of understanding the quality of existing
data sources and specific variables to be used for linked analysis.
There were three (21%) items included from the domain concerning
the technology and analysis of the linked data and one (7%) item
from the domain on ethics, privacy and data security was included
in the final list.
Review of the written comments indicated that most participants
who gave a low score to items in the domain on ethics, privacy
and data security concluded that if the study had the approval of
an ethics committee they could assume that the other items had
been addressed (e.g. security of data was maintained at all times
and consent to the linkage was obtained or not required). This
underscores the importance of ethical review and the governance
of research using linked data, especially if participant consent was
not provided for the collection or linkage of the data initially.
IPRAS scores in the second round showed agreement on all of
the items that met the criteria for acceptance and high agreement
on low rated items.
Discussion
This study is the first to develop guidelines for appraising quality
issues in studies using data linkage. This is an important endeavour
as more studies and reports using linked data are being published.
It is expected that these guidelines could be utilised by authors,
data linkage analysts and reviewers as a basis for understanding
the quality of their data sets, the linkage process and the possible
limitations of the associated findings. The guidelines are intended
to serve as a general framework. It is not expected that every item
will apply to each study. For example, many studies will not be
able to establish the specificity of their linkage results (i.e. if a
one-to-one link is not expected, it may be difficult to quantify the
number of false negatives). Nonetheless, the guidelines assist in
identifying where assumptions about the accuracy or quality of
data have been made.
Given the systematic investigation, the expertise of the Delphi
participants and broad disciplinary representation, this study offers
an exploratory basis for developing an accepted list of reporting
criteria for studies using data linkage. The validation findings
demonstrate that the criteria considered important by the experts
are not consistently reported in the literature, with a median of
only six items reported in the selected studies. This highlights an
important gap that should be addressed. The differences between the
criteria considered important by the experts and those consistently
reported in the literature may relate to the fact that many researchers
utilising linked data are not familiar with the issues that can impact
linkage quality, especially where data are linked by a third party,
such as a data linkage centre. Having standardised guidelines will
help to highlight these concerns for researchers and readers of data
linkage studies.
There are several limitations of this study. First, Delphi methods
that include only anonymous voting components have been criticised
for not including expert group discussion. However, we modified this
process by including the initial group interactions before the voting.
The high levels of agreement precluded the need for further direct
discussions. Although the total number of participants in this project
was small, all participants had publications in the area of expertise
and took part in both rounds of the process, so participant drop-out
did not influence our findings. The reliability of the application
of these guidelines was moderate with a kappa statistic of 0.6.20
While the overall kappa score reflects moderate agreement, this is
consistent with other critical appraisal tools tested for reliability

Table 1: Reporting guidelines for studies using data linkage.

DOMAIN 1: The existing data sources to be linked (complete for each dataset to be linked in the study)
Dataset 1 Dataset 2 Dataset 3 Dataset 4
1. Purpose of the dataset was given
2. Description of the type of dataset (administrative/ clinical registry/ research study) was specified
3. Any standardised coding system/data dictionary used should was stated
4. % population coverage
5. Data collection methods were described
6. Data quality assurance process described, including the frequency of checks
DOMAIN 2: Researcher-selected variables and data preparation
7. Were the participant inclusion criteria specified?

8. Were the variables used for linkage stated (including rates of missing data)?
9. Were changes to the coding systems reported (including changes over time or revisions to disease/risk factor definitions revised during the study period)?
10. Were potential sources of bias adequately described?
DOMAIN 3: Technology/linkage process
11. Was the intended precision of the linkage stated?
12. Was a description of the linkage method given (e.g. deterministic, probabilistic methods including use of blocking and phonetic coding, if used) and a
justification for the use of this type of linkage provided?
13. Was a measure of the quality of the linked data-sets provided (e.g. % linked records, false positive/false negative rates)?
DOMAIN 4: Ethics, privacy, data protection and access arrangements
14. Did the study receive approval from a human research ethics committee?

488 AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH 2011 vol. 35 no. 5
© 2011 The Authors. ANZJPH © 2011 Public Health Association of Australia
Data Quality Reporting guidelines for studies involving data linkage
(inter-rater agreement 0.6 to 0.7, ranging from 0.12 to 0.95).21,22
The poorest agreement was in the domain on existing data
sources, which may reflect the number of items and complexity
of this domain, especially if studies involved more than two data
sources. The domain which considered whether ethics approval had
been received had the highest level of agreement. The use of the
Delphi process ensures a high degree of face validity.
A relationship was not found between impact factor and the
number of items that were considered to be well addressed, but
impact factor is a debated proxy measure for scientific quality.23
This study may also have been under-powered to detect such a
relationship. Nevertheless, the tool is intended to serve as a general
guide to authors and reviewers in relation to aspects of their study
of data linkage. It is likely that other topical content will also be
taken into consideration during the authorship and review of the
article, and high-quality linkage methods do not necessarily imply
that the research is of high scientific merit.
As we restricted the process to 12 participants, this process could
be expanded in the future to include participants from outside
Australia and public health disciplines.
Conclusion
This study is the first to develop reporting guidelines for
appraising the quality of studies using data linkage. It is fundamental
that high quality and systematic data linkage methods be supported
to advance research. As studies relying on linked data are becoming
more prevalent in the Australian research community with the
development of national data linkage capacity, these guidelines may
assist authors and reviewers in producing high-quality research.
Acknowledgements
The authors gratefully acknowledge the members of the Delphi
panel and their colleagues Dr Cameron Willis, Sacha Höttje and
Christine Moje for their helpful review and feedback on drafts of
this manuscript. Megan Bohensky received funding for her PhD
through an Australian Postgraduate Award.
2011 vol. 35 no. 5 AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH 489
© 2011 The Authors. ANZJPH © 2011 Public Health Association of Australia
References
1. Winglee M, Valliant R, Scheuren F. A case study in record linkage. Surv
Methodol. 2005;31:3-11.
2. National Collaborative Research Infrastructure Strategy. Strategic Roadmap
for Australian Research Infrastructure. Canberra (AUST): Commonwealth
Department of Innovation, Industry, Science and Research; 2008.
3. Holman CDA, Bass AJ, Rouse IL, Hobbs MST. Population-based linkage of
health records in Western Australia: development of a health services research
linked database. Aust N Z J Public Health. 1999;23:453-9.
4. Centre for Health Record Linkage. Guide To Health Record Linkage Services.
Version 1.3; Sydney (AUST): Cancer Institute NSW; undated.
5. Deapartment of Health. Victorian Data Linkages [Internet]. Melbourne (AUST):
State Government of Victoria; 2011 [cited 2011 Mar]. Available from: http://
www.health.vic.gov.au/vdl/index.htm
6. Bohensky M, Jolley D, Sundararajan V, Evans S, Scott I, Brand C. Data linkage:
a powerful tool with potential problems. BMC Health Serv Res. 2010;10:346.
7. Yancey WE. Evaluating string comparator performance for record linkage. In:
Research Report Series (Statistics #2005-05) [Internet]. Washington (DC): U.S.
Census Bureau, Statistical Research Division; 2005 [cited 2010 Nov]. Available
from: http://www.census.gov/srd/papers/pdf/rrs2005-05.pdf
8. Gomatam S, Carter R, Ariet M, Mitchell G. An empirical comparison of record
linkage procedures. Stat Med. 2002;21:1485-96.
9. Silveira DP, Artmann E. Accuracy of probabilistic record linkage applied to
health databases: systematic review. Rev Saude Publica. 2009;43:875-82.
10. Li B, Quan H, Fong A, Lu M. Assessing record linkage between health care
and Vital Statistics databases using deterministic methods. BMC Health Serv
Res. 2006;6:48.
11. Bopp M, Minder CE. Mortality by education in German speaking Switzerland,
1990-1997: Results from the Swiss National Cohort. Int J Epidemiol.
2003;32:346-54.
12. Dunn KM, Jordan K, Lacey RJ, Shapley M, Jinks C. Patterns of consent in
epidemiologic research: evidence from over 25,000 responders. Am J Epidemiol.
2004;159:1087-94.
13. Adams MM, Wilson HG, Casto DL, et al. Constructing Reproductive Histories
by Linking Vital Records. Am J Epidemiol. 1997;145:339-48.
14. Gyllstrom ME, Jensen JL, Vaughan JN, Castellano SE, Oswald JW. Linking
birth certificates with Medicaid data to enhance population health assessment:
methodological issues addressed. J Public Health Manag Pract. 2002;8:38-44.
15. Hoving JL, Monaco A, MacFarlane E, et al. Methodological issues in linking
study participants to Australian cancer registries using different methods:
lessons from a cohort study. Aust N Z J Public Health. 2005;29:378-82.
16. O’Reilly D, Rosato M, Connolly S. Unlinked vital events in census-based
longitudinal studies can bias subsequent analysis. J Clin Epidemiol.
2008;61:380-5.
17. Plint AC, Moher D, Morrison A, et al. Does the CONSORT checklist improve
the quality of reports of randomised controlled trials? A systematic review.
Med J Aust. 2006;185:263-7.
18. Fitch K, Bernstein SJ, Aguilar MS, et al. The RAND/UCLA Appropriateness
Method User’s Manual [Internet]. Santa Monica (CA): Rand; 2001 [cited 2010
Sep]. Available from: http://www.rand.org/pubs/monograph_reports/MR1269/
index.html
19. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised
recommendations for improving the quality of reports of parallel-group
randomised trials. Lancet. 2001;357:1191-4.
20. Landis JR, Koch GG. The measurement of observer agreement for categorical
data. Biometrics. 1977;33:159-74.
21. Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of
randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.
22. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the
quality of randomized controlled trials: an annotated bibliography of scales
and checklists. Control Clin Trials. 1995;16:62-73.
23. Grzybowski A. The journal impact factor: how to interpret its true value and
importance. Med Sci Monit. 2009;15:SR1-4.