1

Issues in the Design and Testing of

Business Survey Questionnaires:

What We Know Now that We Didn’t Know Then –

and What We Still Don’t Know

Diane K. Willimack
10 U.S. Census Bureau
Ec
on
om
11 ic
Ce
nsu
s

The International Conference on Reshaping Official Statistics

2
Issues in the Design and Testing of
Business Survey Questionnaires:

Lessons from 10 Years of Pretesting

Business Surveys at the U.S. Census Bureau

Diane K. Willimack
10 U.S. Census Bureau
Ec
on
om
11 ic
Ce
nsu
s

The International Conference on Reshaping Official Statistics

3
 Outline
 Background
– U.S. Census Bureau economic programs
– Characteristics of economic surveys
• Response process

 Pretesting procedures and

methodology

 Issues and recommendations

4
 Economic Survey Programs
at the U.S. Census Bureau
 More than 100 economic survey
programs plus economic census

 Authority – mandatory vs. voluntary

 Data collection modes

5
Characteristics of Economic Surveys

 Survey organization’s perspective

– Population, data, and survey design

 Respondent’s perspective
– Response process

6
The Response Process in
Economic Surveys

Cognitive response model

PLUS

Organizational processes

7
Cognitive Response Model
(Tourangeau, 1984)

4. Comprehension
5. Retrieval
6. Judgment
7. Communication Survey

8
Response Process Model for
Establishment Surveys
(Sudman et al., 2000 ICES-2)


Organizational
 Aspects
4. Comprehension
5. Retrieval
6. Judgment
Busine ss
7. Communication Survey

2.
9
Response Process Model for
Establishment Surveys
 Encoding in Memory / Record Formation
 Selection / Identification of Respondent(s)
 Assessment of Priorities (Motivation)
4. Comprehension
5. Retrieval from Memory and / or Records
6. Judgment
Busine ss
7. Communication Survey

2. Release of the Data

10
 Pretesting Goals

 Identify potential response error, e.g. ---

– Question clarity
– Contents of records
– Response strategy

 Suggest improvements to survey

questions
11
 Economic Survey Pretesting
at the U.S. Census Bureau

 Dedicated staff
 Consultant – client relationship with
survey programs
– More than 25 separate programs plus the
economic census
– Variety of survey types and topics
 Electronic mode
– Usability testing
12
 Pretesting Procedures

 Collaborative approach
 Voluntary participation by respondents
 Site visits
 Post-testing follow-through
 Multiple rounds – test, revise, re-test

13
 Pretesting Methodology

 Qualitative research
 Traditional cognitive interviewing methods
 Adaptations
– Hypothetical questions / probes
– Funnel approach: general specific
– Directive questions / probes
 In-depth unstructured interviews
14
 Six Unresolved or Persistent
Issues for Economic Surveys

 Consequences
 Methodology
 Shortcomings
 Recommendations

15
 Issue #1:
Business survey response is
labor-intensive, burdensome & costly
 Consequences
– Respondents unwilling or unable to
complete a draft questionnaire prior to or
during the pretest interview.
– Researchers unable to use some
traditional cognitive methods
with draft questionnaires

16
 Issue #1:
Business survey response is
labor-intensive, burdensome & costly

 Pretesting methodology
– Hypothetical probes

 Shortcoming
– Hypothetical report ≠ Actual behavior

17
 Issue #1:
Business survey response is
labor-intensive, burdensome & costly
 Recommendations
– Respondent debriefings during field period
• Actual behavior
– “Pilot” test new questionnaires in the field
• Conduct respondent debriefings
• Evaluate data quality

18
 Issue #2:
“Omnibus” surveys require multiple
respondents or data sources
 Consequences
– No single “right” respondent
– Hindrances to gathering data
• Knowledge
• Authority
• Motivation
– Poor data quality

19
 Issue #2:
“Omnibus” surveys require multiple
respondents or data sources

 Pretesting methodology
– Detailed probes about data sources and
communication practices

 Shortcoming
– Unable to meet with or interview
downstream informants
20
 Issue #2:
“Omnibus” surveys require multiple
respondents or data sources
 Recommendations
– New methodologies needed to research ---
• Data locations
• Company policies and practices for exchanging
data internally
• Strategies for internal company communication
– Develop data collection procedures that
encourage, facilitate and support
respondents’ data-gathering
21
 Issue #3:
Mismatch between survey
concepts and company records
 Consequences
– Response error
– Response burden

22
 Issue #3:
Mismatch between survey
concepts and company records

 Pretesting methodology
– Directive questions

 Shortcoming
– Cognitive interviews devolve into a hybrid of
cognitive probes and exploratory questions.
23
 Issue #3:
Mismatch between survey
concepts and company records
 Recommendations
– Involve methodologists in content
development
– Conduct in-depth exploratory interviews to
identify the “business model” for the
concept of interest
• Use directive questions

24
 Issue #4:
Questions request technical or
financial data
 Consequences
– Methodologists ≠ subject area experts
– Methodologists ≠ financial accountants
– Cognitive interviews may not
capture discrepancies between
respondents’ answers and
the intent of the question

25
 Issue #4:
Questions request technical or
financial data
 Pretesting methodology
– Collaboration between methodologists and
experts in subject matter or accounting
 Shortcomings
– Different approaches
• Methodologists: how the respondent answers
questions without help
• Subject area / accounting experts: instruct
respondents on how to report correctly
26
 Issue #4:
Questions request technical or
financial data
 Recommendations
– Improved collaborative methods
• Experts aid methodologists in developing the
interview protocol
• Methodologists report respondent behaviors;
experts assess adequacy of response
• Experts participate in interviews as observers

27
 Issue #5:
Many / most economic surveys
are self-administered

 Consequences
– No interviewer
– Heavy reliance on questionnaire
to convey technical requirements
of the data request Business
Survey

– Need for detailed

instructions

28
 Issue #5:
Many / most economic surveys
are self-administered
 Pretesting Methodology
– Probes on wording, language, visual design
and placement of instructions

 Shortcoming
– Questionnaire design cannot fix every
problem
29
 Issue #5:
Many / most economic surveys
are self-administered
 Recommendation
– Encourage new paradigms in ---
• Questionnaire development
– Top-down and bottom-up
– Identify collectable data with identifiable shortcomings
• Use of data
– Consider context and utility of reported data
– Consider shortcomings in data analysis and interpretation

30
 Issue #6:
Heavy reliance on cognitive
interviewing to identify and correct
questionnaire problems
 Consequence
– Improper or unrealistic expectations
– Difficult to evaluate effectiveness

31
 Issue #6:
Heavy reliance on cognitive
interviewing to identify and correct
questionnaire problems
 Pretesting Methodology
– Multiple iterations of design and testing –
test, revise, re-test
 Shortcomings
– Cognitive testing is ---
• Qualitative methodology
• Somewhat artificial
32
 Issue #6:
Heavy reliance on cognitive
interviewing to identify and correct
questionnaire problems

 Recommendations
– Use multiple complementary methods
• Helps gain confidence in results
– Develop and monitor meaningful objective
measurements of data quality

33
 Conclusion:
10 Years of Pretesting Business
Surveys at the U.S. Census Bureau

 Cognitive testing
– Useful
– Integrated into survey
development

34
 Conclusion:
The Next 10 Years of Pretesting
Business Surveys
 Challenges
– Multiple research methods
– Quantitative evaluation of effectiveness
– Organizational behavior and
the survey response process

35
 Thank you.

 Questions?? Comments??

Diane K. Willimack
U.S. Census Bureau
diane.k.willimack@census.gov
36