Professional Documents
Culture Documents
Sqac 081
Sqac 081
JACKLYN MAJNEMER
Massachusetts Institute of Technology, USA
AND
G U S TAV M E I B A U E R
Using fictitious country names in hypothetical scenarios is widespread in experimental international relations research. We
survey sixty-four peer-reviewed articles to find that it is justified by reference to necessary “neutralization” compared to real-
world scenarios. However, this neutralization effect has not been independently tested. Indeed, psychology and toponymy
scholarship suggest that names entail implicit cues that can inadvertently bias survey results. We use a survey experiment
to test neutralization and naming effects. We find not only limited evidence for neutralization, but also little evidence for
systematic naming effects. Instead, we find that respondents were often more willing to support using force against fictitious
countries than even adversarial real-world countries. Real-world associations may provide a “deterrent” effect not captured
by hypothetical scenarios with fictitious country names. In turn, fictionalization may decrease the stakes as experienced by
respondents. Researchers should therefore carefully explain rationales for and expected effects of fictitious country names,
and test their fictitious names independently.
El uso de nombres de países ficticios en escenarios hipotéticos está muy extendido en la investigación experimental en el
ámbito de las RRII. En un estudio de 64 artículos revisados por pares, observamos que esto se justifica con referencia a la
«neutralización» necesaria en comparación con los escenarios del mundo real. Sin embargo, este efecto de neutralización no
se ha comprobado de forma independiente. De hecho, los estudios en el campo de la psicología y de la toponimia sugieren
que los nombres conllevan pistas implícitas que pueden sesgar inadvertidamente los resultados de las encuestas. Utilizamos un
experimento con encuestas para comprobar los efectos de neutralización y de asignación de nombres. Encontramos pruebas
limitadas de neutralización, pero también pocas pruebas de efectos sistemáticos de la asignación de nombres. En cambio,
observamos que los encuestados suelen estar más dispuestos a apoyar el uso de la fuerza contra países ficticios que incluso
contra países antagonistas del mundo real. Las asociaciones del mundo real pueden proporcionar un efecto «disuasorio» que
no se capta en los escenarios hipotéticos con nombres de países ficticios. A su vez, la ficcionalización puede disminuir lo que
está en juego según los encuestados. Por lo tanto, los investigadores deben explicar cuidadosamente las razones y los efectos
esperados de los nombres ficticios de los países, y comprobar sus nombres ficticios de forma independiente.
L’utilisation de noms de pays fictifs dans les scénarios hypothétiques est courante dans la recherche expérimentale en re-
lations internationales. Nous examinons 64 articles vérifiés par des pairs pour conclure qu’elle se justifie par référence à la
nécessité de « neutralisation » par rapport aux scénarios réels. Néanmoins, cet effet de neutralisation n’a pas fait l’objet d’une
évaluation distincte. En effet, les chercheurs en psychologie et toponymie estiment que les noms véhiculent des indications
implicites susceptibles d’influencer les résultats d’enquête. Nous avons recours à une expérience de sondage pour vérifier la
neutralisation et les effets d’appellation. Nous ne recueillons qu’un nombre limité de preuves attestant de la neutralisation,
mais aussi des effets d’appellation systématiques. Nous observons plutôt que les personnes sondées ont plus souvent tendance
à soutenir l’usage de la force à l’encontre de pays fictifs que de pays réels, même ennemis. Les liens avec le monde réel peuvent
générer un effet « dissuasif » non représenté par les scénarios hypothétiques incluant des noms de pays fictifs. La fictionnal-
isation peut, quant à elle, réduire les enjeux envisagés par les personnes sondées. Ainsi, les chercheurs devraient expliquer
précisément les raisons motivant leur utilisation de noms de pays fictifs, et les effets attendus, mais aussi essayer leurs noms
fictifs séparément.
Introduction
Jacklyn Majnemer is a Stanton Nuclear Security Fellow at the Massachusetts Experimental surveys have become an increasingly popular
Institute of Technology’s Security Studies Program. Her research focuses on nu- tool in IR research. They are used widely to gauge atti-
clear security and alliance politics. tudes, simulate decisional dynamics, and elucidate psycho-
Gustav Meibauer is an Assistant Professor in International Relations at the De- logical and behavioral concepts in political science and
partment of Political Science, Radboud University Nijmegen. His research focuses
on international relations theory and foreign policy decision-making.
Authors’ note: The authors contributed equally to this article. They are thank- Department’s Security/Statecraft and Theory Colloquiums for useful comments
ful to Gokhan Ciflikli, Anna Getmansky, Jörg Meibauer, Paola Solimena, Haley and feedback. The data underlying this article are available on the ISQ Dataverse,
Swedlund, Peter Trubowitz, and Reinout van der Veer, as well as the LSE IR at https://dataverse.harvard.edu/dataverse/isq.
Majnemer, Jacklyn, and Gustav Meibauer. (2023) Names from Nowhere? Fictitious Country Names in Survey Vignettes Affect Experimental Results. International Studies Quarterly,
https://doi.org/10.1093/isq/sqac081
© The Author(s) (2023). Published by Oxford University Press on behalf of the International Studies Association. This is an Open Access article distributed under the terms of the
Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium,
provided the original work is properly cited.
2 Fictitious Country Names Affect Experimental Results
international relations (IR). They underpin work on peace affect the results or indeed offer alternative (better)
and conflict, regime theory, foreign policy decision-making, explanations (McDermott 2011b). This extends to experi-
public opinion, and media, as well as political psychology mental vignettes.
(e.g., Beer, Healy, and Bourne 2004; Hudson and Butler Vignette features (including wording, question order, and
2010; Mintz, Yang, and McDermott 2011; McDermott 2011a; survey length) can bias responses regardless of the underly-
Jensen, Mukherjee, and Bernhard 2014). Frequently, such ing characteristics ostensibly being measured. Even minor
research employs experimental surveys with vignettes de- changes can change the perceived meaning of questions or
scribing hypothetical scenarios that involve fictitious coun- vignettes (Pasek and Krosnick 2010, 43). Also, experimen-
tries (e.g., Kerejistan) or abstracted countries (e.g., “a coun- tal surveys are cognitively taxing. This can lead respondents
try”). We have identified sixty-four peer-reviewed articles in to knowingly or unknowingly use “shortcuts,” including pre-
political science and IR (published between 1971 and 2021) conceived notions, schemas, or prejudices, to guide their
with a total of eighty-six unique fictitious countries in exper- responses, thereby effectively reaching outside of the ex-
imental vignettes (not including abstract names, e.g., “the periment (Groves et al. 2011, 219). The vignette’s features
state”). should not inadvertently incentivize these shortcuts to an
(Mintz and Geva 1993, 491). Around the original dyad of This first hypothesis tests whether respondents care about
Gorendy and Raggol, consecutive authors added Winmont, names provided to them in vignettes in the first place. It
Minalo, Kerzoune, Purportes, Vermaid, and Zorka (Mintz could be the case, otherwise, that preferences about using
et al. 1997; Geva and Hanson 1999; Redd 2002; Christensen military forces are fixed independent of provided context
and Redd 2004; Mintz 2004; Rubenzer and Redd 2010; (e.g., strict pacifism) or that respondents may not pay atten-
Horowitz and Redd 2017, 2018). tion to names at all, or as much as other information also
provided to them.
Neutralization H2 : A change from a real-world country name to a fictitious or ab-
Most authors do not explain why they use fictitious country stract country name will affect respondents’ willingness to use force.
names.1 Boettcher discusses the unintended consequences H3 : Respondents’ willingness to attack a fictious/abstract country
of framing in vignettes, but uses fictitious countries without will lie between their willingness to attack a positively associated
further explanation (Boettcher 1995, 577). Others mention real-world country and a negatively associated real-world country.
their scenario’s real-world basis (Astorino-Courtois 2000;
think are) more like their own, as well as according value below) because of toponymic, lexicographic, and/or sound-
judgments. Consider Westria that entails information about symbolic cues.
a geographical location, but could also be associated with
“the West” as a sociocultural realm (Brooks and Valentino H4 : An increase in toponymic cues evoking negative value judg-
2011). In turn, based on toponymic knowledge of English ments should increase respondents’ willingness to use force.
speakers, Drakhar (Keller and Yang 2016), Nizwar (Davies
and Johns 2016), Bachran (Falomir-Pichastor et al. 2012), H5 : An increase in lexicographic cues evoking negative value judg-
Kibagho (Terris and Tykocinski 2016), etc., may be associ- ments should increase respondents’ willingness to use force.
ated with geographic and/or sociocultural spaces in Africa H6 : An increase in sound-symbolic cues evoking negative value
and the Middle East. Kuzeya (Sirin 2011) or Abazie might judgments should increase respondents’ willingness to use force.
connote Eastern European or Central Asian countries (in-
tentionally so in Falomir-Pichastor et al. 2012).
Lexicographic cues concern the degree of familiarity a
Methods
name evokes given the respondent’s name lexicon. While
Table 1. Overall approval of use of force (1 = lowest, 7 = highest) across all treatment groups
• “Iran” (treatment group size N = 246). Iran is salient and Table 2. Mean approval for use of force, standard deviations, and
schema-consistent to the scenario (Brutger et al. 2022, number of responses by treatment
14–15), given that the vignette is based on this real-world
Country Average approval Standard deviation Frequency
SS df MS Observations 1,613
Model 675.551382 10 67.5551382 F (10,1602) 20.13
Residual 5,377.49884 1,602 3.35674085 Probability > F 0.0000
Total 6,053.05022 1,612 3.75499393 R2 0.1116
Adjusted R2 0.1061
Root MSE 1.8321
Canada
Celesta
Drakhar
Kerejistan
Minalo
The country
−1.5 −1 −.5 0 .5 1
Figure 1. Response effect estimates and confidence intervals of treatment groups mapped against the baseline “Iran” (vertical
line at position “0”).
differ compared to Kerejistan, Minalo, and “the country.” article text or the respective appendices, where available)
Importantly, the fictitious country names did not perform that fictitious country names actually provided a neutraliza-
significantly differently compared to each other in the way tion effect.6 Instead, they relied on two untested assump-
we might expect based on naming effects literature (H4 –6 ). tions: that all fictitious country names are similarly neutral-
Of our experiment’s fictitious countries, Kerejistan per- izing and that there is no biasing effect from the fictitious
formed most similar to Iran (compared to other fictitious name itself. Our experiment seeks to test these assumptions
names). Beyond Kerejistan, we did not find evidence that to investigate what effect, if any, fictionalized or abstract
variation between different fictitious country names has an names have on respondents’ willingness to use force.
effect on survey responses. We find only limited support for the neutralization hy-
pothesis. The example of Kerejistan (as compared to the
Iran baseline) already shows that not all fictitious names
Discussion have a uniform effect. It also runs counter to the neutraliza-
Previous studies that employed fictitious or abstract country tion hypothesis that both fictionalized and abstract names
names did often not provide an explicit reason for doing
so. Even among those articles that did, usually based on in- 6
With the possible exception of Dill and Schubiger, who show that the name
ternal validity concerns, none demonstrated (in either the “Esor” did not drive their results (Dill and Schubiger 2021).
JACKLYN MAJNEMER AND G U S TAV M E I B A U E R 7
Table 4. Kruskal–Wallis equality-of-populations rank test and Dunn–Bonferroni post hoc test (adjusted)
Celesta −9.252439
(0.0000)
Drakhar −9.395682 0.044487
(0.0000) (1.0000)
Iran −6.741038 2.955401 2.976137
(0.0000) (0.0328) (0.0306)
Kerejistan −7.934474 1.743650 1.736797 −1.270894
(0.0000) (0.8528) (0.8654) (1.0000)
Minalo −9.094823 0.763850 0.734430 −2.368092 −1.067978
(0.0000) (1.0000) (1.0000) (0.1877) (1.0000)
The country −8.412077 1.101630 1.080113 −1.890160 −0.642215 0.391386
(0.0000) (1.0000) (1.0000) (0.6167) (1.0000) (1.0000)
increase willingness to use force. We also find only lim- realistic-ness of a country name is (at least partially) a func-
ited evidentiary support for systematic naming effects (with tion of perceived toponymic plausibility and lexicographic
the exception of Kerejistan), that is, differences between familiarity. Note that this could also lead respondents to
fictitious names induced by lexicographic, toponymic, or consider real-life country names unrealistic, depending on
sound-symbolic cues. While Celesta, Drakhar, and Minalo their individual knowledge of country names, geography,
prompted significantly different responses compared to the and their respective native languages. While we have not
baseline Iran, the different fictitious names were not signif- separately asked our respondents to rate the realistic-ness
icantly different from each other. This matters because they of the provided name or scenario, it seems likely that differ-
were selected based on assumed difference of embedded ent fictitious names connote realistic-ness to different de-
cues. Even below significance thresholds, the order and di- grees. Consider the difference between Celesta, Drakhar,
rection in which the fictitious names differ from the baseline and Minalo on the one hand, and Kerejistan on the other—
do not support the naming effect hypotheses. respondents may more readily identify the former as ficti-
Notably, our findings suggest that fictitious and abstract tious, while the latter may challenge respondents not famil-
names increase participants’ willingness to support the use iar with the geography of central Asia. In turn, perceived
of force compared to real-world (or realistic) counterparts. realistic-ness influences their responses.
This holds even compared to schema-consistent real world, Indeed, our results indicate that Kerejistan does not have
adversarial, and/or pariah states. It questions whether the significantly different effects on participant responses com-
neutralization effect actually neutralizes—or whether in- pared to Iran. Kerejistan may evoke connotations that are,
deed there is a “fictionalization effect” that may lead schol- on average and for British respondents, similar to those
ars to overestimate respondents’ willingness to use force evoked by Iran. The close lexicographic and toponymic as-
in hypothetical scenarios involving fictitious or abstract sociation of Kerejistan with, for example, Afghanistan seems
countries. Our results suggest that the more clearly ficti- likely to evoke value judgments that overlap with those
tious a country name, the easier to condone attacking it— respondents make regarding Muslim, Middle-Eastern, or
fictionality and its perceived costlessness can therefore em- autocratic states, despite the considerable real-world differ-
bolden respondents to provide more aggressive responses. ences between Iran and Afghanistan. This possible informa-
These results point to the relevance of perceived realistic- tion equivalence problematizes the neutralization hypothe-
ness: the more “real” a country name sounds to respon- sis: not all fictitious country names are similarly neutraliz-
dents, the weaker the fictionalization effect. In particular, ing. It implies the necessity for experimental researchers to
there seems to be a deterrent effect associated with realistic- carefully justify or pilot fictitious country names and
ness, for example, of being able to imagine more easily the scenarios in advance of collecting results (also see Steiner,
consequences associated with attacking Iran, especially bar Atzmüller, and Su 2017).
any additional information that “fills out” the scenario. The
8 Fictitious Country Names Affect Experimental Results
Among the treatments employed, “the country” is the breakdown of real-world associations induced by fictional-
most abstract fictionalization—it does not offer a name but ization seems to cognitively decrease the stakes as experi-
a placeholder. In terms of results, it constitutes a borderline enced by respondents; in turn, realistic or plausible names
case. While not significantly different from Iran in our cross- may have a deterrent effect. The fictionalization effect con-
treatment comparison above, participants evaluated “the stitutes an important avenue for future research, including
country” similar to the other fictitious names. In principle, with an eye to generalization beyond the characteristics of
its lexicographic, toponymic, and sound-symbolic “empti- the population here surveyed.
ness” could induce respondents to think about which coun- Notably, our results cannot currently tell us whether,
try could be meant—which, given the salience of Iran to for example, the results of Tomz and Weeks’ study would
both the context in which our experiment was conducted differ (in what way) had they used any particular, differ-
and the hypothetical vignette, could have led respondents ent name in their vignette. This is partially a function of
to infer that the vignette meant Iran. Instead, “the country” our experimental design: in particular, future research may
seems to have increased the willingness to support the use of seek to investigate the strength of the fictionalization effect
force in similar ways to the other fictitious names. From this when experimental vignettes include additional informa-
Conclusion
Funder Information
We challenge the assumption that fictitious names are
The International Relations Department at the London
an unproblematic means of removing respondent bias in
School of Economics and Political Science provided funding
experimental IR research. What name is chosen for the
to conduct the experimental survey underlying this article.
vignette matters—and experimental researchers need to fac-
tor this into their survey design (Dafoe, Zhang, and Caughey
2018; Renshon, Dafoe, and Huth 2018). The most com- References
mon rationale for the use of fictitious or abstract coun- ALBERT, MATHIAS, OLIVER KESSLER, AND STEPHAN STETTER. 2008. “On Order and
try names within experimental surveys suggests that they Conflict: International Relations and the ‘Communicative Turn’.” Re-
provide what we label the “neutralization effect,” allowing view of International Studies 34 (1): 43–67.
respondents to provide their views on the phenomena of ASTORINO-COURTOIS, ALLISON. 2000. “The Effects of Stakes and Threat on
interest while avoiding that their personal opinions of real Foreign Policy Decision-Making.” Political Psychology 21 (3): 489–509.
countries and/or available background information bias https://doi.org/10.1111/0162-895X.00200
BAELE, STEPHANE J., TRAVIS G. COAN, AND OLIVIER C. STERCK. 2018. “Security
their responses. Testing this assumption, we find limited sup-
through Numbers? Experimentally Assessing the Impact of Numerical
port for the neutralization effect, which indicates that previ- Arguments in Security Communication.” The British Journal of Politics
ous experimental results may not be as robust as previously and International Relations 20 (2): 459–76.
assumed. We also find only limited support for systematic BALMAS, MEITAL. 2018. “Tell Me Who Is Your Leader, and I Will Tell You Who
naming effects (beyond perceived toponymic closeness in You Are: Foreign Leaders’ Perceived Personality and Public Attitudes
the example of Iran and Kerejistan). We find an overall ef- toward Their Countries and Citizenry.” American Journal of Political Sci-
fect of fictitious and abstract country names on responses ence 62 (2): 499–514.
of increasing willingness to use force. However, these re- BEER, FRANCIS A., ALICE F. HEALY, AND LYLE E. BOURNE. 2004. “Dynamic Deci-
sults seem to be driven by a different kind of logic than the sions: Experimental Reactions to War, Peace, and Terrorism.” In Ad-
hypothesized naming effects. vances in Political Psychology, edited by Margaret G. Hermann, 139–67.
Oxford: Elsevier.
Instead, complementing previous literature (Dafoe,
BEER, FRANCIS A., ALICE F. HEALY, GRANT P. SINCLAIR, AND LYLE E. BOURNE. 1987.
Zhang, and Caughey 2018; Brutger et al. 2022), we find ev- “War Cues and Foreign Policy Acts.” American Political Science Review 81
idence suggestive of a fictionalization effect. Respondents (3): 701–15.
are more willing to use violence against fictitious countries BEER, FRANCIS A., JEFFREY F. RINGER, GRANT P. SINCLAIR, ALICE F. HEALY, AND
regardless of the name employed, more so even than against LYLE E. BOURNE. 1992. “Ranking International Cooperation and Con-
real-world countries that are perceived as adversarial. The flict Events.” International Interactions 17 (4): 321–48.
JACKLYN MAJNEMER AND G U S TAV M E I B A U E R 9
BEER, FRANCIS A., GRANT P. SINCLAIR, ALICE F. HEALY, AND LYLE E. BOURNE. 1995. GROVES, ROBERT M., FLOYD J. FOWLER, MICK P. COUPER, JAMES M. LEPKOWSKI,
“Peace Agreement, Intractable Conflict, Escalation Trajectory: A Psy- ELEANOR SINGER, AND ROGER TOURANGEAU. 2011. Survey Methodology. Lon-
chological Laboratory Experiment.” International Studies Quarterly 39 don: Wiley.
(3): 297–312. GUETZKOW, HAROLD. 1959. “A Use of Simulation in the Study of Inter-Nation
BOETTCHER, WILLIAM A. 1995. “Context, Methods, Numbers, and Words: Relations.” Behavioral Science 4 (3): 183–91.
Prospect Theory in International Relations.” Journal of Conflict Resolu- HAINMUELLER, JENS, DOMINIK HANGARTNER, AND TEPPEI YAMAMOTO. 2015. “Vali-
tion 39 (3): 561–83. dating Vignette and Conjoint Survey Experiments against Real-World
BOURNE, LYLE E., GRANT P. SINCLAIR, ALICE F. HEALY, AND FRANCIS A. BEER. Behavior.” Proceedings of the National Academy of Sciences 112 (8): 2395–
1996. “Peace and Gender: Differential Reactions to International 2400.
Treaty Violations.” Peace and Conflict: Journal of Peace Psychology 2 (2): HEALY, ALICE F., JOSHUA M. HOFFMAN, FRANCIS A. BEER, AND LYLE E. BOURNE.
143–49. 2002. “Terrorists and Democrats: Individual Reactions to International
BRIGHT, WILLIAM. 2003. “What Is a Name? Reflections on Onomastics.” Lan- Attacks.” Political Psychology 23 (3): 439–67.
guage and Linguistics 4 (4): 669–81. HEINRICH, TOBIAS, YOSHIHARU KOBAYASHI, AND TIMOTHY M. PETERSON. 2017.
BRODY, RICHARD A. 1963. “Some Systemic Effects of the Spread of Nuclear “Sanction Consequences and Citizen Support: A Survey Experiment.”
Weapons Technology: A Study through Simulation of a Multi-Nuclear International Studies Quarterly 61 (1): 98–106.
Future.” Journal of Conflict Resolution 7 (4): 663–67. HOPMANN, P. TERRENCE, AND CHARLES WALCOTT. 1976. “The Impact of Interna-
MCDONALD, JARED. 2020. “Avoiding the Hypothetical: Why ‘Mirror Experi- RUBENZER, TREVOR, AND STEVEN B. REDD. 2010. “Ethnic Minority Groups and
ments’ Are an Essential Part of Survey Research.” International Journal US Foreign Policy: Examining Congressional Decision Making and
of Public Opinion Research 32 (2): 266–83. Economic Sanctions.” International Studies Quarterly 54 (3): 755–77.
MCLEAN, ELENA, AND DWIGHT ROBLYER. 2017. “Public Support for Economic SCHAFER, MARK. 1997. “Images and Policy Preferences.” Political Psychology 18
Sanctions: An Experimental Analysis.” Foreign Policy Analysis 13 (1): (4): 813–29.
233–54. ———. 1999. “Cooperative and Conflictual Policy Preferences: The Effect of
MEDWAY, DOMINIC, AND GARY WARNABY. 2014. “What’s in a Name? Place Brand- Identity, Security, and Image of the Other.” Political Psychology 20 (4):
ing and Toponymic Commodification.” Environment and Planning A: 829–44.
Economy and Space 46 (1): 153–67. SCHWARTZ, JOSHUA A., AND CHRISTOPHER W. BLAIR. 2020. “Do Women Make
MINTZ, ALEX. 2004. “Foreign Policy Decision Making in Familiar and Unfa- More Credible Threats? Gender Stereotypes, Audience Costs, and Cri-
miliar Settings: An Experimental Study of High-Ranking Military Offi- sis Bargaining.” International Organization 70 (4): 872–95.
cers.” Journal of Conflict Resolution 48 (1): 91–104. SIRIN, CIGDEM V. 2011. “Examining the Effects of Political Information and In-
MINTZ, ALEX, AND NEHEMIA GEVA. 1993. “Why Don’t Democracies Fight Each tervention Stages on Public Support for Military Interventions: A Panel
Other? An Experimental Study.” Journal of Conflict Resolution 37 (3): Experiment.” Acta Politica 46 (3): 261–93.
484–503. STEINER, PETER M., CHRISTIANE ATZMÜLLER, AND DAN SU. 2017. “Designing Valid
MINTZ, ALEX, NEHEMIA GEVA, STEVEN B. REDD, AND AMY CARNES. 1997. “The Ef- and Reliable Vignette Experiments for Survey Research: A Case Study