Grammaticality Judgments as Linguistic Evidence

Methodological Guidelines for Measuring Grammaticality Brian Murphy
Language, Interaction and Computation Lab Centre for Mind/Brain Sciences (CIMeC) University of Trento

ESSLLI 2009, Bordeaux

Brian Murphy (CIMeC)

Grammaticality: Methods Guidelines

ESSLLI 2009

1 / 23

Course Outline

Notions of Grammaticality, and Current Practice in Linguistics Others Sources of Linguistic Evidence Scales for Measuring Grammaticality

Methodology for Eliciting Judgements of Grammaticality
Objectives of Experimentation Ideal and practical choices Software Materials Statistics
Theoretical Implications

Brian Murphy (CIMeC)

Grammaticality: Methods Guidelines

ESSLLI 2009

2 / 23

Objectives of Experimentation I Validity: are we measuring what we aim to measure? Reliability: are your data purely the result of chance? Replicability: do other researchers have enough information to `rerun' your experiment? Generalizability: do your results apply to other sentences/speakers than the ones you Otherwise. a eld is likely to spend a lot of time following up bad leads and chasing its own tail Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 3 / 23 .

. intuitions are most often the best measure we have of grammaticality but we have to be aware of strategising. misinterpretation of instructions. 1983] Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 4 / 23 .Objectives of Experimentation II Validity: are we measuring what we aim to measure? as we have seen. subconscious biases [Armstrong et al.

Objectives of Experimentation III Reliability: are your data purely the result of chance? this is what inferential statistics are for Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 5 / 23 .

Objectives of Experimentation IV Replicability: do other researchers have enough information to `rerun' your experiment? ask journals about online appendices. or put supplementary materials on your website describe your methods very briey Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 6 / 23 .

Objectives of Experimentation V Generalizability: do your results apply to other sentences/speakers than the ones you tested? are your materials representative of the structure and language variety you're interested? are your speakers representative of the speech community you're interested in? how are you dealing with confounds? Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 7 / 23 .

preferably 20 No single participant should see the same sentence twice The informant should not know your hypothesis You know the informant's age. profession. educational background. handedness.The Ideal Experiment I All other things being equal (and of course they never are) you would like your experiment to have the following characteristics: As many informants as possible. minimum 10 As many judgements for each sentence as possible.. Test items should be interspersed with llers Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 8 / 23 .. dialect. . minimum 10.

.The Ideal Experiment II Materials should be authentic Context should be supplied to control interpretation Presentation order should be randomised Each informant should see a similar proportion of each type of sentence (counter-balancing) The experimenter should not be present during the experiment The informants should not be known to the experimenter . and probably some other things I've forgotten about Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 9 / 23 ..

. for example .What Matters More? I Featherston 2009: Relax. referential abstractness (eg too many pronouns) unclear meaning low accessibility of intended interpretation Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 10 / 23 .... lean back. . and be a linguist plausibility of content of experimental materials sentence or phrasal length and complexity.

What Matters More? II Brian's pet list: Quantity of sentences and informants Clear. naturalistic instructions Training/calibration Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 11 / 23 .

What Matters More? III Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 12 / 23 .

. Symbols represent the following sentence stimuli: squares: . it may take a little longer diagonal crosses: I was surrounded by an endless sorrow diamonds: Did anyone order me a plain cheese triangles: Throw the idol me crosses: Stop thinking sex about Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 13 / 23 . the Y axis the mean score up to that point..What Matters More? IV The X axis represents the number of responses.

.. and handedness of subjects a degree of linguistics training among subjects (but . dialogues) carrying out experiments over the net or on paper the precise methodology. questions..What Matters Less? I Featherston social. educational and professional status of subjects dialect background of subjects (except .) frequency in lexis (as long as extremes are avoided) speed of response required structure type being tested (statements. age.) the sex.. as long as it asks subjects about their receptive responses to examples Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 14 / 23 .

very accurate timing. statistics for free.Software I Lab-based (roughly in order of ease of use): E-Prime. kind-of-free (toolbox is free. code based. very accurate timing. expensive (see if you can use a copy from you Psychology department) Psychotoolbox for Matlab: aimed at vision researchers. windowing-based. many universities have campus licenses for Matlab) PyEPL. statistics for free. no particular support for linguistics. code/windowing based. no inbuilt stats. no particular support for linguistics. both for Python: aimed at vision researchers. need scripting skills. PsychoPy. need scripting skills. free Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 15 / 23 . no particular support for linguistics. very accurate timing. can build your rst experiment in an hour with no programming skills. Presentation: aimed at psychologists.

data/informant/session management. very easy. data/informant/session management. summary of results. web-form/conguration-le based. free WebExp for Java: aimed at linguists and psychologists. has to be installed on server. free Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 16 / 23 . conguration le based. no timing.Software II Internet-based (work in labs too): Linguist-GRID: aimed at linguists. web-form based. has to be installed on webserver. DEFUNCT (but on source-forge still)? LimeSurvey: aimed at social sciences. perfectly adequate timing. user-access control. not-quite-platform-independent.

Software III Paper/Mail-based. Myers. which is treated as a factor (helps control for some processing eects) Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 17 / 23 . 2009]: `small-scale experimental syntax': Experimental sentences only (no llers) Only as many sentence sets as are needed for statistical validity (around 5) Only as many speakers as are needed for statistical validity (around 7) All speakers get all sentences (no counterbalancing of sentence lists) Binary YES/NO judgments Maximum of two binary factors Random sentence order. 2007. for your inner-Luddite: MiniJudge [Myers.

umd.umiacs..*"] lse. authentic materials are easy to nd Google: gave * the * Yahoo API: (gave * the *) OR (send * the *) OR (send * * a *) .org.uk (Wacky. 2009]): "give|gave|given" [pos="DT"] [pos="N.uk: g[i|a]v[ing|e|es|en] * to * corpus.Materials Nowadays..ac. www.webcorp..leeds.edu: (S1 (VP (VB give) NP (PP (TO to) NP))) Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 18 / 23 .*"] "to" [pos="N. [Baroni et al.

probably the most important software package in science (www. and linear correlation. and their non-parametric alternatives Mann-Whitney and rank correlation Ignore theory of scales [Stevens.org) Do a basic practical statistics course at your department of statistics or psychology Try to understand the t-test.Statistics I Download and install R. 1946] normality of the sampling distribution only thing that matters so test more sentences to improve use parametric statistics Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 19 / 23 .cran.

Statistics II Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 20 / 23 .

Software Wishlist I Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 21 / 23 .

and Current Practice in Linguistics Others Sources of Linguistic Evidence Scales for Measuring Grammaticality Methodology for Eliciting Judgements of Grammaticality Theoretical Implications Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 22 / 23 .What Comes Last Notions of Grammaticality.

References I Armstrong. Ferraresi. E. (2009). S. 103:677680. (1983). R. J. and Gleitman.. M. 12(2):175194.. Lingua. The design and analysis of small-scale syntactic judgment experiments. S. S. On the theory of scales of measurement. What some concepts might not be. (1946). S. Bernardini. Stevens. and Zanchetta. L.. 119:425444. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. H.. J. Myers. Cognition. Myers. Language Resources and Evaluation. 13:263308. (2009). Baroni. International Journal of Computational Linguistics and Chinese LanguageProcessing. A. L. Brian Murphy (CIMeC) Grammaticality: Methods Guidelines ESSLLI 2009 23 / 23 . (2007).. Science. Gleitman. MiniJudge: Software for small-scale experimental syntax.

Sign up to vote on this title
UsefulNot useful