Something Interesting About First and Second Order Similarity
Abstract vs. word pairs that are associates but not collocates. Despite
Brief description of what we’ve done.
Keywords: semantic similarity; priming; ???.
Introduction
FS: Point 1 (Corresponding to results from BLP experi-
ment)): Buchanan et al. (2001) showed that the density of
the semantic neighbourhood, as defined by a semantic space
(HAL) measurement, is a significant factor in predicting
performance in the visual word recognition tasks of LDT and
Naming. It is not clear what measure of semantic similarity
is important for the effect. That is, how should the semantic
neighbourhood be defined in order to predict RT? If we
define it in terms of a word’s most paradigmatically related
words, we will get a measure of paradigmatic semantic
density. Conversely, if we define the neighbourhood in terms
of a word’s most syntagmatically related words, we will get
a measure of syntagmatic semantic density. These measures
are substantially different, and it is not clear if both or only
one of the measurements are important factors in predicting
variance in RT. We show that syntagmatic semantic density,
and not paradigmatic semantic density, is an important
semantic-space factor in predicting variance in RT.
FS: Point 2 (Corresponding to results of Experiment 1,
see experiment 1 mean neighbour ranking.png): Within the
priming dataset (SPP), we show that paradigmatically related
pairs are represented as more related with the FOS than the
SOS. Similarly, we show that syntagmatically related pairs
are represented as more related with the FOS than the SOS.
This establishes SOS as a measurement of paradigmatic
similarity and FOS as a measurement of syntagmatic simi-
larity. Establishing this is important because it enables us to
evaluate the effect of paradigmatic and syntagmatic similarity
on paradigmatic categories and syntagmatic categories. It
has been claimed (Jones et al., 2006; Gunther et al., 2016)
that HAL and LSA also represent such relations. However,
this doesn’t seem to be empirically supported. Gunther et
al. (2016) report a very strong correlation between HAL
and LSA similarities in both of their item-level regression
analyses (r = 0.89 and r = 0.91). Similarly, Table 4 from
Jones et al. (2006) paper shows that there is no statistically
signficant difference between word pairs that are collocates
the lack of evidence that HAL and LSA depict paradigmatic
and syntagmatic relationships, respectively, the authors
proceed to claim that HAL is important in accounting for
semantic (i.e. paradigmatic) priming while LSA is important
in accounting for associative priming. By showing that our
semantic space measurements really represent paradigmatic
similarity and syntagmatic similarity, we can really see how
important paradigmatic and syntagmatic similarity are for
both priming of paradigmatically-related words and priming
of syntagmatically associated words.
FS: Point 3, two related subpoints:
• We are the only ones to control for the effect of the
semantic neighbourhood for item-level priming analyses.
This was a methodological issue missed by Gunther et al.
(2016) and Mandera et al. (2016). Without accounting for
this covariate, their analysis is
• We show that both FOS and SOS are important factors in
accounting for priming of paradigmatically related words
(which is surprising, and against findings of Jones et al.
(2006) and Gunther et al. (2016)). We show that both FOS
and SOS are important, overall, in accounting for item-
level RT. This is strengthened by Point 2, where we showed
that FOS is syntagmatic similarity and SOS is paradigmatic
similarity, in that we are the first to show that both paradig-
matic and syntagmatic similarity are important in account-
ing for priming.
What are the main points we’ll make here:
• 1-2 paragraphs: Intro to the general issues
– The study of semantic similarity and why it’s impor-
tant...
– Different types of semantic similarity...
• subsequent paragraphs: what others have done
• final paragraph(s): Our take – what’s new and how do we
show it.
Related Work
Note: This discussion (or parts of it) may be merged
with/included in the intro.
Semantic similarity literature:

Figure 1: Partial Correlations of First Order Semantic Dis-
tance and Second Order Semantic Distance with LDT Reac-
tion Time. Error bars show 95% confidence intervals. Second
Order correlation n.s.
• paper 1 and what’s important about it
LDT and semantic priming literature:
• paper 1 and what’s important about it
Distributional semantics literature:
• paper 1 and what’s important about it
Materials and Methods
• distributional similarity method; corpus for training
• BLP; SPP
• Multiple regression analyses; independent and dependent
variables
Results
Characterization of the Semantic
Neighbourhood
Using simple and effective distributional similarity metrics,
FOS and SOS, our goal is to determine whether increased
densities of both paradigmatic and syntagmatic neighbour-
hoods are facilitatory for lexical processing. We hypothesize
that both paradigmatic density and syntagmatic density would
have significant effects on the LDT RT.
Setup. We evaluate the partial correlations of syntagmatic
density and paradigmatic density with RT results from the
BLP. We compute these correlations on the 3,361 that appear
in all three of the BLP, the SPP, and the vocabulary of the
word2vec model. We partial out the effects of length, log
frequency (calculated from the SUBTL corpus), and ortho-
graphic neighbourhood.
Results. Upon partialling out the effect of the syntagmatic
neighbourhood, Figure 1 shows that there is no significant
correlation between the paradigmatic neighbourhood and the
RT in the LDT task. Further, we see that the semantic neigh-
bourhood as defined by the FOS drives the facilitation of
performance in the task. After partialing out the effect of
FOS, there is no partial correlation between the SOS seman-
tic neighbourhood and RT. Conversely, there is a significant
partial correlation between the FOS semantic neighbourhood
and RT after partialing out the effect of SOS.
Discussion. From this analysis, we replicate the finding from
Buchanan et al. (2001) that semantic processing is indeed
a significant factor in the LDT. However, it is apparent that
the syntagmatic semantic neighbourhood density is facilita-
tory for lexical processing while there is no such evidence for
the paradigmatic semantic neighbourhood density. This re-
sult suggests that, that the single-word lexical decision task,
a word that has a relatively high level of syntagmatic similar-
ity with its closest neighbours, such as ‘school’ or ‘cheese’,
will be recognized more quickly than words that do not (e.g.,
‘theorem’). This contrasts work by Buchanan et al. (2001)
where elements of the syntagmatic neighbourhood as well as
the paradigmatic neighbourhood are conflated (cf. Figure 1
in Buchanan et al. (2001)).
FS: this was a point that we thought might be brought
up by reviewers in our meeting on January 13th. One
possible explanation for this result is that the FOS metric
encapsulates both paradigmatic and syntagmatic similarity,
rendering the SOS metric extraneous. Table 1 clearly shows
that the two metrics are representing different relations. FS:
I haven’t created this table yet. It will just be a table or
figure of the different nearest neighbours for words by FOS
metric vs. SOS metric Roller et al. (2016) and Melamud et
al. (2015) showed that leveraging both FOS, to represent
syntagmatic similarity, and SOS, to represent paradigmatic
similarity resulted in significant increases in performance.
If FOS encapsulated both types of similarity, leveraging
solely FOS would have been sufficient. FS: I could also plot
the Jaccard similarity between the FOS nearest-neighbours
and SOS-nearest-neighbours as a function of the number of
neighbours we consider, if the argument in this paragraph
seems insufficient.
FS: This isn’t a good transition Considering that syntag-
matic neighbourhood density had a significant partial corre-
lation in single-word LDT, we hypothesized that syntagmatic
similarity should also be a significant factor in predicting RT
for the primed LDT.
Paradigmatic and Syntagmatic Similarity in
Semantic Priming
Specifically, our hypothesis is that both syntagmatic similar-
ity should be significant in predicting priming performance
at the item level for words that are paradigmatically-related
and words that are syntagmatically-related. For instance, the
word fork primed by the paradigmatically-related word spoon
should be facilitated by the paradigmatic similarity and the
syntagmatic similarity between the two words (i.e. the ten-
dency for the two words to co-occur in the same sentence).
Conversely, a (prime, target) pair that only has a paradig-
matic relation (e.g. coat and gown) should obtain less fa-
cilitation. There is support for this intuition in the seman-
tic priming literature (Chiarello et al., 1990). Since the dis-
tributional similarity methods we employ, FOS and SOS,
are representative of the two types of similarity, both met-
rics should be significant factors in predicting primed RT for
paradigmatically-related words. So, we delineate the effects
of paradigmatic and syntagmatic similarity on (prime, target)
pairs, within the SPP, that are either syntagmatically-related
or paradigmatically-related.
Paradigmatic and Syntagmatic Relations in the SPP
We first show that FOS and SOS metrics are representative of
paradigmatic and syntagmatic relations, respectively, within
the SPP.
While the interpretations of these metrics are readily ac-
cepted in the NLP literature (Roller & Erk, 2016; Melamud
et al., 2015), it is not apparent that this difference would be
reflected in the smaller SPP dataset.
Setup. We show that there is a statistically significant dif-
ference between FOS and SOS within the SPP. To show this
difference, we selected categories of word pairs in the SPP
that are representative of paradigmatic and syntagmatic rela-
tionships, respectively.
We chose synonyms (e.g., frigid, cold), antonyms (e.g. hot,
cold), superordinates (e.g. sculpture, artwork), and categories
(e.g. actor, actress) as relationships that are representative of
paradigmatic relationships. We chose these categories to be
reflective of paradigmatic similarity since it is clear that ei-
ther word in each respective pair can replace the other in a
sentence. Meanwhile, FPA (e.g. help, wanted) and BPA (e.g.
wanted, help) are, by definition, representative of syntagmatic
relationships. The aforementioned categorizations of associ-
ations are recorded in the SPP.
For each prime word in these categories, we rank a sub-
set of 1,594 target words from the SPP from 1 (most simi-
lar within the DSM) to 1,594 (least similar within the DSM).
The subset of target words is the intersection of words within
our ‘word2vec‘ model vocabulary and all of the 1,661 target
words within the SPP.
For the prime words in the paradigmatic categories, we ex-
pect that the mean ranking of target words will be assessed
higher by the FOS metric than by the SOS metric. Conversely
for the syntagmatic categories, we expect that the mean rank-
ing of target words will be assessed higher by the SOS metric
than by the FOS metric. E.g., given the word ‘frigid’, we ex-
pect the synonym ‘cold’ to be ranked highly by the SOS met-
ric while the phrasal associate ‘temperature’ is given a (rela-
tively) lower rank. We expect an opposite pattern of rankings
by the FOS metric.
Results. Figure 2 shows that this is indeed the case. This
result shows that the difference between FOS and SOS is sta-
tistically significant within the SPP, and that the differences
are in the expected direction.
Discussion. This result shows that the FOS is representa-
tive of syntagmatic similarity while SOS is representative
Figure 2: Target words that are paradigmatically related to
their respective prime are ranked higher by the SOS met-
ric than the FOS metric. Conversely, target words that are
syntagmatically related to their respective primes are ranked
higher by the FOS metric. One-sided paired t-tests show sig-
nificant differences in all categories. FS: Maybe I should add
sample sizes for all of the categories in this figure
of paradigmatic similarity for pairs of words in the SPP.
This result enables us to evaluate whether both paradig-
matic and syntagmatic similarity are significant factors in
accounting for performance in the primed LDT. Specifi-
cally, we can evaluate the effects of both types of similarity
on paradigmatically-related pairs and syntagmatically related
pairs.
Analysis on Related Pairs
Setup.
Results. Figure 3 shows that syntagmatic similarity is
a significant factor for both paradigmatically-related and
syntagmatically-related words. Syntagmatic similarity is im-
portant for words that don’t necessarily have a syntagmatic
association. That is, both degree of substitutability and degree
of co-occurrence are of (independent) importance to lexical
processing time. For instance, it is important (to processing
time) that ‘forever’ and ‘always’ are words that are highly
substitutable and that they also have a greater than chance co-
occurrence ‘forever and always’).
Discussion. This result is significant since other analyses of
priming with distributional models have suggested that mea-
sures of paradigmatic similarity are important in capturing
priming effects of paradigmatic relations while measures of
syntagmatic similarity are not (Jones et al., 2006; Günther et
al., 2016). From the results that we present, it is evident that
both measures of similarity are significant factors in reacting
to target words that were primed with a paradigmatically re-
lated word.
As expected, paradigmatic similarity does not have a sig-
nificant correlation with reaction time for phrasal associates
since phrasal associates are generally not substitutable (e.g.
 Figure 4: Density plots for related words and unrelated words
(blue and orange curves). The reaction time for a related pair
in the SPP is, on average, lower than reaction time for an
unrelated pair (unrelated prime, target). Since both seman-
tic similarity measure relatedness, they should both be im-
portant in depicting the priming effect, after controlling for
covariates. Paired t-tests show significant differences in both
categories: FS: t-test isn’t the right measure for this since the
sample size is so large. I’ll use Cohen’s d instead.
Figure 3: Partial correlation of semantic similarity measures
(FOS and SOS) with RTs for paradigmatically and syntag-
matically related words. Lexical characteristics and single-
word LDT RT for both the prime and the target are partialed
out. In addition, the human measures of similarity of back-
ward and forward association strength are partialled out. FS:
I’ll reproduce this result so that the green marker represents
the effect of partialing out the target covariate instead of the
semantic density, as discussed on January 17th.

(‘honey’, ‘bee’)). Thus, the SOS measure would not be mean-

ingful between two phrasal associates, resulting in a null par-
tial correlation.
We partial out the effects of the single-word LDT since,
as shown in the previous section, the similarity measures are
strongly correlated with single-word reaction time for associ-
ated words Thus, these partial correlations depict the corre-
lation of distributional similarity measures with primed reac-
tion time. Since previous analyses of priming (Günther et al.,
2016; Mandera et al., 2017) did not include this significant
covariate in their analyses, this is the first reliable analysis
that shows distributional similarity measures factor into item-
level regression analyses of priming.
FS: Is this sufficient justification of the small effect size?
Note that while the effect sizes are small, they are commen-
surate with effect sizes of forward and backward associative
strength measures which are known to be a reliable factor of
priming (Hutchison et al., 2008).
Analysis on Related and Unrelated Pairs
The previous section showed that semantic space measures of
similarity are significant predictors of primed RT for related
pairs of words. That is, as semantic similarity increases be-
tween a target and a related prime, RT decreases. Following
Mandera et al. (2017), we conduct a regression analysis on
all of the pairs within the SPP to determine whether both FOS
and SOS were significant factors in predicting reaction time
for related pairs as well as unrelated pairs. The expectation is
that an unrelated pair would have a low semantic similarity,
and consequently a higher reaction time. Figure 4(a) illus-
trates this intuition.
Other models of priming that include distributional simi-
larity (Mandera et al., 2017; Günther et al.,2016) have shown
that distributional similarity is indeed a significant factor in
predicting reaction times for the LDT. There are 3 unresolved
questions following previous analyses: (1) since such analy-
ses only include distributional measurements of paradigmatic
similarity, it is not clear that syntagmatic similarity would be
a significant factor in predicting RT; (2) it is not clear whether
the significant effects of paradigmatic similarity were actually
due to the unique variance captured by syntagmatic similar-
ity; (3) since previous analyses did not control for the effect of
the semantic neighbourhood of the target word, it is unclear
whether a significant effect of any distributional similarity be-
tween prime and target would be found in the first place.
Conclusions
Main contributions:
• contrib 1
• contrib 2
• ...
Discussion points:
• disc 1
• disc 2
• ...
Limitations of the wor:
• lim 1
• lim 2
• ...

References