Article

Cortical reactivations predict future sensory

responses

https://doi.org/10.1038/s41586-023-06810-1 Nghia D. Nguyen1, Andrew Lutas2,3, Oren Amsalem2, Jesseba Fernando2,

Andy Young-Eon Ahn2, Richard Hakim1,4, Josselyn Vergara5, Justin McMahon5,
Received: 10 October 2022
Jordane Dimidschstein5, Bernardo L. Sabatini1,4 & Mark L. Andermann1,2,6 ✉
Accepted: 31 October 2023

Published online: xx xx xxxx

Many theories of offline memory consolidation posit that the pattern of neurons
Check for updates activated during a salient sensory experience will be faithfully reactivated, thereby
stabilizing the pattern1,2. However, sensory-evoked patterns are not stable but,
instead, drift across repeated experiences3–6. Here, to investigate the relationship
between reactivations and the drift of sensory representations, we imaged the
calcium activity of thousands of excitatory neurons in the mouse lateral visual cortex.
During the minute after a visual stimulus, we observed transient, stimulus-specific
reactivations, often coupled with hippocampal sharp-wave ripples. Stimulus-specific
reactivations were abolished by local cortical silencing during the preceding stimulus.
Reactivations early in a session systematically differed from the pattern evoked by
the previous stimulus—they were more similar to future stimulus response patterns,
thereby predicting both within-day and across-day representational drift. In particular,
neurons that participated proportionally more or less in early stimulus reactivations
than in stimulus response patterns gradually increased or decreased their future
stimulus responses, respectively. Indeed, we could accurately predict future changes
in stimulus responses and the separation of responses to distinct stimuli using only
the rate and content of reactivations. Thus, reactivations may contribute to a gradual
drift and separation in sensory cortical response patterns, thereby enhancing sensory
discrimination7.

In the absence of salient ongoing sensory stimuli, the brain may instead by minute-long interstimulus intervals during which reactivations
learn from previous experiences by repeatedly replaying or reacti- should occur1,23,28.
vating neural patterns that were active during past experiences1,2,8–10.
Such reactivations involve temporally condensed, hypersynchronous
events that occur during quiet waking and sleep1,2,8–11. First observed Distributed reactivations across the cortex
and most commonly studied in the hippocampus, reactivations have Awake, head-fixed mice (n = 8) passively viewed 64 presentations per
also been observed in the amygdala, prefrontal cortex, visual cortex day of each of two visual stimuli across daily sessions during cellular
and elsewhere11–23. imaging29,30 (Fig. 1a; stimulus 1 (S1) and stimulus 2 (S2), presented in a
Reactivations, by definition, are patterns of activity that are simi- random order; 2 s duration; 9 ± 1 sessions per mouse, mean ± s.e.m.).
lar to those that occurred during recent experiences1,24. However, in In contrast to conventional sensory mapping protocols, each presenta-
part due to the limited recording of tens to hundreds of neurons in tion was followed by a long 58 s inter-trial interval (ITI) to investigate
previous studies, the extent to which reactivations are faithful cop- possible offline reactivations of stimulus-evoked response patterns
ies of activity patterns that occurred during previous experiences (Fig. 1a). To track activity in glutamatergic neurons throughout the
remains unclear25–27. Given that stimulus response patterns gradu- lateral visual cortex, we performed multiple viral injections of a
ally change across trials (termed representational drift3–6), stimulus genetically encoded calcium indicator (Cre-dependent expression
reactivations might track these changing response patterns or might of jGCaMP7s31 in Emx1-cre32 mice; Fig. 1b). We first defined visual
instead more closely resemble future responses to the same stimulus. cortical areas using a brief epifluorescence imaging protocol33,34.
To more accurately compare the content and dynamics of stimulus We next combined sensory stimulation with wide-field two-photon
response patterns and offline reactivations across trials, we recorded Ca2+ imaging to simultaneously record the activity of thousands of
and tracked the activity of approximately 6,900 neurons simulta- neurons (6,878 ± 118 neurons per session, mean ± s.e.m., 72 sessions
neously in the lateral visual cortex across days while mice passively from 8 mice) across three planes within layer 2/3 of four lateral visual
viewed well-controlled presentations of identical stimuli, separated cortical areas (Fig. 1b). We focused the analyses on stimulus-driven
1
Program in Neuroscience, Harvard University, Boston, MA, USA. 2Division of Endocrinology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA. 3Diabetes,
Endocrinology and Obesity Branch, National Institutes of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, Bethesda, MD, USA. 4Howard Hughes Medical Institute,
Department of Neurobiology, Harvard Medical School, Boston, MA, USA. 5Stanley Center for Psychiatric Research, Broad Institute of Harvard and MIT, Cambridge, MA, USA. 6Department of
Neurobiology, Harvard Medical School, Boston, MA, USA. ✉e-mail: manderma@bidmc.harvard.edu

Nature | www.nature.com | 1
Article
a Stimulus 1
b AAV(PHP.eb)-syn-FLEX-jGCaMP7s c
Emx1-cre (glutamatergic neurons)
No outcome 1 Stimulus-1-
Plane 1 preferring

Normalized deconvolved
neurons 0.12
Stimulus 2 >20 μm

Stimulus-driven

Ca2+ activity
No outcome V1 Plane 2

neurons
>20 μm
or
Plane 3
Stimulus (2 s) Grey screen (58 s) LM
Trial P ~6,900 neurons
structure: LI Stimulus-2-
POR 0
Baseline Stimulus presentations preferring
M 1,500 neurons
Session 0.5 mm Example
structure: … A P subregion 0 2 0 2
–3 93 Time relative to stimulus onset (s)
0.5 h 2.5 h
Azimuth (°) L
d e 1.0 **
Reactivation 1 More

Max. normalized
aroused
probability 0

pupil area
1
0.5

Less
aroused
0
Top –20 –10 0 10 20
S1-preferring
neurons **

Pupil area (ΔA/A)

0

–0.1
200
1 –0.2
–20 –10 0 10 20
f
*

Hippocampal ripple-
0.50

band power (s.d.)

Top
S2-preferring
neurons 0.25

–20 –10 0 10 20
200
Time relative to
10 s reactivation onset (s)
g h
LM LM
NS NS
(normalized deconvolved

0.8 (normalized deconvolved 0.8

Reactivation activity
Ca2+ per second)

Ca2+ per second)

Stimulus activity

0.6 0.6
LI P LI P
0.4 0.4

POR 0.2 POR 0.2

Normalized deconvolved 0 Normalized deconvolved 0
Ca2+ activity per second 0.8 LI POR P LM Ca2+ activity per second 0.8 LI POR P LM

Fig. 1 | Distributed stimulus reactivations in the lateral visual cortex
during quiet waking. a, Two-photon imaging in awake head-fixed mice during
repeated, passive presentation of S1 or S2. The stimuli (2 s duration, 58 s ITI)
were presented in a random order for 2.5 h. b, Cre-dependent jGCaMP7s
expression in glutamatergic neurons through local injections across the visual
cortex in Emx1-cre mice (top). Bottom, epifluorescence retinotopic mapping
identified visual cortical areas: primary visual cortex (V1), lateromedial (LM),
postrhinal (POR), laterointermediate (LI) and posterior (P). Simultaneous
wide-field two-photon Ca2+ imaging of approximately 6,900 neurons across
3 depths within layer 2/3 (white rectangle). Bottom right, example magnified
subregion. Scale bars, 0.5 mm (left) and 0.2 mm (right). A, anterior; P, posterior;
L, lateral; M, medial. c, Trial-averaged, deconvolved peri-stimulus Ca2+ activity
from an example session sorted by stimulus preference. d, Raster plot of
ongoing deconvolved Ca2+ activity of the top S1-driven and S2-driven neurons
for three example trials. We used multinomial logistic regression (Methods and
Extended Data Fig. 1d) to decode whether synchronous patterns during the ITI
resembled the S1-evoked pattern (S1 reactivation probability; green) or the
S2-evoked pattern (S2 reactivation probability; red). e, The mean pupil area
(normalized to the maximum across the session (top) and the relative change
(bottom)) surrounding the onset of reactivations. n = 8 mice. Statistical analysis
was performed using two-tailed paired t-tests; P = 0.0039 (top), P = 0.0019
(bottom). f, Same analysis as in e but for ripple-band power of the local field
potential measured in the dorsal hippocampal CA1. n = 5 mice. Statistical
analysis was performed using a two-tailed paired t-test; P = 0.011. g, Example
stimulus-evoked response of stimulus-driven neurons across the lateral visual
cortex (left). Right, mean stimulus-evoked activity of LI, POR, P and LM neurons.
n = 8 mice. Statistical analysis was performed using one-way analysis of variance
(ANOVA) with correction using Tukey’s honest significant difference (HSD)
test; P > 0.05 for all tests. h, Same analysis as in g but for stimulus reactivation
activity (P > 0.05 for all tests). For g and h, scale bars, 0.25 mm. Data in e–h are
mean ± s.e.m. NS, not significant; *P < 0.05; **P < 0.01.

neurons (1,361 ± 94 neurons per session, mean ± s.e.m.), which either As illustrated in three example trials, we observed many events
showed a preferential increase in activity to S1 or S2, or responded consisting of transient (~350 ms) moments of synchronous activ-
similarly to both (Fig. 1c and Extended Data Fig. 1a–c). ity of stimulus-driven neurons in the tens of seconds after stimulus

2 | Nature | www.nature.com
presentation (Fig. 1d and Extended Data Fig. 1d,e). Using a multinomial
logistic-regression-based classifier, we assigned a probability that
these synchronous offline events resembled patterns of responses to
S1, S2 or neither. We then defined S1 or S2 reactivations as events with
a high classifier matching probability to S1 or S2 (>0.75; Fig. 1d; see the
Methods and Extended Data Fig. 1d–i for classifier details, multiple
shuffle controls and validation).
Reactivations were associated with moments of particularly low
arousal (Fig. 1e). Seconds before the onset of a reactivation, pupil area—
an index of arousal35—was already around one-third of that observed
during active movement, and briefly constricted further during the
reactivation. Similar to previous studies18,20,36, cortical reactivations
were accompanied by an increase in sharp-wave ripple band power
in the dorsal hippocampal CA1, indicating that lateral visual cortical
stimulus reactivations participate in global events (Fig. 1f and Extended
Data Fig. 2a). Reactivations were not accompanied by increased brain
or eye movement (Extended Data Fig. 2b–d).
The neurons that contributed to stimulus reactivations were evenly
distributed across the four lateral visual cortical areas and across simul-
taneously imaged depths within layer 2/3, with similar levels of activity
in each area and depth during both stimulus presentations and stimulus
reactivations (Fig. 1g,h and Extended Data Fig. 2e,f).
We wondered whether imaging thousands of neurons instead of
hundreds18 increased our sensitivity for capturing stimulus reacti-
vations. Indeed, when using only a random 10% of neurons instead
of the full dataset of approximately 6,900 neurons per session, over
two-thirds of identified reactivations were missed, and the rate of
false-positive reactivations also increased (Extended Data Fig. 2g–i).
Rates of reactivations gradually decay
The content of awake reactivations of spatiotemporal patterns of activ-
ity (replay) in the hippocampus can be biased towards salient (novel,
rewarding or aversive) recent experiences, with a reactivation rate
that often decays across trials17,18,37–39. As illustrated for an example
session (Fig. 2a) and quantified below, cortical stimulus reactivations
exhibited similar properties.
We first evaluated changes in the rate of stimulus reactivations
throughout a session across mice. Reactivation rates increased by
approximately fourfold above the baseline levels after the first stimulus
presentation and decayed to the baseline over 2 h (Fig. 2b and Extended
Data Fig. 3a). Within the ITI after each stimulus presentation, stimulus
reactivation rates increased and then decayed across tens of seconds
(Fig. 2b and Extended Data Fig. 3a). The content of reactivations after
a stimulus showed a strong 3:1 bias towards reactivations resembling
the most recent stimulus—an effect that persisted throughout each
session but that gradually declined throughout each ITI (Fig. 2c and
Extended Data Fig. 3b). Notably, these strong rate and bias effects were
far less evident when analysing only hundreds rather than thousands
of simultaneously recorded neurons (Extended Data Fig. 3c).
The gradual decrease in reactivation rates across the session (Fig. 2b)
suggested that reactivation rates may scale inversely with the frequency
of recent exposures to a stimulus. We therefore considered possible
roles for stimulus novelty and peri-stimulus arousal in regulating reac-
tivation rates. Indeed, we observed increased reactivation rates when
the previous stimulus was preceded by a different stimulus compared
with when the previous stimulus was preceded by the same stimulus
(Extended Data Fig. 3d). Furthermore, reactivation rates were posi-
tively correlated with pupil area and response magnitude during the
preceding stimulus presentation (Extended Data Fig. 3e,f). However,
these trial-to-trial correlations could not explain the gradual decrease
in the reactivation rate across each session, as stimulus response mag-
nitude, pupil area, brain motion and other facial features were not
correlated with the reactivation rate across the session (Fig. 3c and
Extended Data Fig. 4a–e).
Local silencing of stimulus responses
We wondered whether the rate and bias effects described above
required the same cortical neurons that participate in reactivations
to be active during the preceding stimulus presentation40. To test
this, we optogenetically silenced stimulus-evoked activity in excita-
tory neurons throughout the imaged region of the lateral visual
cortex in a subset of mice. We performed local viral injections of the
red-shifted opsin Chrimson under the S5E2 enhancer, which selec-
tively targets parvalbumin interneurons41, along with Cre-dependent
jGCaMP7s in Emx1-cre mice (Fig. 2d and Extended Data Fig. 5a). Photo-
stimulation inhibited more than 90% of peri-stimulus activity of
excitatory neurons in the lateral visual cortex (Fig. 2e and Extended
Data Fig. 5b) but did not affect arousal (Extended Data Fig. 5c) or the
overall cortical activity levels during the subsequent ITI (Extended
Data Fig. 5d).
Inhibition during stimulus presentation on a random subset of trials
strongly reduced the subsequent stimulus reactivation rates (Fig. 2f;
n = 3 mice; 8 ± 1 sessions per mouse, mean ± s.e.m.). However, reacti-
vation rates remained higher than during the baseline period before
the first stimulus presentation, even when matched for pupil-indexed
arousal, suggesting that other brain regions may also have a role in
driving local stimulus reactivations40 (Extended Data Fig. 5e,f). Nota-
bly, peri-stimulus inhibition also abolished the subsequent bias in the
content of reactivations towards the stimulus (Fig. 2g and Extended
Data Fig. 5g; n = 3 mice). Thus, local cortical activity during a sensory
experience is necessary for the subsequent appearance of biased
reactivations.
Reactivations track orthogonalization
Previous studies of the hippocampus suggest that reactivations of
certain experiences may have a role in memory consolidation and
learning1,2. Although visual cortical response patterns are known to
gradually change across repeated presentations3–6, how reactivations
might relate to this process remains unclear.
Inspection of single-trial responses during a typical example session
suggested that many neurons initially responded to both stimuli, but
gradually lost their responses to one or the other stimulus (Fig. 3a).
As such, the patterns of population responses to the two stimuli
should become more dissimilar across presentations, potentially
facilitating stimulus discriminability42. We quantified this phenom-
enon using a running Pearson’s correlation between neighbouring S1
and S2 single-trial response patterns. The two stimulus representations
became more orthogonal (less correlated) across trials (Fig. 3b and
Extended Data Fig. 6a–d). This orthogonalization is unlikely to be due
to a stimulus-independent decrease in evoked response magnitudes
driven by a progressive reduction of novelty and/or arousal for several
reasons. First, mean stimulus responses (averaged across neurons)
were stable across the session (Fig. 3c and Extended Data Fig. 6e,f),
consistent with some previous studies43–45. Moreover, only a small
proportion of neurons showed similar decreases in their responses
to both stimuli across the session and removing these neurons from
the analysis did not affect the gradual orthogonalization of response
patterns (Extended Data Fig. 6g,h).
The orthogonalization effects were similar between mice that
did not receive any photoinhibition during stimulus presentation
(non-inhibition mice, n = 5) and from stimulus-inhibition mice (n = 3,
control trials only; Extended Data Fig. 6a–c,e,f). We therefore pooled
these two sets of mice for subsequent analyses (results for each set are
also shown separately in Extended Data Fig. 6, 8, 9 and 10). Note that
stimulus-evoked responses from control trials in stimulus-inhibition
mice were stronger than in the non-inhibition mice (Extended Data
Fig. 6e,f). As stronger stimulus responses are correlated with higher
reactivation rates (Extended Data Fig. 3f), this may explain why
Nature | www.nature.com | 3
Article
a b ***
S1 reactivations after S1 r = –0.63, **** r = –0.53, ****
1 0.15 0.15
Trial number

Mouse 1

(probability per second)

(probability per second)

25 Mouse 2
Mouse 3

Reactivation rate

Reactivation rate
50 0.10 0.10 Mouse 4
Mouse 5
0 10 20 30 40 50 60 Mean
S2 reactivations after S1
1 0.05 0.05
Trial number

25

50 0 0
−0.5 0 0.5 1.0 1.5 2.0 0 10 20 30 40 50 60
0 10 20 30 40 50 60
S1 reactivations after S2 Time relative to first stimulus onset (h) Time relative to stimulus onset (s)
1
c
Trial number

r = –0.35, **** r = –0.65, ****

25 1.0 1.0
~3:1

to the most recent stimulus

to the most recent stimulus

Bias in reactivation content

Bias in reactivation content

odds
50
0.5 0.5
0 10 20 30 40 50 60
S2 reactivations after S2
1 0 0
Trial number

25
−0.5 −0.5
50

0 10 20 30 40 50 60 −1.0 −1.0
Time relative to stimulus onset (s) 0 0.5 1.0 1.5 2.0 0 10 20 30 40 50 60
Time relative to first stimulus onset (h) Time relative to stimulus onset (s)
d f
AAV(PHP.eb)-syn-FLEX-jGCaMP7s and
AAV1-S5E2(PV)-Chrimson-tdTomato 0.15 0.15
Emx1-cre (glutamatergic neurons) Control trials
(probability per second)

(probability per second)

Stimulus-inhibition trials
Reactivation rate

Reactivation rate
0.10 0.10

**
Lateral

**
visual 0.05 0.05
cortex

0 0
−0.5 0 0.5 1.0 1.5 2.0 0 10 20 30 40 50 60
e Stimulus- Stimulus-
Control inhibition Control inhibition g Time relative to first stimulus onset (h) Time relative to stimulus onset (s)
Normalized deconvolved Ca2+ activity

trials trials trials trials

1 1.0 1.0
to the most recent stimulus
Bias in reactivation content

to the most recent stimulus

Bias in reactivation content

Control trials
Stimulus-inhibition trials
0.5 0.5
Neuron number

0.2
*

*
0 0

0 −0.5 −0.5

800 −1.0 −1.0

0 2 0 2 0 2 0 2 0 0.5 1.0 1.5 2.0 0 10 20 30 40 50 60
Time relative to stimulus onset (s) Time relative to first stimulus onset (h) Time relative to stimulus onset (s)

Fig. 2 | Cortical responses to stimuli drive subsequent reactivations. parvalbumin interneurons (due to S5E2 enhancer), respectively. Scale bar,
a, Example single-session raster plot of S1 and S2 reactivations (green and red 0.1 mm. e, The mean stimulus-evoked activity of driven neurons across trials
dots) after the presentation of S1 or S2. b, Left, the reactivation rate across the from one example session. On 50% of trials, stimulus-evoked activity was
session, including the 0.5 h baseline period before any stimulus presentations suppressed from 1 s before to 1 s after stimulus presentation (light green and
(dark shaded region). n = 5 mice. Statistical analysis was performed using a pink bars, stimulus-inhibition trials) using optogenetics (Methods). f, The same
paired t-test (P = 7.7 × 10 −4) and linear least-squares regression (P = 2.7 × 10 −5) analysis of reactivation rate as described in b but for control versus inhibition
with Holm–Bonferroni correction. Right, the reactivation rate during the ITI. trials. n = 3 mice. Statistical analysis was performed using paired t-tests
n = 5 mice. Statistical analysis was performed using linear least-squares between the mean of traces; P = 0.0025 (left), P = 0.0039 (right). g, The bias
regression; P = 0.0067. c, Left, the bias index of reactivation content (positive index as described in c but for control versus inhibition trials. n = 3 mice.
values indicate bias towards the most recent stimulus; Methods). n = 5 mice. Statistical analysis was performed using paired t-tests between the mean of
Statistical analysis was performed using linear least-squares regression; traces; P = 0.029 (left), P = 0.010 (right). Data in b,c,f,g are mean ± s.e.m.
P = 0.025. Right, bias throughout the ITI. n = 5 mice. Statistical analysis was *P < 0.05; **P < 0.01; ***P < 0.001; ****P < 0.0001. Light shaded regions show
performed using linear least-squares regression; P = 3.9 × 10 −4. d, Schematic the excluded portion of the ITI. White-noise bars in b,c,f,g indicate the stimulus
and mean in vivo image of selective viral expression of jGCaMP7s and Chrimson- presentation. All of the statistical tests were two-tailed.
tdTomato in lateral visual cortical glutamatergic neurons (Emx1-cre) and

reactivation rates in control trials from stimulus-inhibition mice were track the same neurons across the first 6 days of imaging (1,255 ± 187
higher than in non-inhibition mice (Extended Data Fig. 5h). neurons tracked across all 6 days, mean ± s.e.m., n = 5 non-inhibition
We wondered whether this gradual decrease in similarity of S1 and mice; Extended Data Fig. 7a,b). We found that the similarity between
S2 responses continued across days. We used ROICaT (Methods) to patterns of responses to the two stimuli continued to decrease across

4 | Nature | www.nature.com
 a b c NS
Single S1 trials
r = –0.30, **** r = 0.06, NS

deconvolved Ca2+ per second)

Early Late

Stimulus activity (normalized

0.6 Mouse 1 1.0
Mouse 2

(correlation between
Response similarity
Mouse 3 0.8
S1-preferring Mouse 4

S1 and S2)
neurons 0.4 Mouse 5
Mouse 6 0.6
Mouse 7
0.2 Mouse 8 0.4
Mean
S2-preferring 0.2 Baseline activity
neurons 0
0
20 neurons First trial Last trial First trial Last trial
(~60 or ~120)
5s

d e
0.6
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6
0.6
** 0.6 *

Initial response similarity

Final response similarity

(correlation between

(correlation between

(correlation between
Response similarity

S1 and S2)

S1 and S2)

S1 and S2)
0.4 0.4 0.4

0.2 0.2 0.2

0 0 0
1 100 200 300 400 500 600 700 1 2 3 4 5 6 1 2 3 4 5 6
Trial number Day Day
f g r = 0.11, *** r = –0.50, **** h NS NS
**** *
deconvolved Ca2+ per second)
Stimulus activity (normalized

1.0 ****

increase (+) or decrease (–)

Fraction of neurons with
0.3 Increase neurons

from early to late trials

0.3

in stimulus selectivity
(correlation between
Response similarity

0.8 Decrease neurons

0.2
S1 and S2)

0.6 0.2
0.1
0.4
No-change neurons 0 0.1
0.2 Increase neurons
Decrease neurons −0.1
0 0
First trial Last trial First trial Last trial + – + – + –
No-change Increase Decrease
neurons neurons neurons
i j k
0.25 *** ****
Cross-correlation between
0.25 1.00
(high-pass-filtered traces)
0.6 0.6
(high-pass-filtered traces)
(probability per second)

response similarity and

(correlation between
Response similarity

and reactivation rate

and reactivation rate

Correlation between

Correlation between
response similarity

response similarity

0.20 0.4 0.4

Reactivation rate

reactivation rate

0.20
S1 and S2)

0.2 0.2 0.50

0.15
0.15 0 0
0.10
−0.2 −0.2 0
0.10 0.05 −0.4 −0.4
r = 0.58
0.05 0 −0.6 −0.6 −0.50
0 20 40 60 80 100 120 −10 −5 0 5 10
Trial number Shift in trial number

Fig. 3 | Progressive separation of stimulus response patterns correlates similarity as described in b but for increase or decrease neurons. n = 8 mice.
with reactivation rate. a, Example early and late S1 trials. b, Response Statistical analysis was performed using linear least-squares regression with
similarity (Methods). n = 8 mice. Statistical analysis was performed using linear Holm–Bonferroni correction; P = 3.6 × 10 −4 (red), P = 1.4 × 10 −60 (blue). h, The
least-squares regression; P = 5.2 × 10 −22. Data are from 5 mice with approximately fraction of neurons that increase or decrease their stimulus selectivity from
120 trials per session and 3 mice with approximately 60 control (no inhibition) early to late trials for each group. n = 8 mice. Statistical analysis was performed
trials per session. c, Stimulus-evoked activity. n = 8 mice. Statistical analysis using paired t-tests with Holm–Bonferroni correction; P > 0.05 (no-change
was performed using a paired t-test (P = 0.095) and linear least-squares neurons and increase neurons), P = 0.025 (decrease neurons). i, Example
regression (P = 0.090). d, Response similarity as described in b but plotted response similarity and reactivation rate traces. j, Correlation between the two
across days (n = 5 mice) using the same tracked neurons. e, Response similarity variables for unfiltered traces (left) and after high-pass filtering (right). n = 8
at start or end of each day in d. n = 5 mice. Statistical analysis was performed mice. Statistical analysis was performed using t-tests versus 0; P = 1.3 × 10 −4
using paired t-tests; P = 0.0026 (left), P = 0.030 (right). f, Stimulus-evoked (left), P = 7.9 × 10 −9 (right). k, Cross-correlation between high-pass-filtered
activity as described in c but for no-change, increase or decrease neurons. response similarity and reactivation probability traces. n = 8 mice. Data in
n = 8 mice. Statistical analysis was performed using unpaired t-tests with Holm– b–h,j,k are mean ± s.e.m. All statistical tests were two-tailed. NS, not significant;
Bonferroni correction; P = 4.0 × 10 −12 (red), P = 4.0 × 10 −11 (blue). g, Response *P < 0.05; **P < 0.01; ***P < 0.001; ****P < 0.0001.

days (Fig. 3d). In particular, the response similarity at the start and end To determine which neurons contributed to the orthogonalization of
of each day both decreased across days (Fig. 3e). This was true despite population responses, we considered groups of neurons that increased,
a partial resetting of response similarity from the end of one day to decreased or showed similar stimulus-evoked activity across trials.
the start of the next (Extended Data Fig. 7c), suggestive of a partial ‘Increase neurons’ were defined as those that exhibited an increase in
‘forgetting’ effect. evoked activity from early to late trials, whereas ‘decrease neurons’

Nature | www.nature.com | 5
Article
showed the opposite trend; ‘no-change’ neurons had stable evoked
activity across the session (Fig. 3f and Extended Data Fig. 8a). These
groups did not exhibit substantial differences in the proportion of
neurons per group, baseline activity, location within visual region or
cortical depth, or within-group noise correlations (Extended Data
Fig. 8b–f). When we quantified the changes in correlation between S1-
and S2-evoked response patterns across trials separately for the sets of
decrease or increase neurons, we observed a similar orthogonalization
of stimulus response patterns for decrease neurons but not for increase
neurons (Fig. 3g and Extended Data Fig. 8g; by definition, response
patterns for no-change neurons remained unchanged).
We hypothesized that the decrease in similarity between S1- and
S2-evoked response patterns in decrease neurons was due to an increase
in response selectivity in these neurons. We therefore quantified the
percentage of neurons in each group in which the stimulus selectivity
(differential response to S1 or S2) increased or decreased from early
to late trials. Indeed, twice as many decrease neurons increased versus
decreased their stimulus selectivity (Fig. 3h and Extended Data Fig. 8h).
By contrast, similar numbers of neurons in the other groups increased
or decreased their selectivity (Fig. 3h and Extended Data Fig. 8h). Thus,
although the opposing changes in response magnitude in increase
neurons and decrease neurons in the lateral visual cortex results in
consistent mean responses across all neurons, decrease neurons seem
to have a greater role in stimulus orthogonalization through increases
in stimulus selectivity.
Consistent with the sharp decreases across the session in both the
similarity of population responses to S1 versus S2 (Fig. 3b) and in the
stimulus reactivation rates (Fig. 2b), these two measures were positively
correlated across trials (Fig. 3i,j). Notably, even after removing slow
changes in these two measures across the session, they co-fluctuated
at a faster time scale of around 10 min (Fig. 3k and Extended Data Fig. 8i;
peak correlation at zero-trial delay). Thus, the evolution of the repre-
sentation of a stimulus in the lateral visual cortex across trials tightly
correlates with the rate of stimulus-specific reactivations, suggesting a
possible relationship between reactivations and subsequent orthogo-
nalization of response patterns.
Reactivations predict future responses
To better understand the relationship between reactivation patterns
and the changes in single-trial stimulus-evoked response patterns
across a session, we projected each pattern onto the axis of change
in stimulus-evoked population activity between early and late trials
within a session (Fig. 4a). Both S1- and S2-evoked response patterns
showed progressive changes from early to late trials (representational
drift3–6), with larger changes early in each session (Fig. 4b and Extended
Data Fig. 9a–c). If stimulus reactivations were copies of the previous
stimulus-evoked response pattern, as suggested in some previous
studies1,8,9, we would expect a matching evolution of stimulus response
patterns and stimulus reactivation patterns from early to late trials.
However, the projected stimulus reactivation patterns were instead
stable across trials (Fig. 4b and Extended Data Fig. 9a–c). Notably, even
after the first stimulus presentation, the projected patterns of stimulus
reactivations already resembled the stimulus-evoked response pattern
much later in the session, after its gradual drift (Fig. 4b). This finding
was consistent across depths within layer 2/3 and was not driven by
neurons with similar decreases in responses to both stimuli (Extended
Data Fig. 9d,e). Furthermore, this finding was not due to the differential
influence of early versus late trials on classifier sensitivity, as we sepa-
rately trained classifiers on early, middle and late portions of a session,
and used the maximum of the three estimated matching probabilities
at each timepoint (Methods and Extended Data Fig. 1d).
We next examined whether stimulus reactivations were predictive
not only of within-day representational drift, but also of across-day
drift. For this analysis, we used the set of neurons tracked across 6 days
6 | Nature | www.nature.com
(Extended Data Fig. 7a,b). We first projected stimulus-evoked and
reactivation patterns from all days onto the axis of change in stimulus
response pattern from the start to the end of day 1. When considered
along this particular axis, the stimulus responses on subsequent days
appear to drift towards or remain similar to the stimulus response
pattern at the end of day 1 (Fig. 4c). By contrast, projections of reactiva-
tion patterns onto this axis remained constant across all days, always
matching the post-drift stimulus response pattern (Fig. 4c). When we
instead projected stimulus and reactivation patterns onto the axis
of change in stimulus-evoked pattern from day 1 to day 6, we found
that the stimulus responses do indeed continue to progressively drift
further across several days (Fig. 4d). Critically, the reactivation pat-
terns at the start of day 1 (and subsequent days), when projected along
this axis, were already similar to the pattern that the stimulus responses
gradually drift to by the end of day 6. Thus, reactivation patterns predict
representational drift both within and across daily sessions.
To gain additional insights into how the patterns of activity differed
between early stimulus-evoked responses and early reactivations,
we compared the mean activity averaged across decrease neurons or
increase neurons with the mean activity across no-change neurons on
each trial. Even during early trials within a session, decrease neurons
showed relatively less activity during reactivations compared with
during stimulus-evoked responses, while the opposite was true for
increase neurons (Fig. 4e and Extended Data Fig. 9f). To compare the
stimulus-evoked and reactivation patterns more directly, we scaled up
activity levels across all neurons during early stimulus reactivations
by a common scale factor (1.3×; Extended Data Fig. 9g) such that the
mean activity (averaged across neurons) of stimulus-evoked response
patterns and reactivations was similar. When we then subtracted early
stimulus-evoked responses from 1.3×-scaled early stimulus reactiva-
tions, we again found that decrease neurons were relatively less active
during early reactivations compared with during early stimuli, while
the converse was true for increase neurons (Fig. 4f and Extended Data
Fig. 9h). Meanwhile, no-change neurons showed relatively similar levels
of participation in early stimulus responses and reactivations (Fig. 4f
and Extended Data Fig. 9h). Together, these data show that neurons
that participate in early reactivations below, at or above their relative
level of participation in early stimulus-evoked response patterns will
gradually decrease, show no change or increase their stimulus-evoked
activity later in the session, respectively (Extended Data Fig. 9i). The
above findings suggest that both the rate and pattern of stimulus reac-
tivations could be important in predicting the future content and rate
of change in stimulus-evoked response patterns, rather than stabilizing
the previous stimulus-evoked response pattern.
Modelling future stimulus responses
We wondered whether stimulus reactivations alone might be sufficient
to predict the nature and rate of the drift in future stimulus-evoked
response patterns. We developed a simple heuristic model that uses
only the stimulus-evoked response pattern on the first trial and all
1.3×-scaled stimulus reactivations observed during each trial to itera-
tively predict stimulus-evoked response patterns on each subsequent
trial (Fig. 4g). In this model, each time a S1 stimulus reactivation occurs
after a S1 trial (and likewise for S2), we modified the estimate of the
upcoming stimulus-evoked response pattern by adding the difference
between the 1.3×-scaled reactivation and the current stimulus-evoked
pattern, multiplied by a fixed plasticity term (Fig. 4g). We parametrically
varied this plasticity term to find the best fit value and applied the same
single value across all sessions and mice (Extended Data Fig. 10a). Intui-
tively, this model should drive faster changes in the stimulus-evoked
response pattern early in each session (as seen in Fig. 4b), due both
to the larger differences between the reactivation patterns and the
stimulus-evoked response patterns, and to the increased number of
reactivations per trial (and consequent model iterations; Fig. 2b).
 a Activity of
b *** ***

Similarity to early versus late

Similarity to early versus late

neuron 1 Early S1-evoked response S2-evoked response

S1 response pattern

S2 response pattern
stimulus S1 reactivation S2 reactivation
Early stimulus response –1.0 –1.0
response

Vs −0.5 −0.5
Vs

Activity of Late
neuron 2 stimulus 0 0
Activity of response
neuron n Late stimulus
response First trial Last trial First trial Last trial
c *** ***
Similarity to early versus late

Similarity to early versus late

Day 1 S1-evoked response S2-evoked response
early S1 response pattern
−1.5 −1.5

S2 response pattern
stimulus S1 reactivation S2 reactivation
response
–1.0 –1.0
Vs

Day 1 −0.5 −0.5

late
stimulus Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 1 Day 2 Day 3 Day 4 Day 5 Day 6
response 0 0
1 50 100 150 200 250 300 350 1 50 100 150 200 250 300 350
Trial number Trial number
d *** ***

day 6 S2 response pattern

day 6 S1 response pattern

S1-evoked response S2-evoked response

Similarity to day 1 versus

Day 1

Similarity to day 1 versus

stimulus −1.5 S1 reactivation −1.5 S2 reactivation
response

Vs –1.0 –1.0

Day 6
stimulus −0.5 −0.5
response
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 1 Day 2 Day 3 Day 4 Day 5 Day 6
0 0
1 50 100 150 200 250 300 350 1 50 100 150 200 250 300 350
Trial number Trial number
e f **** ****
**** **** **** **** ** **** * ***
deconvolved Ca2+ per second)

deconvolved Ca2+ per second)

2.0 2.0
no-change neuron stimulus

no-change neuron stimulus

0.6 0.6
Ratio of decrease neuron/
Ratio of increase neuron/

1.3 × S1R – S1 activity in

1.3 × S2R – S2 activity in

early trials (normalized

early trials (normalized

activity in early trials

activity in early trials

0.4 0.4
1.5 1.5
0.2 0.2
1.0 1.0 0 0
−0.2 −0.2
0.5 0.5
−0.4 −0.4

0 0 −0.6 −0.6
S1 S1R S2 S2R S1 S1R S2 S2R Increase No-change Decrease Increase No-change Decrease
neurons neurons neurons neurons neurons neurons

g h
Cross-correlation

Cross-correlation
1 1
of data versus

of data versus
Actual stimulus
Similarity to early versus late

Similarity to early versus late

St1 response on trial 1

model

model
S1 response pattern

S2 response pattern

St2,j = St1,j + (1.3 × Rt1,j – St1,j) for all Rt1,j 0 0

−1.0 −1.0
St2 Modelled stimulus −10 0 10 −10 0 10
response on trial 2 Shift in trial number Shift in trial number
Sti+1,j = Sti,j + (1.3 × Rti,j – Sti,j) for all Rti,j −0.5 −0.5
Modelled stimulus
Sti response on trial i
0 S1-evoked response 0 S2-evoked response
S = stimulus-evoked response i = trial number S1 modelled evoked response S2 modelled evoked response
R = stimulus reactivation j = number within trial First trial Last trial First trial Last trial
= plasticity term (fixed across mice)

i NS j r = 0.23, **** r = –0.43, **** k Activity of

0.4
neuron 1
Increase neurons S11 S21
(correlation between
Response similarity

0.3
similarity (correlation
between S1 and S2)
Modelled response

0.3 Decrease neurons S12 S22

R1 R2
S1 and S2)

NS S2i
0.2 S1i R1 R2
0.2 R2
R1 Late S2 pattern;
0.1 0.1 Late S1 pattern; R2 pattern
Real data R1 pattern
Modelled data Activity of
0 Activity of neuron 2
0
First trial Last trial First trial Last trial neuron n

Fig. 4 | Reactivations predict representational drift. a, Schematic of drifting Statistical analysis was performed using one-way ANOVA with correction
stimulus response patterns along Vs. Vs denotes the vector along the axis from using Tukey’s HSD test; left: P = 2.6 × 10 −8 (increase versus decrease), P = 0.0068
early to late response patterns. b, Projection of S1-evoked response patterns (increase versus no-change), P = 3.1 × 10 −5 (no-change versus decrease); right:
and reactivations onto Vs (left). Right, the same but for S2. n = 8 mice. Statistical P = 3.2 × 10 −7 (increase versus decrease), P = 0.021 (increase versus no-change),
analysis was performed using paired t-tests; P = 1.3 × 10 −4 (left), P = 1.5 × 10 −4 P = 2.0 × 10 −4 (no-change versus decrease). g, Model using reactivations to
(right). c, Projection of S1- and S2-evoked response patterns and reactivations predict future stimulus responses. h, Comparison of actual versus modelled
onto Vs as described in b but across days using tracked neurons and projected projection. n = 8 mice. Insets: cross-correlation between high-pass-filtered
onto the day 1 axis. n = 5 mice. Statistical analysis was performed using paired actual and modelled projections. i, The response similarity for actual versus
t-tests; P = 5.0 × 10 −4 (left), P = 3.7 × 10 −4 (right). d, S1- and S2-evoked response modelled data. n = 8 mice. Statistical analysis was performed using paired
patterns and reactivations as in c but projected onto the day 1 to 6 axis. n = 5 t-tests with Holm–Bonferroni correction; P = 0.13 (first), P = 0.18 (last).
mice. Statistical analysis was performed using paired t-tests; P = 9.0 × 10 −4 (left), j, Response similarity as in Fig. 3g, but for modelled data. n = 8 mice. Statistical
P = 3.8 × 10 −4 (right). e, Left, early-trial activity of increase neurons relative to analysis was performed using linear least-squares regression with Holm–
no-change neurons during stimulus presentation (S1 or S2) versus reactivations Bonferroni correction; P = 1.8 × 10 −13 (red), P = 1.7 × 10 −43 (blue). k, Summary.
(S1R or S2R). Right, the same but for decrease neurons. n = 8 mice. Statistical S1- and S2-evoked response patterns are pulled towards their respective
analysis was performed using paired t-tests with Holm–Bonferroni correction; reactivation pattern across trials. Data in b–f,h–j are mean ± s.e.m. All statistical
from left to right, P = 1.4 × 10 −8, P = 3.0 × 10 −8, P = 1.6 × 10 −5, P = 1.0 × 10 −7. f, Early- tests were two-tailed. NS, not significant; *P < 0.05; **P < 0.01; ***P < 0.001;
trial 1.3×-scaled reactivation activity minus stimulus activity. n = 8 mice. ****P < 0.0001.

Nature | www.nature.com | 7
Article
Indeed, this model accurately captured the evolution of future stimulus
response patterns, including the more rapid rate of change in early trials
(Fig. 4h and Extended Data Fig. 10b,c). Furthermore, for any given ses-
sion, the projections of stimulus-evoked response patterns exhibited
small fluctuations across several trials (Extended Data Fig. 10b). By
high-pass filtering the actual and modelled stimulus-evoked responses
to remove the global drift in the patterns over the session, we found
that our model was even able to use single-trial fluctuations in reactiva-
tion content and rate to capture these short-timescale fluctuations in
stimulus-evoked response patterns on upcoming trials (Fig. 4h (insets)
and Extended Data Fig. 10d). This analysis highlights the capacity of the
instantaneous content and rate of reactivations to predict subsequent
changes in the stimulus-response pattern.
Our simple model also captured the gradual orthogonalization of
responses to S1 and S2 (Fig. 4i and Extended Data Fig. 10e). As with
the actual data (Fig. 3g), the orthogonalization in the modelled data
was driven by decrease neurons and not increase neurons (Fig. 4j
and Extended Data Fig. 10f; neuron groups defined using actual
data; Fig. 3f). Thus, a very simple model using only reactivations can
capture the dynamics of drift in stimulus-evoked response patterns
and stimulus orthogonalization across timescales from minutes
to hours.
In summary, stimulus reactivations during the ITI, while by defini-
tion somewhat similar to early stimulus-evoked responses, neverthe-
less differ in pattern from early stimulus-evoked responses. These
stimulus reactivations accurately predict future stimulus-evoked
responses over the course of several trials and the accumulated drift
in response patterns over a session at rates proportional to the reacti-
vation rate (Fig. 4k). Overall, these findings show that passive stimuli
induce reactivations in the lateral visual cortex that predict representa-
tional drift and orthogonalization of distinct stimulus representations.
Discussion
We observed transient, hippocampal ripple-coupled reactivations
of specific stimuli in the lateral visual cortex during periods of quiet
waking in the tens of seconds after stimulus presentation, provid-
ing a bridge between studies of offline reactivation in the sensory
cortex13,16,18,23 and studies in the hippocampus and elsewhere with
similar observations during spatial navigation tasks11–17,22. In contrast
to many previous studies18, stimulus reactivations in our recordings
occurred in the absence of any primary reinforcer. Despite this lack
of reinforcement, lateral visual cortex population responses to dis-
tinct stimuli not only drifted but also became more orthogonal across
repeated presentations3 and across days, while maintaining homeo-
static levels of mean evoked activity across neurons43–45. The fact that
increase and decrease neurons were initially more weakly and more
strongly driven, respectively (Fig. 3f), also points to a role for daily
homeostatic regulation of response magnitudes across the population.
Previous research had not examined whether offline reactivations
were related to representational drift and/or separation of sensory
activity patterns in any brain area. Several theories posited that hip-
pocampal reactivations are copies of previous experiences1,8,9, and that
they serve to stabilize the patterns of activity that occurred during these
experiences1,8,46–48. Meanwhile, other theories have emphasized the
potential value of reactivations that differ in content from those of pre-
vious experiences25–27. We found that patterns of cortical reactivations
that followed stimulus presentations early in each session already dif-
fered in a systematic manner from previous stimulus-evoked response
patterns in that they more strongly resembled stimulus-evoked pat-
terns later in the session and in future sessions. Indeed, by feeding
only the set of recorded reactivations that followed each stimulus into
a simple model, we could predict the evolution of stimulus response
patterns and response orthogonalization across single trials and
throughout a session, including the more rapid plasticity rate early in
8 | Nature | www.nature.com
the session as well as the minute-to-minute fluctuations in responses.
Thus, our findings suggest that stimulus reactivations may have a more
instructive role than previously appreciated in shaping and orthogonal-
izing neural representations of recently experienced stimuli. Causal
examination of this hypothesis should soon be possible using emerging
electrophysiological technologies that enable simultaneous record-
ings of thousands of neurons49, thereby matching the sensitivity of our
approach, while also enabling sufficiently fast closed-loop silencing of
content-specific reactivations50–52.
Single-trial optogenetic silencing of the lateral visual cortex dur-
ing a stimulus presentation prevented the selective increase in reac-
tivations of that stimulus during the following tens of seconds. This
demonstrates that the participation of lateral visual cortex neurons
in stimulus reactivations requires previous activation of these same
neurons during the stimulus. Furthermore, these results suggest that
some of the changes underlying response orthogonalization may
involve local synaptic plasticity, in addition to other potential mech-
anisms3. Indeed, the model that we used to accurately predict which
neurons would increase or decrease their stimulus responses could
be implemented biologically using a simple learning rule. Specifically,
neurons that over- or under-participate in stimulus reactivations may
strengthen or weaken their connectivity, respectively, to other neu-
rons in the co-activated stimulus-evoked ensemble12,53,54. If so, future
stimuli that activate parts of this ensemble would more strongly recruit
over-participating increase neurons, and would less strongly recruit
under-participating decrease neurons.
The orthogonalization of patterns of responses to distinct stimuli
in the lateral visual association cortex might prevent overgenerali-
zation when subsequently associating a stimulus with an outcome.
Offline reactivations might accelerate such separation of activity pat-
terns without requiring frequent experience of each stimulus (which
is unlikely for most salient stimuli in nature), while also helping to link
visual stimuli to other reliably co-occurring, non-visual stimuli dur-
ing a given experience55. Future studies can assess whether similar
network changes occur when stimuli are more similar to each other,
when larger sets of stimuli are presented or when stimuli are coupled
to primary reinforcers.
It remains unclear how early stimulus reactivation patterns could
resemble late stimulus-evoked patterns. This may reflect pre-existing
biases in local cortical connectivity that result in a manifold of preferred
cortical activity patterns56,57. Feedforward input during presentation of
our visual stimuli (particularly given their deviation from natural stimuli
encountered in nature) may drive activity patterns that initially stray
somewhat from this manifold, with experience-dependent plasticity
then pulling the sensory-evoked patterns back to the nearest location
on the manifold.
In summary, our study indicates that local stimulus-evoked activity
patterns in the lateral visual cortex give rise to reverberating reac-
tivations of similar but not identical activity patterns in the tens of
seconds after the stimulus, particularly when the stimuli are salient or
unexpected. Stimulus-evoked patterns gravitate towards reactivated
patterns at a pace that is proportional to the reactivation rate and to
the residual difference between the evoked and reactivated patterns.
This highlights the idea that, more generally, when sensory experiences
are sparse and punctuated by longer periods of quiet rest, offline reac-
tivations may actively reorganize sensory-evoked response patterns
to enhance the separability of population responses during distinct
experiences55,58 while also potentially supporting pattern completion59,
memory consolidation60, stabilization46 and associative learning18.
Online content
Any methods, additional references, Nature Portfolio reporting summa-
ries, source data, extended data, supplementary information, acknowl-
edgements, peer review information; details of author contributions
and competing interests; and statements of data and code availability 32. Gorski, J. A. et al. Cortical excitatory neurons and glia, but not GABAergic neurons, are
produced in the Emx1-expressing lineage. J. Neurosci. 22, 6309–6314 (2002).
are available at https://doi.org/10.1038/s41586-023-06810-1. 33. Ramesh, R. N. et al. Intermingled ensembles in visual association cortex encode stimulus
identity or predicted outcome. Neuron 100, 900–915 (2018).
34. Zhuang, J. et al. An extended retinotopic map of mouse cortex. eLife. 6, e18372 (2017).
1. Foster, D. J. Replay comes of age. Annu. Rev. Neurosci. 40, 581–602 (2017). 35. Bradley, M. M. et al. The pupil as a measure of emotional arousal and autonomic activation.
2. Tambini, A. & Davachi, L. Awake reactivation of prior experiences consolidates memories Psychophysiology 45, 602–607 (2008).
and biases cognition. Trends Cogn. Sci. 23, 876–890 (2019). 36. Jeong, H. et al. Sensory cortical ensembles exhibit differential coupling to ripples in distinct
3. Failor, S. W., Carandini, M. & Harris, K. D. Visuomotor association orthogonalizes visual hippocampal subregions. Curr. Biol. https://doi.org/10.1016/j.cub.2023.10.073 (2023).
cortical population codes. Preprint at bioRxiv https://doi.org/10.1101/2021.05.23.445338 37. Berners-Lee, A. et al. Hippocampal replays appear after a single experience and
(2022). incorporate greater detail with more experience. Neuron 110, 1829–1842 e5 (2022).
4. Schoonover, C. E. et al. Representational drift in primary olfactory cortex. Nature 594, 38. Gillespie, A. K. et al. Hippocampal replay reflects specific past experiences rather than a
541–546 (2021). plan for subsequent choice. Neuron 109, 3149–3163 (2021).
5. Marks, T. D. & Goard, M. J. Stimulus-dependent representational drift in primary visual 39. Singer, A. C. & Frank, L. M. Rewarded outcomes enhance reactivation of experience in the
cortex. Nat. Commun. 12, 5169 (2021). hippocampus. Neuron 64, 910–921 (2009).
6. Deitch, D., Rubin, A. & Ziv, Y. Representational drift in the mouse visual cortex. Curr. Biol. 40. Zutshi, I. & Buzsaki, G. Hippocampal sharp-wave ripples and their spike assembly content
31, 4327–4339 (2021). are regulated by the medial entorhinal cortex. Curr. Biol. 33, 3648–3659 (2023).
7. Clifford, C. W. et al. Orthogonal adaptation improves orientation discrimination. Vision 41. Vormstein-Schneider, D. et al. Viral manipulation of functionally distinct interneurons in
Res. 41, 151–159 (2001). mice, non-human primates and humans. Nat. Neurosci. 23, 1629–1636 (2020).
8. Karlsson, M. P. & Frank, L. M. Awake replay of remote experiences in the hippocampus. 42. Schmid, C. et al. Passive exposure to task-relevant stimuli enhances categorization
Nat. Neurosci. 12, 913–918 (2009). learning. eLife 12, RP88406 (2023).
9. Lee, A. K. & Wilson, M. A. Memory of sequential experience in the hippocampus during 43. McGuire, K. L. et al. Visual association cortex links cues with conjunctions of reward and
slow wave sleep. Neuron 36, 1183–1194 (2002). locomotor contexts. Curr. Biol. 32, 1563–1576 (2022).
10. Nadasdy, Z. et al. Replay and time compression of recurring spike sequences in the 44. Slomowitz, E. et al. Interplay between population firing stability and single neuron dynamics
hippocampus. J. Neurosci. 19, 9497–9507 (1999). in hippocampal networks. eLife 4, e04378 (2015).
11. Tang, W. et al. Hippocampal-prefrontal reactivation during learning is stronger in awake 45. Hengen, K. B. et al. Firing rate homeostasis in visual cortex of freely behaving rodents.
compared with sleep states. J. Neurosci. 37, 11789–11805 (2017). Neuron 80, 335–342 (2013).
12. Carrillo-Reid, L. et al. Imprinting and recalling cortical ensembles. Science 353, 691–694 46. Roux, L. et al. Sharp wave ripples during learning stabilize the hippocampal spatial map.
(2016). Nat. Neurosci. 20, 845–853 (2017).
13. Ji, D. & Wilson, M. A. Coordinated memory replay in the visual cortex and hippocampus 47. Grosmark, A. D. et al. Reactivation predicts the consolidation of unbiased long-term
during sleep. Nat. Neurosci. 10, 100–107 (2007). cognitive maps. Nat. Neurosci. 24, 1574–1585 (2021).
14. O’Neill, J. et al. Superficial layers of the medial entorhinal cortex replay independently of 48. van de Ven, G. M. et al. Hippocampal offline reactivation consolidates recently formed
the hippocampus. Science 355, 184–188 (2017). cell assembly patterns during sharp wave-ripples. Neuron 92, 968–974 (2016).
15. Reitich-Stolero, T. & Paz, R. Affective memory rehearsal with temporal sequences in 49. Jun, J. J. et al. Fully integrated silicon probes for high-density recording of neural activity.
amygdala neurons. Nat. Neurosci. 22, 2050–2059 (2019). Nature 551, 232–236 (2017).
16. Rothschild, G., Eban, E. & Frank, L. M. A cortical-hippocampal-cortical loop of information 50. Ego-Stengel, V. & Wilson, M. A. Disruption of ripple-associated hippocampal activity
processing during memory consolidation. Nat. Neurosci. 20, 251–259 (2017). during rest impairs spatial learning in the rat. Hippocampus 20, 1–10 (2010).
17. Shin, J. D., Tang, W. & Jadhav, S. P. Dynamics of awake hippocampal-prefrontal replay for 51. Jadhav, S. P. et al. Awake hippocampal sharp-wave ripples support spatial memory.
spatial learning and memory-guided decision making. Neuron 104, 1110–1125 (2019). Science 336, 1454–1458 (2012).
18. Sugden, A. U. et al. Cortical reactivations of recent sensory experiences predict 52. Girardeau, G. et al. Selective suppression of hippocampal ripples impairs spatial memory.
bidirectional network changes during learning. Nat. Neurosci. 23, 981–991 (2020). Nat. Neurosci. 12, 1222–1223 (2009).
19. Euston, D. R., Tatsuno, M. & McNaughton, B. L. Fast-forward playback of recent memory 53. Fauth, M. J. & van Rossum, M. C. Self-organized reactivation maintains and reinforces
sequences in prefrontal cortex during sleep. Science 318, 1147–1150 (2007). memories despite synaptic turnover. eLife 8, e43717 (2019).
20. Khodagholy, D., Gelinas, J. N. & Buzsaki, G. Learning-enhanced coupling between ripple 54. Mau, W., Hasselmo, M. E. and Cai, D. J. The brain in motion: how ensemble fluidity drives
oscillations in association cortices and hippocampus. Science 358, 369–372 (2017). memory-updating and flexibility. eLife 9, e63550 (2020).
21. Lines, J. & Yuste, R. Visually evoked neuronal ensembles reactivate during sleep. Preprint 55. Hanert, A. et al. Sleep in humans stabilizes pattern separation performance. J. Neurosci.
at bioRxiv https://doi.org/10.1101/2023.04.26.538480 (2023). 37, 12238–12246 (2017).
22. Chang, H. et al. Cortical reactivation of non-spatial and spatial memory representations 56. Miller, J. E. et al. Visual stimuli recruit intrinsically generated cortical ensembles. Proc. Natl
coordinate with hippocampus to form a memory dialogue. Preprint at bioRxiv https://doi. Acad. Sci. USA 111, E4053–E4061 (2014).
org/10.1101/2022.12.16.520658 (2022). 57. Vaz, A. P. et al. Backbone spiking sequence as a basis for preplay, replay, and default
23. Eagleman, S. L. & Dragoi, V. Image sequence reactivation in awake V4 networks. Proc. states in human cortex. Nat. Commun. 14, 4723 (2023).
Natl Acad. Sci. USA 109, 19450–19455 (2012). 58. Karlsson, M. P. & Frank, L. M. Network dynamics underlying the formation of sparse,
24. Genzel, L. et al. A consensus statement: defining terms for reactivation analysis. Philos. informative representations in the hippocampus. J. Neurosci. 28, 14271–14281 (2008).
Trans. R. Soc. Lond. B 375, 20200001 (2020). 59. Rolls, E. T. The mechanisms for pattern completion and pattern separation in the
25. Swanson, R. A. et al. Variable specificity of memory trace reactivation during hippocampus. Front. Syst. Neurosci. 7, 74 (2013).
hippocampal sharp wave ripples. Curr. Opin. Behav. Sci. 32, 126–135 (2020). 60. McClelland, J. L., McNaughton, B. L. & O’Reilly, R. C. Why there are complementary learning
26. Gupta, A. S. et al. Hippocampal replay is not a simple function of experience. Neuron 65, systems in the hippocampus and neocortex: insights from the successes and failures of
695–705 (2010). connectionist models of learning and memory. Psychol. Rev. 102, 419–457 (1995).
27. Terada, S. et al. Adaptive stimulus selection for consolidation in the hippocampus. Nature
601, 240–244 (2022). Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in
28. Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. published maps and institutional affiliations.
Science 364, 255 (2019).
29. Cooke, S. F. et al. Visual recognition memory, manifested as long-term habituation, requires Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this
synaptic plasticity in V1. Nat. Neurosci. 18, 262–271 (2015). article under a publishing agreement with the author(s) or other rightsholder(s); author
30. Frenkel, M. Y. et al. Instructive effect of visual experience in mouse visual cortex. Neuron self-archiving of the accepted manuscript version of this article is solely governed by the
51, 339–349 (2006). terms of such publishing agreement and applicable law.
31. Dana, H. et al. High-performance calcium sensors for imaging activity in neuronal
populations and microcompartments. Nat. Methods 16, 649–657 (2019). © The Author(s), under exclusive licence to Springer Nature Limited 2023

Nature | www.nature.com | 9
Article
Methods
Data reporting
No statistical methods were used to predetermine sample size. Experi-
ments did not involve experimenter blinding and were not randomized.
Mice
All animal care and experimental procedures were approved by the
Beth Israel Deaconess Medical Center Institutional Animal Care and
Use Committee. Animals were group housed before surgery and sin-
gly housed after surgery. Animals were provided ad libitum access to
standard mouse chow and water for all experiments. Animals were
kept under a 12 h–12 h dark–light cycle with the temperature of the
room ranging between 20 °C and 22 °C and the humidity of the room
ranging between 30% and 70%. We used 5 mice (4 female, 1 male) for
standard experiments, 5 mice (2 female, 3 male) for experiments that
also involved hippocampal CA1 recordings and 3 mice (2 female, 1 male)
for experiments that also involved optogenetic inhibition. All mice
were adult (older than postnatal day 56) transgenic (Emx1-cre32) mice.
All experiments were performed during the light cycle.
Behavioural training
Mice were first habituated to head fixation for 4–7 days. On the first
day, mice were head-fixed for 1 h and allowed to run on a wheel in the
dark. On the following days and during imaging experiments, mice
were head-fixed and the wheel was also fixed such that the mice could
not run, but could adjust their posture laterally. We also presented a
grey screen using a Dell 60 Hz LCD monitor positioned on the right
side of the mouse for habituation purposes. Mice were progressively
habituated for an additional 30 min each day. We continued these
daily habituation sessions until the point at which the mice remained
calm, and their eyes remained clear without physical indications of
stress for 2 h.
We then began daily recording sessions. Each day, the mice were
first presented with a grey screen for approximately 0.5 h. After this
baseline period, the mice were presented with one of two chequerboard
patterns (S1 or S2), with each square in the chequerboard containing
the same binarized white-noise video61. These S1 and S2 patterns drove
greater activity than drifting gratings (data not shown). Each stimu-
lus lasted for 2 s followed by a 58 s ITI during which a grey screen was
shown. Binarized white-noise visual stimuli and the mean luminance
grey screen during the ITI were luminance matched. Each session lasted
around 3 h and consisted of the 0.5 h baseline period followed by 64
presentations of each of the two stimuli in a random order. The visual
stimulation spanned a large part of the visual field contralateral to the
imaging window, from around −3° to 93° in azimuth and about −42° to
42° in elevation.
Surgical procedures
We followed previous surgical procedures for cranial windows62. In
brief, in anaesthetized mice, we performed a 3 mm circular crani-
otomy with a dental drill on the left lateral visual cortex (centred at
anteroposterior (AP), −4.6 mm; mediolateral (ML), −4.35 mm with
respect to bregma). We placed a 3 mm circular clear window (glued
to a 5 mm clear window on top with edges that rest on the thinned
skull) onto the brain surface. We fixed the window in place with C&B
Metabond (Parkell). Before performing viral injections of AAV(PHP.
eb)-syn-FLEX-jGCaMP7s31 (Addgene), we first waited approximately
1 week for mice to recover, brain oedema to decline and blood to clear
from below the cranial window. We then removed the window under
anaesthesia and performed 18 injections at a total of 6 sites (3 depths per
site: 200, 350 and 500 µm; speed of injection: 10–30 nl min−1; 33–100 nl
per depth), evenly spaced throughout the exposed 3-mm-diameter
circular brain surface, at various dilutions of jGCaMP7s:saline for each
mouse (1:0–1:5 for the 5 non-inhibition mice). We then replaced the
window with a new one and waited for the mice to recover for at least
1 week. For optogenetic inhibition studies, the same procedure was per-
formed but we instead injected a mixture of jGCaMP7s, S5E2-Chrimson
and saline (at a ratio of 0.75:3:3.75).
Retinotopy
We used brief epifluorescence imaging of jGCaMP7s signals to obtain
retinotopic maps34 of local preference for specific locations in visual
space to identify several visual areas: primary visual cortex, LM, POR,
LI and P. We presented low-contrast vertical gratings displayed at one
of four different retinotopic locations. A 470 nm LED passed through a
long-pass emission filter (500 nm cut-off). Images were collected using
an EMCCD camera. To determine which neurons belonged to which
visual region (P, LI, POR, LM), we manually drew region boundaries
(after alignment and resizing to a reference atlas34,63) using the roipoly
function in MATLAB. For each neuron, we estimated its centre of mass
from its spatial mask and then determined whether it was located in
the P, LI, POR or LM region.
Two-photon calcium imaging
We measured Ca2+ activity using two-photon microscopy. We used the
Insight X3 laser from Spectra-Physics to excite at 920 nm (20–40 mW).
Imaging was performed using an Olympus ×10 water-immersion
objective (0.6 NA; 796 × 512 pixels spanning an area of about
2,000 µm × 1,500 µm) on a resonance-scanning two-photon micro-
scope (Scanbox v.11.0, Optotune, and Neurolabware; near-simultaneous
imaging of three planes each spaced >20 µm apart, 31.25 Hz total imag-
ing rate for a sampling rate of 10.42 Hz for neurons at each plane). We
imaged layer 2/3 of the lateral visual cortical regions (LI, POR, P, LM
and sometimes very lateral primary visual cortex). We collected data
in 34 min runs (1 baseline run and 4 stimulus presentations runs) in a
dark and quiet room with limited entry. Mice were imaged on consecu-
tive days or every other day over several weeks (maximum duration,
less than 1 month).
Image processing and source extraction
We used suite2p64 to align, register, detect cell masks, extract Ca2+ fluo-
rescence traces and deconvolve these traces. In brief, we used non-rigid
motion correction in blocks of 128 × 128 pixels and registered each
chunk for each frame to a reference. This resulted in a phase correla-
tion of each plane compared to the reference plane. We defined ‘brain
motion’ (Extended Data Fig. 2b) as the absolute amount of shift when
aligning each image. For cell detection, suite2p decomposes the data
into a low-dimensional form and clusters to find regions of interest
(ROIs) consisting of correlated pixels. Mean fluorescence intensities are
then extracted from each ROI and the surrounding neuropil (excluding
other ROI masks). For deconvolution, we first corrected for neuropil
contamination by subtracting the mean neuropil fluorescence sur-
rounding each ROI from each ROI’s fluorescence trace using a neuropil
coefficient (scale factor) of 0.7. Fluorescence traces were then cor-
rected for long time-scale drift by subtracting a 60 s sliding window
median filter. The OASIS algorithm65 was then applied to this corrected
fluorescence trace to obtain non-negative spike deconvolution. For all
analyses, we used peak-normalized, deconvolved Ca2+ activity. Specifi-
cally, we normalized each cell’s deconvolved activity trace separately,
dividing all values by the mean of the top 1% of values. To avoid consider-
ing duplicate masks belonging to the same neuron imaged in different
planes, we first aligned the planes relative to each other using displace-
ment field estimates (imregdemons in MATLAB). After alignment, we
computed the correlation of the extracted fluorescence traces across
the entire recording for pairs of neuron masks that exhibited any x–y
overlap. If a pair of masks from different planes exhibited greater cor-
relation in their fluorescence traces than the maximum correlation of
that mask with all other masks within its own plane, one of the masks
was removed from further analyses.

Face and pupil tracking
An infrared camera was positioned below the monitor to record the
right eye of each mouse. To extract the pupil size, we manually created a
mask around the eye and selected the centre of the pupil. This was then
fit using the starburst pupil detection algorithm from the openEyes
toolkit and a ransac algorithm. Rare frames with low-quality images
of the eye were replaced with interpolated data. To match pupil states
between the baseline period and ITI period after stimulus inhibition, we
first calculated the mean pupil area during the ITI period after stimulus
inhibition (Extended Data Fig. 5f). We then sorted time frames during
the baseline period with the lowest to highest pupil area and included
an increasing number of frames until the mean pupil area during this
subset of the baseline period was equal to the mean pupil area during
the ITI period after stimulus inhibition. To track facial movements
across each session, an infrared camera was positioned on the opposite
side of the monitor to record the face of each mouse (excluding the
eye, which often was hidden by the light-shielding that surrounded
the microscope objective). We used Facemap66 (https://github.com/
MouseLand/facemap) to analyse facial videos. We manually selected
and trained using eight keypoints (nose top, nose tip, nose bottom,
whiskers I–III, mouth and lower lip) on the face of mice in 50 frames
of data per session. We then ran the Facemap algorithm, which tracks
each keypoint for all of the frames recorded, and converted the output
from units of pixels to millimetres. Facemap outputs a probability that
each keypoint is tracked correctly. We set this threshold to 75% for each
frame and linearly interpolated data between the small subset of bad
frames (probability below 75%).
Concurrent two-photon imaging of the visual cortex and
hippocampal CA1 LFP recordings
We simultaneously imaged reactivations in the lateral visual cortex
and recorded hippocampal sharp-wave ripples in a separate cohort
of five mice. To this end, we used an electrode bundle chronically
implanted into the dorsal CA1 region of the hippocampus, followed
in the same surgery by implant of a headpost implant and a cranial
window in the contralateral hemisphere, performed similarly to the
cranial window implants and jGCaMP7s injections described above
(Cre-dependent jGCaMP7s restricted to excitatory neurons in one
Emx1-cre mouse; Cre-dependent jGCaMP7s restricted to excitatory
neurons by co-injecting AAV-CamKII-Cre in one Vgat-ires-cre mouse;
and Cre-independent jGCaMP7s expressed in all cortical neurons in one
Vgat-ires-cre mouse and two Hdc-cre mice). This enabled concurrent
imaging and CA1 local field potential recordings in head-fixed mice
for weeks. The custom electrode bundle was assembled as follows: we
took four tungsten wires (CFW, CFW2043604) and bent them 90° using
tweezers at 0.6 mm, 1.5 mm, 1.6 mm and 1.7 mm from the cut tip, and
glued them together with the tips exposed. This allowed insertion into
the brain at a precise distance below the skull to ensure targeting of the
dorsal CA1 pyramidal layer (3 wires) and the dorsal cortex (1 wire). We
extended the back end of the wires 5–7 mm beyond the bend, and sol-
dered the tips of each wire to a common connector which was attached
to a headstage (RHD 64-Channel Recording Headstage) and interface
board (Intan Technologies). We also used a coated platinum-iridium
wire (A-M systems) as an external reference in the frontal cortex, and
soldered the other end to the same connector. During surgery, the
skull was exposed, levelled and dried, and a small burr hole was made
at AP −2.0 mm and ML 1.5 mm (relative to bregma), and the electrode
bundle was inserted to dorsoventral −1.25 mm. The electrodes and
wire were sealed with C&B metabond with only a small portion of the
connector exposed. Care was taken to maintain a low profile of sealant
and a horizontal orientation of the connector so that the headpost over
contralateral cortex would not encounter steric hindrance.
For recordings (using Intan software), we collected signals at 4 kHz.
Electrode signals were referenced to the electrode within the frontal
cortex. To estimate instantaneous ripple-band power, we applied a
Hilbert transform on the band-pass-filtered (150 Hz to 250 Hz) LFP,
and performed Gaussian smoothing of the square of the output
(4 ms s.d.). We chose the CA1 electrode channel with the highest ripple
power in the period before any stimuli. All traces of ripple-band power
were z scored.
In all mice, after manual inspection, we further verified that we were
recording bona fide ripple events by estimating ripple events (thresh-
old: 3 s.d.) and confirming that the ripple rate decreased with increasing
pupil area as expected. We confirmed that these ripple events were
accompanied by sharp waves in unfiltered traces (not shown). In a sub-
set of mice (n = 2), we also confirmed that the electrodes were located
within the CA1 pyramidal cell layer by post hoc histology.
Defining early and late trials in non-inhibition and
stimulus-inhibition mice
Non-inhibition mice on average had twice as many no-inhibition trials
as stimulus-inhibition mice. To account for the difference in the trial
number between these two mouse groups when combining results
and for graphical display purposes, we upsampled the traces from
stimulus-inhibition mice by 2× using polyphase filtering to match
the number of datapoints from non-inhibition mice. For both the
stimulus-inhibition and non-inhibition mice, we defined early or late
trials as approximately those in a time window of ~10 min from the start
or end of the sessions, respectively. As on average only half of the trials
in those time windows were non-inhibition trials in stimulus-inhibition
mice, we selected the first and last five trials at the start and end of a
session in stimulus-inhibition mice, and the first and last ten trials in
non-inhibition mice.
Stimulus-driven neurons
Neurons with significant evoked activity during S1 or S2 were deter-
mined using a nonparametric test (Wilcoxon rank-sum test). For each
neuron, normalized deconvolved Ca2+ activity at all timepoints in the 2 s
before stimulus presentations (pooled across all trials) was compared
to activity during the 2 s stimulus presentation periods (pooled across
all trials). Neurons with significantly (P < 0.01) increased activity for
S1 or S2 were defined as stimulus-driven neurons. We used the same
method to assess if neurons were driven either early and/or late in a
session, but used P < 0.05 as the threshold, as we used only the first
or last five trials (stimulus-inhibition mice, control trials) or 10 trials
(non-inhibition mice), resulting in lower statistical power than when
using all trials. For each stimulus-driven neuron, we also calculated
the neuron’s visual response selectivity index using the following
equation: (S1response − S2response)/(S1response + S2response). To determine the
change in a neuron’s stimulus selectivity from early to late trials, we
used a permutation approach. For each neuron, we calculated the
change in stimulus selectivity by subtracting the selectivity index
calculated for late trials from the index calculated for early trials. We
then determined whether this difference was significant by using ran-
dom permutation tests. We randomly permuted the stimulus-evoked
responses between stimuli (activity for early and late S1 and S2
trials were all randomly permuted) 1,000 times and recalculated the
change in selectivity index. Neurons for which the difference in stim-
ulus selectivity index from early versus late trials was significantly
different from the selectivity index derived from random permuta-
tions (P < 0.05, two-tailed) were determined to have a significant
difference.
Classifying stimulus reactivations
An overview of our approach for classifying stimulus reactivations is
shown in Extended Data Fig. 1d. For classification of stimulus reacti-
vation patterns, we used the normalized deconvolved Ca2+ activity
of S1 and S2 stimulus-driven neurons. We assumed that the classifier
should identify transient synchronous reactivation events lasting at
Article
least several hundreds of milliseconds (4 frames or ~380 ms), a time
scale roughly similar to that used in previous studies1,9,13,16,18. Thus, to
define brief epochs used to assess the presence of stimulus reacti-
vations, we estimated population activity patterns using the rolling
maximum activity of each cell across around 380 ms. We next removed
slow changes in activity. To remove slow changes in activity on the order
of ~1.5 s, ~6 s and ~25 s (42/sampling rate, 43/sampling rate and 44/sam-
pling rate, respectively; where the sampling rate for each activity trace
was 10.42 samples per second), we built three difference-of-Gaussian
filters by subtracting each of the three broad Gaussians separately
from the narrow Gaussian (broad Gaussians full width at half maximum
of 42, 43 and 44 samples; narrow Gaussian full width at half maximum
of 4 samples)18. We then high-pass filtered the normalized decon-
volved Ca2+ activity of stimulus-driven neurons using each of the
three difference-of-Gaussian filters. For each timepoint, we took the
minimum value across the three filtered traces to maximally remove
slow changes in activity. This resulted in a filtered, normalized, decon-
volved Ca2+ activity trace for stimulus-driven neurons that removes
slow changes in activity and preferentially preserves rapid, transient
synchronous activity.
We then built a prior for synchronous activity of stimulus-driven neu-
rons (that is, those driven by S1 and/or S2) at each timepoint. Because
the top 5% of stimulus-driven S1 or S2 neurons contained the most reli-
able information about stimulus identity (Extended Data Fig. 1h), we
found timepoints during the ITI and baseline period where the average
activity across the top 5% of stimulus-driven neurons was greater than
5 s.d. above the mean. These timepoints were used as a binary prior for
synchronous activity of stimulus-driven neurons, and we classified
only the content of stimulus reactivations within these timepoints
(Extended Data Fig. 1d).
To classify the similarity of synchronous events involving stimulus-
driven neurons during the ITI and baseline period to S1- or S2-evoked
patterns, we used multinomial logistic regression, an extension of
logistic regression. We excluded the 6 s immediately after stimulus
offset during the ITI from all classifier training and testing, to enable
activity to return close to the baseline (Extended Data Fig. 4b (right)).
We trained the classifier with three different classes of timepoints: all
timepoints during S1 presentation, all timepoints during S2 presen-
tation and all timepoints during the ITI and baseline period (other)
other than timepoints with synchronous activity of S1 or S2 neurons
mentioned above and other than the 6 s post-stimulus offset. We then
tested on all timepoints during the ITI and baseline period that exhib-
ited synchronous activity of S1 and/or S2 neurons (excluding the 6 s post
stimulus offset). This resulted in matching probability estimates that
the pattern at each timepoint matches the S1-evoked response pattern
or the S2-evoked response pattern (that is, S1 or S2 reactivation prob-
abilities), or ‘other’ patterns, with the sum of these three probabilities
equalling 1 for each timepoint.
S1-evoked and S2-evoked patterns changed across time with repeated
presentations (Figs. 3 and 4). To account for changes in stimulus repre-
sentations across a session, we built three separate classifiers so as not
to bias our final classification towards detecting reactivation patterns
that were similar only to early or late periods within a session. We split
the trials in each session into three equal chunks (from early, middle
and late periods) and trained the classifier on each chunk separately.
We then applied each classifier to all eligible timepoints during all
ITIs (or during the baseline period). For each timepoint, we compared
the performance of early, middle and late classifiers, and selected the
probabilities from the classifier with the highest matching probability
(summed across S1 and S2). If multiple classifiers had the same summed
matching probability across S1 and S2 for a given frame, we used the
classifier trained on data that included responses to nearby stimulus
presentations. Note that all of the main results held when we instead
used a single classifier trained on stimulus responses during all trials
(not shown).
We excluded any timepoints when any increase in brain motion
occurred or when pupil motion was high (>6% displacement relative
to the diameter of the eye) for the classification of reactivations.
Shuffle analyses involving the stimulus reactivation classifier
We used two shuffling methods to assess the specificity of the classifier
in identifying bona fide stimulus reactivations. In the first method, we
shuffled the neurons that defined the synchronous activity prior. As
described above, the synchronous activity prior normally uses only
the top 5% of S1-driven neurons and the top 5% of S2-driven neurons
(Extended Data Fig. 1h). In the shuffle, we chose the 5% of neurons at
random (from all neurons imaged) as the 5% of neurons that were used
to calculate the synchronous activity prior. We then classified stimulus
reactivations as we would normally, using stimulus-driven neurons but
only within synchronous time periods defined by this shuffled prior.
In the second method, we built the classifier as we normally would, as
described above. However, when we applied this classifier to identify
stimulus reactivations during synchronous periods in the ITI, we shuf-
fled the identities of all stimulus-driven neurons, thereby removing any
selectivity of the patterns for S1 or S2. We performed both shuffles ten
times and averaged the results.
Quantifying reactivation location
For each reactivation event, we determined its centroid location (using
all stimulus-driven neurons in the field of view) by multiplying each
stimulus-driven neuron’s mask location in x or y by its reactivation
activity. The centroid of each reactivation is therefore a weighted aver-
age of each stimulus-driven neuron’s location multiplied by its activity
during reactivations.
Quantifying reactivation rate and bias
Rates of reactivation of a given stimulus were calculated as the summed
matching probability of reactivations of that stimulus per second.
Reactivation bias towards S1 was calculated as the difference between
the rates of S1 and S2 reactivations, divided by the sum of S1 and S2
reactivation rates. Reactivation bias towards S2 was calculated as the
difference between the rates of S2 and S1 reactivations, divided by the
sum of S1 and S2 reactivation rates. Reactivation duration was calcu-
lated as the duration of contiguous timepoints during the reactivation
where the matching probability exceeded 0.
Classifying stimulus reactivations using subsets of neurons
To train and test the classifier using subsets of neurons, we randomly
selected a fraction of the total imaged neurons (from 10–90%, increas-
ing 10% each time) and re-ran the same classifier. We chose neurons
either completely at random from the entire field of view, or by ran-
domly selecting a neuron and then selecting neurons in a local region
defined by a disc tangential to the cortical surface and surrounding that
neuron (with increasing disc size as the percentage of all neurons was
increased). This was performed for ten iterations for each fraction of
neurons analysed. The rate of false-negative and false-positive stimulus
reactivations using fractions of the total number of imaged neurons
(Extended Data Fig. 2h,i) and the reactivation bias using a random
10% of neurons (Extended Data Fig. 3c) was averaged from the results
of the ten iterations.
Optogenetic inhibition
For optogenetic inhibition of visual stimulus-evoked activity, photo-
stimulation of Chrimson-expressing parvalbumin interneurons began
1 s before stimulus onset and ended 1 s after stimulus offset on inhibi-
tion trials, which occurred on a random 50% of all trials. The photomul-
tiplier tube (H11706-40, Hamamatsu) was gated at the beginning of each
frame for 6 ms to protect it from the LED light that was delivered for the
first 4 ms of each frame. As a result, approximately 19% of each frame
during stimulation was blanked, but allowed for simultaneous imaging
of jGCaMP7s in the majority of the field of view during photostimula-
tion (16 fps). A 617 nm LED (10 mW, Thor labs, M617L3) controlled by
a driver (Thorlabs, T-Cube) was used.
Response metric
For all analyses in Figs. 3 and 4, we used all neurons that were considered
to be stimulus-driven by either S1 and/or S2. To calculate the running
Pearson’s correlation between S1- and S2-evoked patterns, we used
the mean-normalized deconvolved Ca2+ activity for all neurons during
the entire stimulus period for all pairs of nearest-neighbour S1 and
S2 trials. Thus, running correlations were solely computed on trials
in which an S1 trial was preceded by an S2 trial or in which an S2 trial
was preceded by an S1 trial. Due to this, the ‘first trial’ (Figs. 3 and 4) in
our data occurs once both stimuli have been presented to the mouse.
The ‘last trial’ was set at trial 60 (stimulus-inhibition mice) or trial 120
(non-inhibition mice) to keep the data consistent across all sessions.
For plotting the stimulus-evoked activity and the running Pearson’s
correlation across trials (Figs. 3b–d,f,g and 4i,j and Extended Data
Figs. 6a–f, 8a,g and 10e,f), we smoothed the traces by taking a moving
mean of three trials.
Defining the increase, decrease and no-change groups of
neurons
To group neurons on the basis of their changes in stimulus-evoked
activity from early to late in a session, we first calculated the percent-
age difference in normalized deconvolved Ca2+ activity for S1 or S2
stimulus-driven neurons between the mean of the first ten trials and
the mean of the last ten trials for non-inhibition mice (we used five trials
for stimulus-inhibition mice). We had found that, for individual ses-
sions and when averaged across sessions and mice, stimulus-evoked
responses averaged across all stimulus-driven neurons did not change
across a session (Fig. 3c and Extended Data Fig. 6e,f). No-change neurons
were classified as neurons of which the percentage change in activity
was within 0.5 s.d. of the population mean change in activity across
stimulus-driven cells. Increase neurons were classified as neurons of
which the percentage change in activity was greater than 1 s.d. above
the population mean change in activity across stimulus-driven cells.
Decrease neurons were classified as neurons of which the percentage
change in activity was less than 1 s.d. below the population mean change
in activity across stimulus-driven cells. The average cut-off for increase
neurons was +74 ± 3% (1 s.d. above the mean) from early to late trials, with
a range across all sessions and mice of 17% to 323%. The average cut-off
for decrease neurons was −50 ± 2% (1 s.d. below the mean) from early to
late trials, with a range across all sessions and mice of −10% to −284%.
To determine which decrease neurons decreased for both S1 and
S2 trials equally (non-differential decrease neurons), we took all
decrease neurons and tested whether the decrease in normalized
deconvolved Ca2+ activity during the first five (stimulus-inhibition
mice) or ten (non-inhibition mice) trials versus the normalized decon-
volved Ca2+ activity during the last five (stimulus-inhibition mice) or ten
(non-inhibition mice) trials in a session was the same for both S1 and
S2 trials (P > 0.05, two-tailed paired t-test) or whether the percentage
decrease in normalized deconvolved Ca2+ activity was the same for
both S1 and S2 trials (P > 0.05, two-tailed paired t-test).
Cross-correlation analysis
For the display of correlation and cross-correlation analyses in Fig. 3i,k
and Extended Data Fig. 8i, we smoothed the traces by taking a moving
average of eight trials.
Noise correlation analysis
To calculate within-group noise correlations of stimulus-evoked
responses for pairs of no-change, increase or decrease neurons, we first
calculated the mean zero-lag correlation in stimulus-evoked responses
of all pairs within each group. For each zero-lag correlation, we
subtracted the one-trial-lag correlation of stimulus-evoked responses.
This removed the correlation due to a general increase or decrease in
activity while preserving the trial-to-trial fluctuations.
High-pass filtering
To high-pass filter trial-by-trial time series (that is, to remove slow trends
from the eight-trial moving-average traces for analyses in Fig. 3j,k and
Extended Data Fig. 8i, and for analyses in Fig. 4h (insets) and Extended
Data Fig. 10d), we used a second-order Butterworth high-pass filter
with a critical frequency of 1/20 trials (that is, removal of any slow
trends lasting around 20 trials or longer). This was sufficient to allow
for short-timescale fluctuations to pass while removing the roughly
exponential decay in the traces across the session. As the exponen-
tial decay was large early in the time series, the filter was unable to
fully remove this, and we therefore omitted the first five trials of each
session for related analyses. Note that, for stimulus-inhibition mice,
filtering and subsequent analyses were performed on the time series
of non-inhibition trials only.
Tracking neurons across days using ROICaT
ROI tracking was performed using the tracking pipeline of the ROICaT
software package (https://github.com/RichieHakim/ROICaT). ROI
masks and field-of-view images were supplied using Suite2p output
files. ROICaT’s default settings were used with the following param-
eters: automatic hyperparameter tuning was used to align fields of
view and to calculate, mix and prune pairwise ROI similarity matri-
ces. The parameter controlling the degree of pruning in the similarity
graph was slightly increased to increase cluster sizes (‘stringency’=1.3).
For clustering of the final similarity matrix, ROICaT’s recommended
method was used: if an experiment contained eight or more recorded
sessions, ROICaT uses its standard cluster fitting method based on
robust-single-linkage-clustering with the default parameters ‘min_clus-
ters’=2 and ‘alpha’=0.999. For animals with seven or fewer recorded
sessions, ROICaT’s alternative cluster fitting method based on the
sequential Hungarian method algorithm was used with ‘thesh_cost’=0.6.
The resulting clusters were inspected for quality using ROICaT’s output
quality metrics and visualization tools, and an inclusion criterion was
set using the ‘cs_sil’ metric (‘cluster similarity silhouette score’) of 0.2.
We used only the results from the first 6 days (although many mice had
more than 6 days of aligned data) to maximize the number of mice
with the same number of aligned sessions in our cross-day analysis.
Vector analysis
To calculate a vector that defined S1 stimulus responses from early to
late in a session (Fig. 4a), we took the mean stimulus response of the first
three S1 trials (early response), and of the last three S1 trials (late response),
for each of the ‘N’ stimulus-driven and reactivation-participating neu-
rons. We then calculated the N-dimensional vector that defined the evo-
lution from early to late responses as the late response vector minus the
early response vector. We then projected the single-trial mean responses
of S1 trials onto this S1 vector (scalar projection, the dot product with
the S1 vector divided by the norm of the S1 vector). We also projected
single S1 reactivations onto this S1 vector. Thus, population response
patterns that were more similar to the late response pattern than the
early response pattern exhibited projection values that were more
positive. We performed a similar procedure for the across-day vector
projection analysis. For the projection of day 1 to day 6 (Fig. 4d), we used
the mean day 1 and day 6 stimulus response across all S1 trials. We then
repeated these analyses separately for S2 trials and S2 reactivations.
We used local regression to fit a smooth curve through the scatterplot
of projected values across trials.
Scaling reactivation responses
To scale the reactivation responses to have the same magnitude as
the stimulus responses, we calculated the mean deconvolved activity
Article
of stimulus-driven neurons across all reactivations. We then divided
the mean stimulus response (averaged across these same neurons)
by the mean activity during reactivations to obtain the scale factor
for each session, and then averaged this value across sessions for
each mouse. We then averaged the per-mouse mean value across all
mice to obtain a single scale factor (1.3) that we applied to all sessions
and mice.
Modelling stimulus responses using reactivations
To model future S1 stimulus responses using S1 reactivations, we
used the actual mean response of stimulus-driven and reactivation-
participating neurons during the first S1 trial. For each subsequent
ITI, we iteratively estimated a modelled S1 response as the sum of the
previous trial’s estimated response and the difference between the
1.3×-scaled S1 reactivation pattern that occurred during the ITI and
the current S1 response pattern, multiplied by a single plasticity value
(Fig. 4g). We parametrically varied the plasticity value such that the
error was lowest and chose a single value (0.2) that we applied to all
sessions and mice (Extended Data Fig. 10a). We calculated the error
as the mean of the absolute difference between modelled and actual
S1 projections, averaged across all sessions and mice. This update
to the estimate was applied for each reactivation event in each ITI.
Thus, a greater number of reactivations during a given ITI will lead to
more iterations of the model to update the predicted S1 response, and
therefore to a faster instantaneous trial-to-trial learning rate. We then
repeated this procedure for S2 stimuli and S2 reactivations throughout
the session.
Data analysis
All analyses were performed using custom scripts in MATLAB and
Python. In all figures, the mean ± s.e.m. is shown. We performed the
Shapiro–Wilk tests on our data to test for normality. All tests (one-tailed
t-tests, two-tailed t-tests, Wilcoxon rank-sum tests, ANOVA, two-tailed
linear least-squares regression, permutation) with multiple hypotheses
were corrected for multiple comparisons using the Tukey HSD or Holm–
Bonferroni methods. For statistical tests in Fig. 2f,g and Extended Data
Figs. 3c and 5c,d, we compared the means of the two traces.
Reporting summary
Further information on research design is available in the Nature Port-
folio Reporting Summary linked to this article.
Data availability
Processed imaging and behavioural data are available online (https://
research.bidmc.harvard.edu/datashare/DataShareInfo.ASP?Submit=
Display&ID=11). Raw imaging and behavioural data are available on
request.
Code availability
Code is available at GitHub (https://github.com/nguyenr95/reactivation).
61. Liang, L. et al. Retinal inputs to the thalamus are selectively gated by arousal. Curr. Biol.
30, 3923–3934 (2020).
62. Goldey, G. J. et al. Removable cranial windows for long-term imaging in awake mice. Nat.
Protoc. 9, 2515–2538 (2014).
63. Wang, Q. & Burkhalter, A. Area map of mouse visual cortex. J. Comp. Neurol. 502, 339–357
(2007).
64. Pachitariu, M. et al. Suite2p: beyond 10,000 neurons with standard two-photon microscopy.
Preprint at bioRxiv https://doi.org/10.1101/061507 (2017).
65. Friedrich, J., Zhou, P. & Paninski, L. Fast online deconvolution of calcium imaging data.
PLoS Comput Biol. 13, e1005423 (2017).
66. Syeda, A. et al. Facemap: a framework for modeling neural activity based on orofacial
tracking. Preprint at bioRxiv https://doi.org/10.1101/2022.11.03.515121 (2022).
Acknowledgements We thank C. Harvey, B. McNaughton, S. Zhang, R. Essner, A. Lowet,
D. Tingley, A. Sugden, J. Zaremba, M. Nguyen and the members of the Andermann laboratory
for feedback; A. Sambangi for help validating AAV-S5E2-Chrimson; and K. Lensjø for advice
on AAV-PHP.eb-jGCaMP7s. This project was supported by a National Defense Science and
Engineering Fellowship and a Howard Hughes Medical Institute Gilliam Fellowship (to N.D.N.),
NIH F32 DK112589 and Davis Family Foundation awards (to A.L.), and NIH DP2 DK105570, R01
MH12343, DP1 AT010971, a McKnight Scholar Award and a Harvard Brain Science Initiative
Bipolar Disorder Seed Grant, supported by K. and L. Dauten (to M.L.A.). Icons in Figs. 1a,b,f
and 2d were created using BioRender.
Author contributions N.D.N. and M.L.A. conceived the project, designed the experiments and
analyses, and wrote the manuscript. N.D.N. performed experiments and analysed the data.
A.L. designed and helped with optogenetic silencing studies. O.A. and A.Y.-E.A. performed and
analysed the hippocampal ripple experiments. J.F. helped with surgical procedures. R.H. and
B.L.S. developed the cross-day tracking analysis. J.V., J.M. and J.D. provided the AAV-S5E2-
Chrimson-tdTomato virus.
Competing interests The authors declare no competing interests.
Additional information
Supplementary information The online version contains supplementary material available at
https://doi.org/10.1038/s41586-023-06810-1.
Correspondence and requests for materials should be addressed to Mark L. Andermann.
Peer review information Nature thanks Pieter Goltstein and the other, anonymous, reviewer(s)
for their contribution to the peer review of this work. Peer reviewer reports are available.
Reprints and permissions information is available at http://www.nature.com/reprints.
Extended Data Fig. 1 | See next page for caption.
Article
Extended Data Fig. 1 | Classifying stimulus-specific reactivations. a, Trial-
averaged, deconvolved peri-stimulus Ca2+ activity of example neurons driven
by S1, S2, or both (“S1 and S2 neurons”). S1- and S2-driven neurons exhibited
highly selective responses to their preferred stimulus. b, Quantification of
neuron count for: all neurons, S1 and S2 neurons, S1 only neurons, and S2 only
neurons (average across all trials, and separately for early and for late trials,
n = 8 mice). c, Distribution of selectivity index values (see Methods) of stimulus
driven neurons (n = 8 mice). d, Brief summary of method for classifying
reactivations (for additional details, see main text and Methods). Left (Step 1):
the classifier should identify transient synchronous reactivations that we
assume should last at least several hundred milliseconds1,9,13,16,18, and thus we
estimate population activity patterns using the rolling maximum activity of
each cell across ~380 ms. We then remove slow changes in ongoing Ca2+ activity
by using 3 difference-of-Gaussian filters to high-pass filter activity changes
at time scales of 1.5, 6, and 25 s. Middle (Step 2): we define S1 or S2 stimulus
reactivations during the inter-trial interval (in which the mouse passively views
a mean-luminance blank screen) as epochs of synchronous activity lasting
hundreds of milliseconds across neurons driven by stimulus S1 or S2,
respectively. To focus on synchronous events, we use a binary prior such that
we only classify reactivation pattern content during epochs in which the
ongoing activity trace averaged across the top stimulus-driven neurons
exceeds 5 standard deviations above the mean. Right (Step 3): we then apply
multinomial logistic regression to epochs specified by this temporal prior.
We train the classifier on time points that occur during all S1 trials, all S2 trials,
and all time points during inter-trial intervals and during the baseline period
that do not exhibit synchronous activity of stimulus-driven neurons (temporal
prior = 0). We then apply the classifier to all time points with synchronous
activity of stimulus-driven neurons during inter-trial intervals and during the
baseline period prior to any stimulus presentations (temporal prior = 1). This
results in matching probability estimates that the pattern at each time point
matches the S1-evoked response pattern, the S2-evoked response pattern
(i.e. S1 or S2 reactivation probabilities), or ‘other’ patterns, with the sum of
these three probabilities equalling 1 for each time point. e, Reactivation
duration during the baseline period before any stimulus presentations vs.
during the inter-trial intervals between stimulus presentations (n = 8 mice,
two-tailed paired t-test, P = 0.025). f, Left: distribution of reactivation
probabilities of the classifier trained using the actual data and trained using
data after shuffling using one of two different methods. The first shuffle
method defines the temporal prior using an equal number of randomly selected
neurons instead of only stimulus-driven neurons, and the second method
randomly shuffles the identity of stimulus-driven neurons (n = 8 mice; one-way
ANOVA, Holm-Bonferroni corrected, all data points below the significance
line indicate classifier probabilities that differ significantly from both
shuffled versions, P < .05). Right: fold change in density of each reactivation
probability using the actual data as compared to each shuffle (n = 8 mice).
We defined reactivation events as those with a peak probability greater than
0.75, as they were greater than three times more common than reactivations
detected in shuffled data. g, Reactivation rate during times of synchronous
stimulus activity during the ITI vs. all other ITI times with non-synchronous
stimulus activity (n = 8 mice, two-tailed paired t-test, P = 3.6 x 10 −7). Classifier
probability during the ITI was low outside of moments of synchronous
activation of stimulus-driven neurons. In this case, classification of reactivations
was performed without removing slow changes in ongoing Ca2+ activity using
the 3 difference-of-Gaussian filters to preserve all activity during the ITI. h, We
confirmed the similarity of stimulus reactivations to stimulus-evoked response
patterns by grouping neurons based on their mean response magnitude during
stimulus presentations. As expected, the neurons most strongly driven by S1 or
S2 were selectively active during S1 or S2 reactivations, respectively. Left: mean
S1-evoked activity (green) or S2-evoked activity (red) for the top 5% and bottom
95% of S1- or S2-driven neurons and for other neurons lacking stimulus-evoked
Ca2+ activity (n = 8 mice, two-tailed paired t-test, Holm-Bonferroni corrected,
from left to right: P = 0.0012, P = 1.9 × 10 −4, P = 0.0013, P = 0.0012, P = 0.42).
Middle: same as left but for mean Ca2+ activity during reactivation events
(n = 8 mice, two-tailed paired t-test, Holm-Bonferroni corrected, from left to
right: P = 0.0017, P = 6.0 x 10 −4, P = 0.50, P = 0.86, P = 0.55). Right: baseline Ca2+
activity (in the 0.5 h prior to any stimulus presentations) for the top 5% and
bottom 95% of S1- or S2-driven neurons and for other neurons lacking stimulus-
evoked Ca2+ activity (n = 8 mice). i, Fraction of neurons that remained in the top
5% of driven neurons during both early trials and late trials (n = 8 mice). Data are
mean ± SEM. n.s.: not significant; * P < .05; ** P < .01; *** P < .001; **** P < .0001.
Extended Data Fig. 2 | Characterizing stimulus reactivations. a, Mean
hippocampal ripple-band power surrounding the onset of classified
reactivations (derived from the cortical imaging data) across each session for
all mice (n = 14 sessions from 5 mice that differed from the mice used for any
other analyses). SD: standard deviations above the mean. b, Brain motion plotted
surrounding the onset of classified reactivations (n = 8 mice, two-tailed paired
t-test, P = 0.19). c, Phase correlation to the reference frame, plotted surrounding
the onset of classified reactivations (n = 8 mice, two-tailed paired t-test, P = 0.011).
The phase correlation measures how well each individual frame correlates with
the reference frame used for motion correction. d, Peak-normalized pupil
movement (absolute change in movement) plotted surrounding the onset of
classified reactivation events (n = 8 mice, two-tailed paired t-test, P = 0.47).
e, Comparison of mean stimulus-evoked activity (left) or stimulus reactivation
activity (right) between neurons located in upper layer 2/3 (~ 156 µm from the
brain surface) vs. lower layer 2/3 (~ 266 µm from the brain surface) of lateral
visual cortex (n = 8 mice, two-tailed unpaired t-test, stimulus: P = 0.89,
reactivation: P = 0.13). f, Change in mean location (centroid, estimated using
each stimulus-driven neuron’s activity during reactivations) of stimulus
reactivations across the session along the anterior-posterior (left) and
lateral-medial axes (right, n = 8 mice, two-tailed paired t-test, Holm-Bonferroni
corrected, left: S1: P = 0.035, S2: P = 0.035, right: S1: P = 0.060, S2: P = 0.065).
g, Raster plot of ongoing deconvolved Ca2+ activity of the top stimulus-driven
neurons during and following an example S1 stimulus presentation (green
square) and an example S2 stimulus presentation (red square), using all neurons
or using a random 10% of neurons (see lower raster). Classification of stimulus
reactivations using a random 10% of neurons results in several false positive
(blue arrows) and false negative (magenta arrow) classification errors when
compared to using all neurons. Inset at right: expanded view of data from green
rectangle, illustrating a false-positive classification using a random 10% of
neurons. h, Percent of false negative (left) or false positive (right) classifications
of reactivations relative to reactivations classified using all neurons, plotted as
a function of the percent of randomly selected neurons used in the classifier
(n = 8 mice, permutation test, Holm-Bonferroni corrected, P < .05 for all tests).
i, Same as h but selecting neurons randomly from the same subregion of the
field of view such that they are all close in distance (see Methods, n = 8 mice,
permutation test, Holm-Bonferroni corrected, P < .05 for all tests). Data are
mean ± SEM. n.s.: not significant; * P < .05.
Article
Extended Data Fig. 3 | Reactivation rate and bias effects are consistent
across sessions and correlate with stimulus novelty and pupil-indexed
arousal. a, Left: reactivation rates (sum of probabilities of S1 or S2 reactivations)
across each session, including the 0.5-hour baseline period prior to any
stimulus presentations for all daily sessions (n = 5 mice, 48 sessions total).
Right: reactivation rate during the inter-trial interval (n = 5 mice, 48 sessions
total) for all daily sessions. b, Left: bias index of reactivation content (positive
values indicate bias towards the most recent stimulus, n = 5 mice, 48 sessions
total) for all daily sessions. Right: bias throughout the inter-trial interval
(n = 5 mice, 48 sessions total) for all daily sessions. c, Reactivation content bias
during stimulus presentations across the session using all neurons vs. a random
10% of neurons (n = 8 mice, permutation test between mean of traces, P = 0.0016).
d, Stimulus reactivation rates when the stimulus on the preceding trial was
different vs. when it was the same as on the current trial (n = 8 mice, one-tailed
t-test vs. 0, P = 9.8 x 10 −4). e, Correlation between pupil area during stimulus
presentation and stimulus reactivation rate during the subsequent ITI (n = 8
mice, one-tailed t-test vs. 0, P = 0.026). f, Correlation between stimulus activity
during stimulus presentation and stimulus reactivation rate during the
subsequent ITI (n = 8 mice, one-tailed t-test vs. 0, P = 0.033). Data are mean ± SEM.
n.s.: not significant; * P < .05; ** P < .01; *** P < .001.
Extended Data Fig. 4 | Physical correlates of arousal remain constant
throughout the session. a, Left: peak-normalized pupil area during stimulus
presentations across trials (n = 8 mice, two-tailed paired t-test, P = 0.056). Right:
brain motion during stimulus presentations across trials (n = 8 mice, two-tailed
paired t-test, P = 0.48). Coloured lines: individual mice. Black line: mean across
mice. b, Left: Ca 2+ activity during the baseline period before any stimulus
presentation (dark shaded region) and during inter-trial intervals between
stimulus presentations (n = 8 mice, two-tailed paired t-test, P = 0.016, two-tailed
linear least-squares regression, P = 0.018, Holm-Bonferroni corrected). Right:
Ca2+ activity during stimulus presentation and throughout the inter-trial
interval (n = 8 mice, two-tailed linear least-squares regression, P = 1.2 x 10 −19).
Dark shaded region: stimulus presentation. Light shaded region: excluded
portion of inter-trial interval. Coloured lines: individual mice. Black line: mean
across mice. c, Top: peak-normalized pupil area during the baseline period
before any stimulus presentation and during inter-trial intervals between
stimulus presentations (n = 8 mice, two-tailed paired t-test, P = 0.019, two-
tailed linear least-squares regression, P = 0.092, Holm-Bonferroni corrected).
Bottom: peak-normalized pupil area during stimulus presentation and
throughout the inter-trial interval (n = 8 mice, two-tailed linear least-squares
regression, P = 1.7 × 10 −6). Dark shaded region: stimulus presentation. Light
shaded region: excluded portion of inter-trial interval. Coloured lines:
individual mice. Black line: mean across mice. d, Left: example image from a
recording of the mouse’s face during imaging. Each coloured dot denotes a
keypoint on the face that was tracked across each session. Right: example
traces of 8 tracked keypoints on the nose, whiskers, and mouth (nose top, nose
tip, nose bottom, whiskers I-III, mouth, and lower lip). Traces are in units of
absolute movement. e, Absolute movement of 8 tracked keypoints (nose top,
nose tip, nose bottom, whiskers I-III, mouth, and lower lip) during the baseline
period before any stimulus presentation and during 2.5 h of stimulus
presentations (n = 3 mice, two-tailed linear least-squares regression, nose top:
P = 0.39, nose tip: P = 0.54, nose bottom: P = 0.51, whisker I: P = 0.21, whisker II:
P = 0.18, whisker III: P = 0.40, mouth: P = 0.19, lower lip: P = 0.70). Data are
mean ± SEM. n.s.: not significant; * P < .05; **** P < .0001.
Article
Extended Data Fig. 5 | Characterizing the effects of peri-stimulus inhibition.
a, Coronal sections of visual cortex displaying virally-mediated expression of
Cre-dependent jGCaMP7s in glutamatergic neurons (green, in Emx1-Cre mice)
and Chrimson in parvalbumin interneurons (red, S5E2 enhancer) in 3 mice.
Local injections ensured targeted expression of Chrimson throughout lateral
visual cortical areas in all 3 mice. b, Left: stimulus-evoked Ca2+ activity during
control vs. stimulus-inhibition trials (n = 3 mice, two-tailed paired t-test,
P = 0.0056). Right: percent reduction in Ca2+ activity on stimulus-inhibition
trials compared to control trials (n = 3 mice, one-sample t-test vs. 0, P = 0.0022).
For stimulus-inhibition trials, we pulsed 10 mW of red light for 4 ms at 16 Hz
from 1 s before stimulus onset to 1 s after stimulus offset. c, Peak-normalized
pupil area during stimulus presentation and during the inter-trial interval for
control vs. stimulus-inhibition trials (n = 3 mice, two-tailed paired t-test
between mean of traces during the stimulus period plus the period immediately
following the stimulus, P = 0.75, or during the specified inter-trial interval,
P = 0.76, Holm-Bonferroni corrected). Red horizontal bar at top indicates
timing of optogenetic silencing. Noise bars indicate time of visual stimulus.
Grey shaded area indicates post-stimulus period excluded from reactivation
analyses. d, Ca2+ activity during stimulus presentation and during the inter-trial
interval for control vs. stimulus-inhibition trials (n = 3 mice, two-tailed paired
t-test between mean of traces during the specified inter-trial interval, P = 0.19).
Red horizontal bar at top indicates timing of optogenetic silencing. Noise bars
indicate time of visual stimulus. Grey shaded area indicates post-stimulus
period excluded from reactivation analyses. e, Reactivation rate during the
inter-trial interval for all stimulus-inhibition trials that were proceeded by a
control trial (n = 3 mice, two-tailed linear least-squares regression, Holm-
Bonferroni corrected, control trials: P = 0.016, stimulus-inhibition trials:
P = 0.38). f, Pupil size-matched reactivation rate during the baseline period
before stimulus presentation and during the ITI following stimulus-inhibition
trials (n = 3 mice, two-tailed paired t-test, P = 0.018). Times were selected such
that the mean pupil size between the two conditions was identical (see Methods).
g, Bias index of reactivation content during the inter-trial interval for all stimulus-
inhibition trials that were proceeded by a control trial (n = 3 mice, two-tailed
linear least-squares regression, Holm-Bonferroni corrected, control trials:
P = 0.40, stimulus-inhibition trials: P = 0.66). Here, the bias index is calculated
throughout as the bias in reactivation content towards the stimulus presented
on the control trial (black vertical line). h, Mean reactivation rate for non-
inhibition mice (n = 5 mice) across all trials and for stimulus-inhibition mice
(n = 3 mice) across all trials or control (no-inhibition) trials (two-tailed unpaired
t-test, Holm-Bonferroni corrected, non-inhibition mice all trials vs. stimulus-
inhibition mice all trials: P = 0.11, non-inhibition mice all trials vs. stimulus-
inhibition mice control trials: P = 0.019). Data are mean ± SEM. n.s.: not
significant; * P < .05; ** P < .01.
Extended Data Fig. 6 | Compared to non-inhibition mice, stimulus-inhibition
mice exhibit similar stimulus response orthogonalization but higher
response magnitudes during control trials. a, Response similarity shown
separately for non-inhibition mice (n = 5 mice) and stimulus-inhibition mice
(n = 3 mice; using no-inhibition control trials only). b, Change in response
similarity (same as a but after subtracting the mean response similarity in first
3 trials) shown separately for non-inhibition mice (n = 5 mice) and stimulus-
inhibition mice (n = 3 mice). c, Heatmap of change in response similarity shown
separately for non-inhibition mice (n = 5 mice) and stimulus-inhibition mice
(n = 3 mice) for all sessions. d, Response similarity using neurons located in
upper layer 2/3 or lower layer 2/3 (n = 8 mice, two-tailed unpaired t-test between
the mean of the first or last 3 datapoints of the two traces, Holm-Bonferroni
corrected, first: P = 0.74, last: P = 0.85). e, Mean stimulus-evoked activity per
trial (across all S1 and S2 trials) averaged across all stimulus-driven neurons
shown separately for non-inhibition mice (n = 5 mice) and stimulus-inhibition
mice (n = 3 mice; using control trials only). f, Heatmap of mean stimulus-evoked
activity shown separately for non-inhibition mice (n = 5 mice) and stimulus-
inhibition mice (n = 3 mice) for all sessions. g, Percent of decrease neurons that
showed a similar decrease in response to both S1 and S2 (defined as a similar
drop in Ca2+ events/second and/or a similar proportional drop in response
magnitude to S1 and S2, n = 8 mice). h, Response similarity when using all
neurons or when omitting non-differential decrease neurons (n = 8 mice,
permutation test between the mean of the first or last 3 datapoints of the two
traces, Holm-Bonferroni corrected, first: P = 0.73, last: P = 0.79). Data are
mean ± SEM. n.s.: not significant.
Article
Extended Data Fig. 7 | Tracking the same neurons across days. a, Left:
example zoom-in of the same subregion of a field of view across six days of
imaging. Green, orange, purple, and red boxes highlight the same neurons
tracked across all six days. Right: example neurons and masks tracked
across six days of imaging, colour-matched to the neurons outlined in the
left panel. b, The number of neurons tracked across all six days of imaging
(n = 5 non-inhibition mice). c, Change in response similarity from the end of
the previous day to the start of the next day across six days of imaging (n = 5
non-inhibition mice, two-tailed paired t-test, P = 0.42). Positive values indicate
an increase in response similarity, reflecting a partial relapse in response
similarity. Data are mean ± SEM. n.s.: not significant.
Extended Data Fig. 8 | Characterization of no-change, increase, and
decrease neurons. a, Mean stimulus-evoked activity across trials shown
separately for non-inhibition mice (n = 5 mice) and stimulus-inhibition mice
(n = 3 mice, control trials only) for no-change (left), increase (middle), or
decrease (right) neuron groups. b, Numbers of neurons that are characterized
as no-change, increase, or decrease neurons (n = 8 mice). c, Baseline activity
before any stimulus presentation for no-change, increase, and decrease
neurons (n = 8 mice, one-way ANOVA, Tukey HSD corrected, P > .05 for all tests).
d, Percent of all neurons in visual region LI, POR, P, or LM that were defined as
no-change, increase, or decrease neurons (n = 8 mice, one-way ANOVA, Tukey
HSD corrected, P > .05 for all tests). e, Percent of all neurons in upper layer 2/3
or in lower layer 2/3 that were characterized as no-change, increase, or
decrease neurons (n = 8 mice, two-tailed unpaired t-test, Holm-Bonferroni
corrected, P > .05 for all tests). f, Within-group noise correlation (see Methods)
of no-change, increase, or decrease neurons (n = 8 mice, one-way ANOVA,
Tukey HSD corrected, P > .05 for all tests). g, Response similarity (running
correlation between response patterns during neighbouring S1 and S2 trials),
plotted in the same manner as the mean trace in Fig. 3g for increase neurons
(left) and decrease neurons (right), but shown separately for non-inhibition
mice (n = 5 mice) and stimulus-inhibition mice (n = 3 mice). h, Fraction of
neurons that increase or decrease their stimulus selectivity (selectivity index:
(S1response – S2response) / (S1response + S2response); see Methods) from early to late trials
for no-change, increase, or decrease neurons, shown separately for non-
inhibition mice (n = 5 mice) and stimulus-inhibition mice (n = 3 mice). i, Cross-
correlation between high-pass filtered (see Methods) response similarity and
reactivation probability traces shown separately for non-inhibition mice
(n = 5 mice) and stimulus-inhibition mice (n = 3 mice). Data are mean ± SEM.
n.s.: not significant.
Article
Extended Data Fig. 9 | Stimulus reactivations consistently predict future
stimulus responses. a, For each trial, we projected single-trial response
patterns (during S1 or S2) and stimulus-specific reactivations during the inter-
trial interval (S1R or S2R) onto the axis between early and late stimulus-evoked
response patterns within a session (see Fig. 4a, b for additional graphical
details). Here, data from a typical example session is shown. b, Same as a but for
the mean across all sessions and mice shown separately for non-inhibition mice
(n = 5 mice) and stimulus-inhibition mice (n = 3 mice). c, Same as Fig. 4b but
shown separately for each of the first six days of imaging per mouse, using all
neurons that were tracked across the six days (n = 5 non-inhibition mice). The
change in overall y-axis offset per day is not meaningful in this case since each
projection uses a different projection axis estimated on each day ‘i’, i = 1–6.
d, Same as Fig. 4b but using only neurons from upper layer 2/3 or from lower
layer 2/3 (n = 8 mice, permutation test between the mean of the first or last 3
datapoints of the upper layer vs. lower layer traces, Holm-Bonferroni corrected,
P > .05 for all tests). e, Same as Fig. 4b but after removing non-differential
decrease neurons (see Extended Data Fig. 6g, n = 8 mice, permutation test
between the mean of the first or last 3 datapoints of the traces generated using
data from all neurons vs. from all neurons after removing non-differential
decrease neurons, Holm-Bonferroni corrected, P > .05 for all tests). f, Ratio of
mean activity across trials early in each session in increase (top) or decrease
(bottom) neurons relative to no-change neurons during S1 presentations vs.
S1R reactivation events, and during S2 presentations vs. S2R reactivation
events, shown separately for non-inhibition mice (n = 5 mice) and stimulus-
inhibition mice (n = 3 mice). g, Stimulus-evoked activity vs. reactivation activity,
averaged across all stimulus-driven neurons (n = 8 mice). h, Difference between
a neuron’s 1.3x-scaled peri-reactivation activity and its peri-stimulus activity
early in each session, averaged across neurons in each group, and shown
separately for non-inhibition mice (n = 5 mice) and stimulus-inhibition mice
(n = 3 mice). i, Difference between a neuron’s 1.3x-scaled peri-reactivation
activity and its peri-stimulus activity early in each session for S1 and S2 trials for
neurons that change their stimulus preference (from early to late trials), either
from being driven only by S1 to being driven only by S2 (top) or from being
driven only by S2 to being driven only by S1 (bottom, n = 8 mice, two-tailed
paired t-test, top: P = 3.7 × 10 −4, bottom: P = 0.0034). Data are mean ± SEM. n.s.:
not significant; ** P < .01; *** P < .001.
Extended Data Fig. 10 | Modelling future stimulus responses using only
stimulus reactivations. a, We parametrically varied the plasticity variable γ
and measured the error in the modelled stimulus-evoked response patterns vs.
actual stimulus-evoked response patterns (mean of the absolute difference
between actual and modelled data). The value of 0.2 had the least mean error
for both S1 and S2 (n = 8 mice). We used this same value for modelling all
sessions and mice. This value is likely an overestimate as we do not consider
plasticity that occurs during the stimulus response period. b, Comparison of
projection of the actual stimulus-evoked response patterns (dark green/red
dots and smoothed trace) with the modelled patterns (light green/pink dots
and smoothed trace), projected onto Vs for a single session. c, Comparison of
projection of the actual stimulus-evoked response patterns with the modelled
patterns, projected onto Vs across all days and mice, shown separately for
non-inhibition mice (n = 5 mice) and stimulus-inhibition mice (n = 3 mice).
d, Cross-correlation between high-pass filtered actual and modelled
projections of stimulus-evoked response patterns shown separately for
non-inhibition mice (n = 5 mice) and stimulus-inhibition mice (n = 3 mice,
see Methods). e, Response similarity as measured by the correlation between
the mean response patterns during nearby S1 and S2 trials, plotted for actual
and modelled stimulus responses, shown separately for non-inhibition mice
(n = 5 mice) and stimulus-inhibition mice (n = 3 mice). f, Response similarity
using modelled data, for increase or decrease neuron groups, shown separately
for non-inhibition mice (n = 5 mice) and stimulus-inhibition mice (n = 3 mice).
As in Fig. 3g, only decrease neurons show orthogonalization across trials, and
this was evident in both sets of mice.
 nature portfolio | reporting summary
Corresponding author(s): Mark Andermann
Last updated by author(s): Oct 10, 2023

Reporting Summary
Nature Portfolio wishes to improve the reproducibility of the work that we publish. This form provides structure for consistency and transparency
in reporting. For further information on Nature Portfolio policies, see our Editorial Policies and the Editorial Policy Checklist.

Statistics
For all statistical analyses, confirm that the following items are present in the figure legend, table legend, main text, or Methods section.
n/a Confirmed
The exact sample size (n) for each experimental group/condition, given as a discrete number and unit of measurement
A statement on whether measurements were taken from distinct samples or whether the same sample was measured repeatedly
The statistical test(s) used AND whether they are one- or two-sided
Only common tests should be described solely by name; describe more complex techniques in the Methods section.

A description of all covariates tested

A description of any assumptions or corrections, such as tests of normality and adjustment for multiple comparisons
A full description of the statistical parameters including central tendency (e.g. means) or other basic estimates (e.g. regression coefficient)
AND variation (e.g. standard deviation) or associated estimates of uncertainty (e.g. confidence intervals)

For null hypothesis testing, the test statistic (e.g. F, t, r) with confidence intervals, effect sizes, degrees of freedom and P value noted
Give P values as exact values whenever suitable.

For Bayesian analysis, information on the choice of priors and Markov chain Monte Carlo settings
For hierarchical and complex designs, identification of the appropriate level for tests and full reporting of outcomes
Estimates of effect sizes (e.g. Cohen's d, Pearson's r), indicating how they were calculated
Our web collection on statistics for biologists contains articles on many of the points above.

Software and code

Policy information about availability of computer code
Data collection Two-photon calcium imaging and facial data were collected with Scanbox version 11.0 (Neurolabware) in MATLAB.

Data analysis All data analysis was performed using MATLAB or Python. Analysis code is available at https://github.com/nguyenr95. Tracking neurons across
days code is available at https://github.com/RichieHakim/ROICaT.
For manuscripts utilizing custom algorithms or software that are central to the research but not yet described in published literature, software must be made available to editors and
reviewers. We strongly encourage code deposition in a community repository (e.g. GitHub). See the Nature Portfolio guidelines for submitting code & software for further information.

Data
Policy information about availability of data
All manuscripts must include a data availability statement. This statement should provide the following information, where applicable:
- Accession codes, unique identifiers, or web links for publicly available datasets
- A description of any restrictions on data availability
- For clinical datasets or third party data, please ensure that the statement adheres to our policy
April 2023

Processed imaging and behavioral data is available at https://research.bidmc.harvard.edu/datashare/DataShareInfo.ASP?Submit=Display&ID=11. Raw imaging and
behavioral data is available upon request.

1
 nature portfolio | reporting summary
Research involving human participants, their data, or biological material
Policy information about studies with human participants or human data. See also policy information about sex, gender (identity/presentation),
and sexual orientation and race, ethnicity and racism.
Reporting on sex and gender N/A

Reporting on race, ethnicity, or N/A

other socially relevant
groupings

Population characteristics N/A

Recruitment N/A

Ethics oversight N/A

Note that full information on the approval of the study protocol must also be provided in the manuscript.

Field-specific reporting
Please select the one below that is the best fit for your research. If you are not sure, read the appropriate sections before making your selection.

Life sciences Behavioural & social sciences Ecological, evolutionary & environmental sciences
For a reference copy of the document with all sections, see nature.com/documents/nr-reporting-summary-flat.pdf

Life sciences study design

All studies must disclose on these points even when the disclosure is negative.
Sample size No statistical methods were used to predetermine sample size but our sample sizes were similar to or larger than previous related studies and
minimized the number of experimental animals for ethical considerations.

Data exclusions We excluded data in which the targeting of the window did not mostly (>50%) cover visual cortex because our study focused on this region.
This criterion was predetermined.

Replication For all experiments, each individual mouse was a replicate (n = 13 mice in total). We replicated the imaged region (lateral visual cortex),
behavior, stimulus presentation protocol, and analysis for all mice. We imaged each mouse every day or every other day for 2-14 days.

Randomization Randomization was used when possible. We randomized the sex of mice used for initial surgery. Randomization was not possible for which
mice was used for imaging as the surgery had to meet criteria for expression of jGCaMP7s in thousands of neurons across lateral visual cortex.
Randomization was not possible for analysis of each mouse as it was completed successively. We used random shuffles in data analysis as
needed.

Blinding Experiments did not involve experimenter blinding. The nature of the experiments did not allow for blinding with one experimenter.

Reporting for specific materials, systems and methods

We require information from authors about some types of materials, experimental systems and methods used in many studies. Here, indicate whether each material,
system or method listed is relevant to your study. If you are not sure if a list item applies to your research, read the appropriate section before selecting a response.

Materials & experimental systems Methods

n/a Involved in the study n/a Involved in the study
Antibodies ChIP-seq
Eukaryotic cell lines Flow cytometry
Palaeontology and archaeology MRI-based neuroimaging
Animals and other organisms
Clinical data
April 2023

Dual use research of concern

Plants

2
Animals and other research organisms

nature portfolio | reporting summary

Policy information about studies involving animals; ARRIVE guidelines recommended for reporting animal research, and Sex and Gender in
Research

Laboratory animals All animals were adult (> P56) transgenic mice. Animals were on a 12-hour dark/light cycle with temperature ranging between 20-22
degrees Celsius and humidity ranging between 30-70%. Strains: Emx1-cre, The Jackson Laboratory; HDC-Cre, The Jackson Laboratory;
Vgat-ires-Cre, The Jackson Laboratory.

Wild animals No wild animals were used in this study.

Reporting on sex We used 13 mice (8 female, 5 male) in this study. Sex was assigned. Sex was not considered relevant in this study.

Field-collected samples No field-collected samples were used in this study.

Ethics oversight All animal care and experimental procedures were approved by the Beth Israel Deaconess Medical Center Institutional Animal Care
and Use Committee.
Note that full information on the approval of the study protocol must also be provided in the manuscript.

April 2023