Available online at www.sciencedirect.
com
ScienceDirect
Oscillations, neural computations and learning during
wake and sleep
Hector Penagos, Carmen Varela and Matthew A Wilson
Learning and memory theories consider sleep and the
reactivation of waking hippocampal neural patterns to be crucial
for the long-term consolidation of memories. Here we propose
that precisely coordinated representations across brain regions
allow the inference and evaluation of causal relationships to train
an internal generative model of the world. This training starts
during wakefulness and strongly benefits from sleep because its
recurring nested oscillations may reflect compositional
operations that facilitate a hierarchical processing of
information, potentially including behavioral policy evaluations.
This suggests that an important function of sleep activity is to
provide conditions conducive to general inference, prediction
and insight, which contribute to a more robust internal model
that underlies generalization and adaptive behavior.
Address
Center for Brains, Minds and Machines, Picower Institute for Learning
and Memory, Department of Brain and Cognitive Sciences,
Massachusetts Institute of Technology, Cambridge, MA, USA
Corresponding author: Wilson, Matthew A (mwilson@mit.edu)
Current Opinion in Neurobiology 2017, 44:193–201
This review comes from a themed issue on Neurobiology of sleep
Edited by Yang Dan and Thomas Kilduff
For a complete overview see the Issue and the Editorial
http://dx.doi.org/10.1016/j.conb.2017.05.009
0959-4388/ã 2017 Elsevier Ltd. All rights reserved.
Introduction
The challenge faced by the brain in perceiving and
interpreting the external world is to map a high-dimen-
sional input into neural representations in the form of
distributed spiking activity across brain regions, and to
then infer causal relationships behind this sensory-driven
code. A consequence of this inference is the generation of
actions that allow an optimal interaction with the envi-
ronment in different contexts. Computationally, the com-
plexity of this challenge renders solutions that rely on
discriminative methods and lookup tables as rigid and
inefficient. Instead, solutions based on probabilistic gen-
erative models aim to learn the underlying rules behind
external world observations, allowing more general appli-
cability [1,2]. For instance, in the case of spatial explora-
tion, using a lookup table constructed on past navigational
experiences, subjects would rely on trial and error
www.sciencedirect.com
exploration to discover different courses when familiar
routes are not accessible. By contrast, using a map as a
model, subjects could gain in generalization and flexibil-
ity because they would be able to devise, evaluate and
plan alternative routes without previously having experi-
enced them, as predicted by the cognitive map hypothe-
sis [3,4]. Importantly, knowledge of the spatial layout can
have limited value when planning and executing a plan,
because a map does not consider conditions and rules that
may govern navigation in a given environment. Answer-
ing questions such as what paths are available and why
they are accessible or not is necessary for an individual to
decide how to execute a plan to reach a goal. Hence,
incorporating information to answer these what, why and
how questions can lead to a more robust model that
generates appropriate actions under varying requirements
and contexts. Training such a generative model relies on
extracting meaningful structure from its inputs to learn
statistical representations that can account for the broad
set of conditions associated with them [5]. For neural
systems this training could start with the encoding of
external information during awake behavior and continue
during periods of sleep, resulting in more robust repre-
sentations that increase behavioral flexibility [6]. Indeed,
several studies demonstrate that sleep, in addition to
promoting memory consolidation, enables the discovery
of implicit rules and insights, which are essential ele-
ments for generalization and learning [7]. What compu-
tational principles are at work during sleep to facilitate the
consolidation of memories while also promoting generali-
zation as well as the inference of causal relationships?
While a dialogue between the neocortex and hippocam-
pus has been thought to mediate the systems consolida-
tion of memory and the slow incorporation of statistical
regularities into general cortical schemas [8,9,10], it is
unclear whether or how it could also contribute to the
training of a generative model that infers causal relation-
ships. In the following sections, we explore the idea that,
in the case of spatial navigation, the coordinated neural
representations in the hippocampus, neocortex and thal-
amus during sleep, train a generative model that infers
contextual and spatial contingencies, and which can be
used during navigation to flexibly select actions to meet
contextual conditions.
Predictive coding and neural network
representations
Machine learning methods can provide helpful theoreti-
cal frameworks for the implementation and training of
flexible generative models in the brain. Although deep
Current Opinion in Neurobiology 2017, 44:193–201
194 Neurobiology of sleep

architectures have enjoyed a recent wave of success
[11,12], they largely require explicit external feedback
during training, which contrasts with how the brain learns
despite the lack of external feedback. Generative net-
works, by contrast, while typically not structurally deep,
can extract statistical patterns in an unsupervised manner
[13]. These types of models cast the brain as an inference
device that compares predictions generated by an internal
model of the world against the ongoing spike train code.
Perception, for example, is then a constructive process by
which the brain continuously tries to account for its
sensations in terms of internally generated expectations
or, when a mismatch occurs, updates the model that
generates the predictions [14,15]. Stochastic recurrent
neural networks offer a formal implementation that is
consistent with this type of predictive coding because
they allow the sequential estimation of probabilistic
relationships between time-dependent random variables
through generative models. The temporal restricted
Boltzmann machine (TRBM) [16,17], a forerunner to
modern recurrent neural networks, offers an intuition of
how this process unfolds. The TRBM is composed of a
sequence of individual RBMs [18] (Box 1) each of which
contains two sets of stochastic binary units. A layer of
visible units receives the input to the RBM and is linked
to a set of hidden units through connection weights in
such a way that the state of the units and the strength
of their connections can, through training, extract and
encode the statistical regularities of the inputs. Using this
computational design as a conceptual framework, even
without a specific correspondence with detailed brain
anatomy, can provide insights about how neocortical,
hippocampal and thalamic networks act together to
implement and train a generative model of the world
that could be used for flexible behavior. For example,
during goal-directed navigation, a subject would benefit
from knowing the upcoming spatial layout of the envi-
ronment and the rules affecting potential choices. In
rodents, the spatial component is given by the anticipa-
tory firing of CA1 place cells within cycles of the ongoing
theta rhythm (8–12 Hz) in which place cells with partially
overlapping receptive fields fire in a temporal sequence
that reflects their relative position on the maze [19,20].
These theta sequences represent the upcoming position
of the animal and, at decision points, reveal sweeps in the
direction of all available options, consistent with an
active, constructive evaluation of potential choices [21].
Neocortical areas, including prefrontal and retrosplenial
cortices, could represent the rules and actions that ulti-
mately impact the decision of the subject [22–24]. Con-
sistent with this cooperative interaction, prefrontal neu-
rons are selectively phased locked to the hippocampal
theta rhythm as subjects approach decision points [25,26].
An additional structure that could assist in coordinating
consistent expectations and representations across brain
areas is the thalamus, given its role in the expression of
intended navigational trajectories in prefrontal cortex and
hippocampus [27], its projections to multiple neocortical
regions and its potential role in modulating the hippo-
campal theta oscillation [28,29]. How is this complex
network, involving multiple brain regions, trained to
support predictive coding?
Training a generative model during awake
behavior
Predictive coding relies on correcting errors resulting
from comparisons between internal predictions and actual
observations. This error, estimated through a process

Box 1 Restricted Boltzmann Machine as a computational model for learning across brain states

The stochastic nature of RBMs allows the estimation of the statistical structure implicit in its inputs in an unsupervised way mimicking the challenge
faced by the brain in interpreting the outside world. (a) In the RBM formulation, a set of binary visible units interacts directly with incoming stimuli
and is linked to a set of binary hidden units; the state of the units and the strength of their connections, represented in a connection matrix, can
encode statistical regularities of the input [49]. Computationally, this requires a two-step optimization heuristic known as contrastive divergence: in
the first (encoding) step, the visible units are fixed to represent a sample of the external input and the state of the hidden units is obtained and
expressed as a hidden vector; in the second (prediction) step, the hidden units are clamped to the recently obtained hidden vector to generate a
prediction quantified by casting the value of the visible units into a visible vector. The difference between the joint state of visible and hidden units
during the encoding and predictive steps provides an update to the connection matrix so that the internal model better approximates the training
sample [29]. Intuitively, the weight between two coactive units will increase in the encoding step, similar to a Hebbian rule, to encourage the hidden
units to model the incoming stimulus, whereas the weight between coactive units in the prediction step will decrease to minimize nonspecific
correlations. The temporal RBM (TRBM, (b) top) is an extension of the RBM that is useful in representing sequential data. At each time step, an
RBM corrects its prediction based on the ongoing sample of the external stimulus, and it provides the initial conditions for the hidden units in the
immediately following time step. Physiologically, this could correspond to the encoding and retrieval phases associated with the hippocampal theta
oscillation; in a given theta cycle, the state of the environment would be communicated through the feedforward connections from entorhinal cortex
(EC), allowing for a contrastive divergence-like operation by the recurrent CA3 network, while the expectation of the model, in the retrieval phase,
would be reflected by CA1 spiking activity. As an outcome, this process would set up the next expectation of the internal model as navigation
unfolds.
The recurrent TRBM (RTRBM, (b) bottom) provides a mathematical advantage over the TRBM because, at any given time step, the state of the
hidden units is obtained by a deterministic operation based on the state of an intermediary hidden layer (H’) at the previous time step along with the
current state of the visible layer. Although mapping the exact equivalence between each component of the RTRBM to physiological elements is
difficult, the temporal progression of the RTRBM suggests an analogy to the coordinated activity of the hippocampus and neocortex during sleep.
Specifically, the notion that temporally coincident reactivation events in both areas, analogous to the state of the intermediary hidden and visible
layers at each time step, can be used to infer statistical regularities in their representations that are incorporated into a generative model

Current Opinion in Neurobiology 2017, 44:193–201 www.sciencedirect.com

 Neural computations in sleep Penagos, Varela and Wilson 195

represented by the hidden layer of the RTRBM. As explored in the main text, the interconnected anatomical relationships between the thalamus,
neocortex and hippocampus, along with the nested temporal structure of sleep could support linking successive time steps in the training of the
RTRBM.

(a) General RBM training algorithm

Step 1 Step 2
2.- Update hidden Units 1.- Fix Hidden Units

V
1.- Fix Visible Units (input data) 2.- Sample Visible Units (prediction)

(b) RBM variants for sequential data

TRBM
init
H0 H1 H2 H3 H4 H5

V0 V1 V2 V3 V4 V5

time

RTRBM
init
H0 H1 H2 H3 H4

H’0 H’1 H’2 H’3 H’4

V0 V1 V2 V3 V4

Current Opinion in Neurobiology

known as contrastive divergence [30] in the RBM formu-
lation, drives an update in the connection weights
between visible and hidden layers (Box 1). In models
of the neocortex, this is postulated to occur through the
interaction of lower and higher cortical layers, but training
could also start at earlier processing stages if the neocortex
sent its expectations to the thalamus [15]. The error
signal could then be generated through the inhibitory
effect of the thalamic reticular nucleus, which receives
inputs from both the neocortex and the dorsal thalamus,
on thalamocortical cells. In the hippocampus, error esti-
mation can be calculated by comparing spiking activities
during the encoding and retrieval phases of the theta
rhythm [26,31]. The outcome of this operation can lead to
model updates as evidenced by the finding that, in
response to changes in reward amounts at spatial loca-
tions, hippocampal place cell activity changes from sig-
naling the animal’s current position to depicting, in
reverse order, the trajectory that led the subject to its
current rewarded location [32,33]. Coupling this reverse
trajectory replay with bursting dopaminergic activity [34]
would produce a location-value gradient (highest at the
reward location) that could serve two purposes: first, in the
hippocampus, establish location-based reward expecta-
tions and second, in neocortical sites, attribute values to
the actions that led to reward. Combining these new
associations would effectively yield a more abstract
state-action representation that combines relevant

www.sciencedirect.com Current Opinion in Neurobiology 2017, 44:193–201

196 Neurobiology of sleep
information such as space, reward and additional contin-
gencies of a particular experience that can alter the
subject’s behavioral policy to maximize reward collection.
Note that this reverse representation of spatial trajectories
is accompanied by fast oscillations in the hippocampal
local field potential (LFP), known as sharp wave ripples
(100–300 Hz), which are otherwise predominantly found
during non-rapid eye movement (NREM) sleep. Because
neocortical activity during this state is temporally related
to hippocampal ripples, sleep could be an ideal state
for model training that generalizes these state-action
representations to include additional scenarios and
environments.
Training a generative model during sleep
If training through error correction occurs as behavior
takes place, what is the role of sleep? If no additional
modifications occurred during sleep, each internal activity
pattern produced during wakefulness would be almost
exclusively tied to the set of conditions directly experi-
enced by an individual. Although for a few navigational
experiences this might seem like a pragmatic solution, the
brain would rapidly face an impossible challenge: to
uniquely encode an exponentially large number of states
with a finite number of internal spiking patterns. This
would hinder the ability of the brain to generalize and
infer causal relationships beyond those experienced by
the subject. Because during sleep the external input that
normally drives the update of the model is not present, we
propose that sleep provides an opportunity for the model
to evaluate novel representations that could not otherwise
be considered, given the strong constraints imposed by
external input during wakefulness.
In the following sections we will consider how the tem-
poral organization of sleep and the computations across
brain regions can contribute to this extended training.
Nested oscillations during NREM sleep
The recurring organization of NREM sleep provides an
appealing temporal setting for model training that could
favor a hierarchical and compositional processing of infor-
mation where increasing levels of abstraction are identi-
fied at correspondingly increasing timescales and incor-
porated into the model. Specifically, neuronal activity
during NREM sleep is characterized by alternations of
suppressed (down) and elevated (up) spiking that corre-
spond to the cortical slow oscillation (SO, 0.1–1 Hz),
where a large-amplitude biphasic wave known as K-
complex marks the down state. Because the SO is present
throughout NREM sleep and because it groups additional
sleep rhythms [35], it could represent a functional unit
that serves two functions: first, to trigger the recursive
execution of elementary operations across brain areas
within each up state and, second, to link the outcomes
of these operations in a sequential fashion over an
extended period of time, leading to a temporally nested
Current Opinion in Neurobiology 2017, 44:193–201
structure that promotes the integration of information in a
hierarchical manner. In its first proposed role, this func-
tional unit mediates specific computations depending on
its internal temporal microstructure, which is supported
by neocortical cells preferentially spiking at different
phases of the up state [36]. In addition, spindles (7–
15 Hz) may be present within the SO, typically following
the K-complex, and their peak amplitude can happen at
different phases of the up state. A related empirical
observation is that spindles peaking in the late phase
of the SO are more predictive of subsequent learning [37].
In addition to spindles, delta waves (1–4 Hz) may also be
present at various phases of the SO [38,39]. Non-neocor-
tical areas such as the thalamus and hippocampus con-
tribute to the microstructure of the SO [36,40]. Thalamic
activity is involved in spindle pacemaking and could also
be involved in the initiation of an up state [41,42].
Hippocampal ripples (50–100 ms) preferentially occur
before and after K-complexes, delta waves and spindles
[43]; when they happen in multi-ripple bursts (including
up to eight ripples) [44], they can also be phased-locked to
individual cycles of a spindle [45]. Overall, the nested
nature of these oscillations suggests a means by which
complex computations can be decomposed into elemen-
tary operations across brain areas. These operations can
be used recursively in a sequential manner to extract
fundamental associations between neural representations
potentially revealing causal relationships between them.
Artificial recurrent networks and sleep dynamics
Current learning models posit that statistical regularities
of the environment are slowly consolidated over time
through cortico-hippocampal interactions during sleep,
resulting in more robust internal representations that
can facilitate subsequent learning [8,10]. We propose
that the extraction of the statistics of the world during
sleep is one of a more extensive set of computations
whose ultimate goal is to infer causal dependencies
between rules, actions and states that produce favorable
outcomes during behavior. Therefore, as statistical regu-
larities emerge, they are sampled and combined proba-
bilistically with rules and action representations to reveal
alternative means to solve previously encountered pro-
blems. In the case of navigation, the process of sampling
and combining representations of spatial paths with rules
and outcomes could result in the partition of behavioral
trajectories into shorter paths that can reveal intermediate
positions as key nodes, or sub-goals, in reaching a desired
goal location. Paths that link several of these nodes can
then be recombined as modules that produce novel routes
to be evaluated as potential navigational policies in sub-
sequent behavior [46,47].
Consistent with this proposal, a recent experiment in
which rats learned odor-place associations highlights
the co-occurrence of these processes during sleep: rats
learned the rules and spatial mapping of the task over
www.sciencedirect.com

several days and, once performance was asymptotic, they
were able to extrapolate the rules to new odor-location
pairs in a single training session. Critically, the ability to
perform the task successfully with the newly paired items
was eliminated if the hippocampus was lesioned before
rats had the opportunity to sleep following training. In
other words, when the task required the extrapolation of
rules beyond purely spatial changes to the layout of the
environment, an intact hippocampus was required for
sleep to have a positive impact on behavior [48].
Intuitively, establishing causal relationships requires that
several neural representations interact over time to infer
their correct temporal dependencies. Several recurrent
networks can successfully learn and generate sequential
data and suggest potential implementations of model
training during sleep. What these networks have in com-
mon is that operations between visible and hidden layers
are performed iteratively over time and the outcome at
one time step influences the immediately following com-
putation, consistent with the recurring structure of sleep.
They differ, however, in how they incorporate longer-
range influences of one computation over upcoming
representations and whether their operations can be
broadly mapped onto physiological events. For example,
the long short-term memory formulation adds an internal
state or memory cell to the hidden layer at each time step
[49]. The content of the memory cell remains unchanged
for, and can influence, an arbitrary number of computa-
tion time steps. While effective in enabling long-range
interactions across representations, it is unclear what the
physiological equivalent of this memory cell would be. By
contrast, the recurrent TRBM (RTRBM, Box 1) does not
explicitly include long-range relationships, while empha-
sizing dependencies between adjacent computations
[16]. Despite this shortcoming, the RTRBM is reminis-
cent of the cortico-hippocampal temporal dynamics
observed within and across individual SOs (Figure 1)
suggesting a broad correspondence to how the neocortex
and hippocampus influence each other’s representations
within individual periods of the SO, in line with recent
electrophysiological findings [50].
An interesting consideration is that sleep representations
could violate the temporal order in which they are pro-
duced during wake. This notion is captured in the bidi-
rectional network formulation, which adds a second hid-
den layer so that at each computation step, one hidden
layer receives information from previous states and the
other hidden layer receives information from future states
[51]. This configuration emphasizes the notion that the
role of one representation may change upon its specific
position within a sequence of events. Contrasting with
bidirectional networks, models using dilated causal con-
volutions, although not recurrent, can effectively link past
computations over long intervals preserving the temporal
order with which the network models the data [52]. The
www.sciencedirect.com
Neural computations in sleep Penagos, Varela and Wilson 197
observation that neurons in the medial entorhinal cortex
can sustain elevated firing activity across several neocor-
tical up states [53] suggests an effective means by which
the brain could implement a dilated causal convolution-
like operation to link temporal relationships across cycles
of the SO.
In summary, while several artificial network architectures
can discover causal relationships across representations,
the RTRBM provides guiding principles that are consis-
tent with the temporal structure of sleep oscillations.
Learning through simulations during sleep
An important difference between wake and sleep is that,
during the latter, the generative model is exclusively
producing and comparing internal predictions across
layers. Physiologically, this means that the neocortex
and hippocampus are, in principle, not constrained to
encode representations that match identical navigational
experiences. Instead, each area can produce variations of
those representations that can be combined in a compo-
sitional manner to produce simulations that yield increas-
ing levels of abstraction in the internal network. At the
most basic level, neocortical and hippocampal patterns
that are not linked to identical spatial experiences can
help reveal equivalent features of an environment, lead-
ing to the reconstruction of a map that can be used for
navigation. Next, including non-spatial sensory dimen-
sions into these internal simulations can bring about rules
associated with this map. Lastly, combining these
inferred rules and locations with representations of
actions could assess their value in different contexts
and improve the predictive power of the model. Ulti-
mately, these hierarchical processes can lead to richer and
more flexible representations because they can answer
the what, why and how questions that allow for generaliza-
tion and appropriate behavioral policies within and across
environments.
These tiered processes can correspond to the multiple
scales of temporal dynamics of the SO reflected by its
nested oscillations (Figure 1A). For example, recall that
multi-ripple bursts can be phase-locked to spindles
resulting in a repeating pattern of alternating cortico-
hippocampal activity. Because representations of
extended trajectories in a multi-ripple burst arise from
the sequential expression of shorter segments within
single ripples [44], individual spindle cycles provide an
opportunity for cortical influence on the content of imme-
diately upcoming ripple-encoded spatial snippets. A
potential consequence is the expression of novel trajec-
tories, like shortcuts [54], that link segments of the
environment over a multi-ripple burst. Likewise, indi-
vidual ripples could trigger the neocortical expression of
additional dimensions that allow the inference of rules
and actions relevant to the environment. Importantly,
ripples can also be tied to K-complexes and delta waves;
Current Opinion in Neurobiology 2017, 44:193–201
198 Neurobiology of sleep

Figure 1

(a)
Delta Spindle K-complex
Neocortex

Multi-ripple burst
Hippocampus

Slow Oscillation
(b)

V0 V1 V2 V3 V4 V5

H’5
H’0 H’1 H’2 H’3 H’4

H0 H1 H2 H3 H4 H5

Current Opinion in Neurobiology

Nested sleep oscillations under the RTRBM framework.

Sleep is characterized by thalamocortical and hippocampal oscillations whose temporal evolution is reminiscent of the sequential processing in
the RTRBM. (a) Diagram showing three schematic consecutive cycles of neocortical slow oscillation (SO) and the nested accompanying rhythms
in neocortex and hippocampus. The neocortical slow oscillation is a recurring pattern that groups the major thalamocortical sleep rhythms
including K-complexes, spindles and delta waves. Hippocampal ripples are also grouped by the SO and tend to occur in temporal proximity to
neocortical sleep waves. Each up state in the SO may serve as a functional unit to process information and extract patterns that train different
aspects of an internal generative model. (b) RTRBM-like interactions may happen within individual up states in which different neocortical regions
assume the role of the visible layer, Vt, and the hippocampus that of the intermediate hidden layer, H’t. The network update realized within a given
SO functional unit depends on its internal temporal structure, the neocortical areas engaged dominantly with the hippocampus and the state of
the network in the previous block of computations. For example, during a delta-dominated SO, interactions (red lines) between neocortical (V) and
hippocampal (H’) representations could result in the incorporation of rules to the internal generative model (H). This, in turn, could bias (green
lines) cortical and hippocampal representations to be evaluated iteratively in individual cycles (dotted box) of an upcoming spindle-multi-ripple-
burst event. The result can provide a means to infer spatial layouts or to simulate actions (e.g., a novel trajectory) in these inferred virtual
environments. Ultimately, these sequential processing chains can lead to more abstract representations that support generalization and casual
inference.

the finding that elevated prefrontal-hippocampal spike
correlations were observed in the absence of spindles
within up states [43] suggests further computations
linked to these additional neocortical oscillations. In
particular, spiking patterns encoding rules in prefrontal
cortex are subsequently replayed toward the end of
neocortical up states, coinciding with ripples in the hip-
pocampus as well as delta oscillations in prefrontal cortex
[24]. Together, these observations suggest two possibili-
ties: the extraction of rules could take place either during
the late, post-spindle, phase of an SO, or within SOs
dominated by delta oscillations alone. Besides inferring
relations between rules and space, sleep may allow the
opportunity to improve behavioral policies through the
evaluation of actions in simulated contexts. The retro-
splenial cortex lies at the intersection of allocentric and

Current Opinion in Neurobiology 2017, 44:193–201 www.sciencedirect.com


egocentric representation streams and its spiking activity
also encodes action-related variables [55,56]. Hence, its
interactions with the hippocampus and ventral striatum
[57] could carry out this final state-action evaluation for
network training during SOs. In this stage, the coordi-
nated activities of retrosplenial cortex and hippocampus
could simulate multiple route options in order to assess
their value for efficient navigation, with multi-ripple
bursts potentially mediating this simulation, given their
representation of extended behavioral sequences, and
ventral striatum providing reward associations.
Importantly, the recurrent operations of the RTRBM do
not change across time steps. Instead the representations
of the intermediate and visible layers dictate what aspects
of the generative model can be learned (Figure 1B).
Physiologically, this implies that multiple brain structures
can differentially interact with the hippocampus to fulfill
different training needs. For example, sensory cortices
would preferentially contribute to extraction of map
features, prefrontal cortex would enable rule identifica-
tion and retrosplenial cortex would be favored for policy
and action evaluation. The thalamus, with its extensive
projections and gating mechanisms, could bias cortical
activation and representation patterns that engage with
the hippocampus for different aspects of model training.
For instance, it could modulate the duration, structure
and content of ripple-spindle computations and it could
provide a link across SOs to implement an effective
strategy that prioritizes meaningful representations to
update the generative model while avoiding exhaustive
comparisons that may not add value to it.
In summary, sleep network training can be thought of as a
simulation involving coordinated activity between tha-
lamic, cortical and hippocampal areas. During this state,
representations of past experiences are combined to infer
the spatial structure and valid rules in an environment.
Together with the evaluation of simulated actions, these
inferences effectively answer the what, why and how
questions that lead to a flexible model of the external
world. In the end this model may enable the brain to
compose unexplored scenarios, something that could not
otherwise happen during awake behavior given the strong
constrains imposed by external input.
Conclusions
Previous work has emphasized the contribution of sleep
to the formation of episodic and semantic knowledge and
to the identification of similar contexts in different envir-
onments [8,10]. However, the discovery of insights or
implicit rules for problem solving during sleep has been
largely unaccounted for [7,58,59]. Here, we explored
computational principles during sleep that endow the
brain with the ability to find new solutions to familiar
problems or to extrapolate previous strategies to new
scenarios. We discussed how the temporal architecture
www.sciencedirect.com
Neural computations in sleep Penagos, Varela and Wilson 199
of sleep could enable the identification of modules that
can be flexibly combined in a compositional fashion to
yield novel behavioral policy representations that can be
evaluated through sleep simulations [47,60]. To this end,
the RBM and its recurrent variants provide a general
framework of computational principles across different
brain states that lead to the establishment of robust
generative models of the world. The observation that
humans are able to imagine and anticipate scenarios, even
in the absence of external cues, provides intuitive proof of
the existence of this type of models in the brain, which
can identify and establish causal relationships between
contexts, actions and outcomes, and not merely recall past
events. In addition to providing a useful framework to
interpret physiological observations, the RBM formula-
tion raises important questions that require further theo-
retical and experimental work. For instance, experiments
in which generalization within and across environments
can be assessed before and after sleep at the electrophysi-
ological and behavioral level, can examine the contribution
of sleep to our proposed hypotheses of model updating
during this state. In addition, manipulations that target the
temporal relationships across areas by altering phase rela-
tionships between specific oscillatory events and/or the
representations associated with them, can shed light into
the actual computations that take place in neural circuits.
For example, experiments directed at breaking multi-
ripple bursts into individual ripples, changing the duration
of spindles, or altering the phase relationships with one
another, could provide opportunities to examine neural
representations in these altered states. Moreover, directing
these manipulations to retrosplenial, prefrontal or hippo-
campal areas could provide further assessments into their
contributions to inferring maps, rules and actions, and lead
to improved theoretical architectures that better account
for the structure and function of the brain.
Conflict of interest statement
Nothing declared.
Acknowledgements
We thank members of the Wilson lab for comments on an earlier version of
this manuscript. This material is based upon work supported by the Center
for Brains, Minds and Machines (CBMM), funded by National Science
Foundation Science and Technology Center award CCF-1231216 and by a
NARSAD Young Investigator Award from the Brain & Behavior Research
Foundation to C.V.
References and recommended reading
Papers of particular interest, published within the period of review,
have been highlighted as:
 of special interest
 of outstanding interest
1. Tenenbaum JB, Kemp C, Griffiths TL, Goodman ND: How to grow
a mind: statistics, structure, and abstraction. Science 2011,
331:1279-1285.
2. Chater N, Tenenbaum JB, Yuille A: Probabilistic models of
cognition: conceptual foundations. Trends Cogn. Sci. 2006,
10:287-291.
Current Opinion in Neurobiology 2017, 44:193–201
200 Neurobiology of sleep
3. O’Keefe J: A computational theory of the hippocampal
cognitive map. Prog. Brain Res. 1990, 83:301-312.
4. Pfeiffer BE, Foster DJ: Hippocampal place-cell sequences
depict future paths to remembered goals. Nature 2013,
497:74-79.
5. Hinton G: Where do features come from? Cogn. Sci. 2014,
 38:1078-1101.
This paper offers a brief and informative review of the evolution of two
major efforts in machine learning: deep networks and the Restricted
Boltzmann Machine. Using intuitive descriptions, the author contrasts the
differences between these types of networks. The author also outlines
how stacking RBMs can lead to deep architectures that can learn
statistical features in an unsupervised fashion and can then be fine-tuned
with external feedback. This paper is relevant because the Recurrent
Temporal Restricted Boltzmann Machine is composed of sequential
layers of RBMs that are stacked in time.
6. Hinton GE, Dayan P, Frey BJ, Neal RM: The ‘wake-sleep’
algorithm for unsupervised neural networks. Science 1995,
268:1158-1161.
7. Wagner U, Gais S, Haider H, Verleger R, Born J: Sleep inspires
insight. Nature 2004, 427:352-355.
8. Kumaran D, Hassabis D, McClelland JL: What learning systems
 do intelligent agents need? Complementary learning systems
theory updated. Trends Cogn. Sci. 2016, 20:512-534.
This paper reviews and updates the complementary learning systems
(CLS) theory to better account for behavioral and theoretical results. In its
original form, the CLS theory proposed that the hippocampus can quickly
store detailed experiences, while neocortex updates synaptic weights
slowly to eventually store knowledge representations that are less
detailed but reflect environment statistics. Two updates to the CLS theory
are discussed: first, results from machine learning in multilayer networks
suggest that the neocortex can also learn fast, provided that the new
information is consistent with the model already stored in neocortical
networks; and second, the hippocampus can ensure that certain experi-
ences are remembered in detail by tagging salient experiences during
their hippocampal reactivation.
9. Marr D: Simple memory: a theory for archicortex. Philos. Trans.
R. Soc. Lond. B Biol. Sci. 1971, 262:23-81.
10. McClelland JL, McNaughton BL, O’Reilly RC: Why there are
complementary learning systems in the hippocampus and
neocortex: insights from the successes and failures of
connectionist models of learning and memory. Psychol. Rev.
1995, 102:419-457.
11. Mnih V et al.: Human-level control through deep reinforcement
learning. Nature 2015, 518:529-533.
12. Silver D et al.: Mastering the game of Go with deep neural
networks and tree search. Nature 2016, 529:484-489.
13. Freund Y, Haussler D: In Advances in Neural Information
Processing Systems 4. Edited by Moody JE, Hanson SJ, Lippmann
RP. Morgan-Kaufmann; 1992:912-919.
14. Clark A: Whatever next? Predictive brains, situated agents, and
 the future of cognitive science. Behav. Brain Sci. 2013, 36:181-
204.
The author reviews evidence in support of a unified hypothesis of
perception, cognition and action that presents predictive coding, imple-
mented through probabilistic generative models, as a fundamental com-
putational tool in neocortical networks. Under this hypothesis, sensory,
cognitive and motor systems rely on estimated probability-density dis-
tributions of external world variables to generate top-down predictions
that can be compared with ongoing sensory input. The resulting error
signal is sent forward along the cortical hierarchy to drive perception and
allow further tuning of the internal model. In humans, the top-down
predictions and feedforward error signals are proposed to also incorpo-
rate the statistics of sociocultural constrains, and to account for dysfunc-
tional states such as hallucinations and delusions.
15. Rao RPN, Sejnowski TJ: Probabilistic Models of the Brain:
 Perception and Neural Function. The MIT Press; 2002:297-315.
The authors discuss and test the hypothesis that recurrent connectivity in
neocortex implements temporal predictive coding through a generative
model that can be trained using the rules of spike-timing dependent
synaptic plasticity (STDP). They test this hypothesis using a compart-
mental model of interconnected neurons with synaptic weights that
change according to STDP when presented with sequential stimuli,
Current Opinion in Neurobiology 2017, 44:193–201
and conclude that such recurrent networks can be trained to implement
generative models that can be used for temporal prediction, e.g., moving
stimuli, in neocortical physiological networks.
16. Sutskever I, Hinton GE, Taylor GW: In Advances in Neural
 Information Processing Systems 21. Edited by Koller D,
Schuurmans D, Bengio Y, Bottou L. Curran Associates, Inc.;
2009:1601-1608.
This paper introduces the mathematical formulation for the Recurrent
Temporal Restricted Boltzmann Machine (RTRBM) and it shows the
mathematical equivalence between deep architectures and the RTRBM.
17. Sutskever I, Hinton GE: Learning multilevel distributed
representations for high-dimensional sequences. AISTATS
2007, 2:548-555.
18. Colin McDonnell: Lecture 12.3 — Restricted Boltzmann Machines
[Neural Networks for Machine Learning].
19. Skaggs WE, McNaughton BL, Wilson MA, Barnes CA: Theta
phase precession in hippocampal neuronal populations and
the compression of temporal sequences. Hippocampus 1996,
6:149-172.
20. Foster DJ, Wilson MA: Hippocampal theta sequences.
Hippocampus 2007, 17:1093-1099.
21. Redish AD: Vicarious trial and error. Nat. Rev. Neurosci. 2016,
17:147-159.
22. Smith DM, Barredo J, Mizumori SJY: Complimentary roles of the
hippocampus and retrosplenial cortex in behavioral context
discrimination. Hippocampus 2012, 22:1121-1133.
23. Vedder LC, Miller AMP, Harrison MB, Smith DM: Retrosplenial
cortical neurons encode navigational cues, trajectories and
reward locations during goal directed navigation. Cereb.
Cortex 2016 http://dx.doi.org/10.1093/cercor/bhw192.
[Epub ahead of print].
24. Peyrache A, Khamassi M, Benchenane K, Wiener SI, Battaglia FP:
Replay of rule-learning related neural patterns in the prefrontal
cortex during sleep. Nat. Neurosci. 2009, 12:919-926.
25. Jones MW, Wilson MA: Theta rhythms coordinate
hippocampal–prefrontal interactions in a spatial memory task.
PLoS Biol. 2005, 3(12):2187-2199.
26. Hyman JM, Hasselmo ME, Seamans JK: What is the functional
relevance of prefrontal cortex entrainment to hippocampal
theta rhythms? Front. Neurosci. 2011, 5:24.
27. Ito HT, Zhang S-J, Witter MP, Moser EI, Moser M-B: A
prefrontal-thalamo-hippocampal circuit for goal-directed
spatial navigation. Nature 2015, 522:50-55.
28. Deschênes M, Veinante P, Zhang ZW: The organization of
corticothalamic projections: reciprocity versus parity.
Brain Res. Brain Res. Rev. 1998, 28:286-308.
29. Vertes RP, Hoover WB, Viana Di Prisco G: Theta rhythm of the
hippocampus: subcortical control and functional significance.
Behav. Cogn. Neurosci. Rev. 2004, 3:173-200.
30. Hinton GE: Training products of experts by minimizing
contrastive divergence. Neural Comput. 2002, 14:1771-1800.
31. Ketz N, Morkonda SG, O’Reilly RC: Theta coordinated error-
driven learning in the hippocampus. PLoS Comput. Biol. 2013,
9:e1003067.
32. Foster DJ, Wilson MA: Reverse replay of behavioural sequences
in hippocampal place cells during the awake state. Nature
2006, 440:680-683.
33. Ambrose RE, Pfeiffer BE, Foster DJ: Reverse replay of
hippocampal place cells is uniquely modulated by changing
reward. Neuron 2016, 91:1124-1136.
34. Gomperts SN, Kloosterman F, Wilson MA: VTA neurons
coordinate with the hippocampal reactivation of spatial
experience. eLife 2015, 4:e05360.
35. Steriade M: Grouping of brain rhythms in corticothalamic
systems. Neuroscience 2006, 137:1087-1106.
www.sciencedirect.com

36. Neske GT: The slow oscillation in cortical and thalamic
networks: mechanisms and functions. Front. Neural Circuits
2015, 9:88.
37. Demanuele C et al.: Coordination of slow waves with sleep
spindles predicts sleep-dependent memory consolidation in
schizophrenia. Sleep 2016.
38. Steriade M, Dossi RC, Nuñez A: Network modulation of a slow
intrinsic oscillation of cat thalamocortical neurons implicated
in sleep delta waves: cortically induced synchronization and
brainstem cholinergic suppression. J. Neurosci. 1991,
11:3200-3217.
39. Nuñez A, Curró Dossi R, Contreras D, Steriade M: Intracellular
evidence for incompatibility between spindle and delta
oscillations in thalamocortical neurons of cat. Neuroscience
1992, 48:75-85.
40. Gardner RJ, Hughes SW, Jones MW: Differential spike timing
and phase dynamics of reticular thalamic and prefrontal
cortical neuronal populations during sleep spindles. J.
Neurosci. 2013, 33:18469-18480.
41. Steriade M, Domich L, Oakson G, Deschênes M: The
deafferented reticular thalamic nucleus generates spindle
rhythmicity. J. Neurophysiol. 1987, 57:260-273.
42. MacLean JN, Watson BO, Aaron GB, Yuste R: Internal dynamics
determine the cortical response to thalamic stimulation.
Neuron 2005, 48:811-823.
43. Peyrache A, Battaglia FP, Destexhe A: Inhibition recruitment in
prefrontal cortex during sleep spindles and gating of
hippocampal inputs. Proc. Natl. Acad. Sci. U. S. A. 2011,
108:17207-17212.
44. Davidson TJ, Kloosterman F, Wilson MA: Hippocampal replay of
extended experience. Neuron 2009, 63:497-507.
45. Staresina BP et al.: Hierarchical nesting of slow oscillations,
 spindles and ripples in the human hippocampus during sleep.
Nat. Neurosci. 2015, 18:1679-1686.
This study explores the relationship between the slow oscillation, spindles
and ripples in the human hippocampus. The authors obtained intra-
cranial electroencephalogram (iEEG) recordings from 12 subjects and
assessed the phase-amplitude relationship between sleep oscillations.
They report a nested structure in which hippocampal ripples occur at the
troughs of spindles, which in turn, take place during the negative phase of
the hippocampal slow oscillation. Importantly, they find that ripples occur
in bursts with a repetition rate of 14.5 Hz, reinforcing the observation that
multi-ripple bursts can also have a tight timing relationship with spindles.
These observations are significant because they provide a framework like
that of the neocortical slow oscillation through which the hippocampus
and neocortex can communicate with precise timing during sleep.
www.sciencedirect.com
Neural computations in sleep Penagos, Varela and Wilson 201
46. Botvinick M, Toussaint M: Planning as inference. Trends Cogn.
Sci. 2012, 16:485-488.
47. Donnarumma F, Maisto D, Pezzulo G: Problem solving as
probabilistic inference with subgoaling: explaining human
successes and pitfalls in the Tower of Hanoi. PLoS Comput.
Biol. 2016, 12:e1004864.
48. Tse D et al.: Schemas and memory consolidation. Science 2007,
316:76-82.
49. Graves A: Generating Sequences With Recurrent Neural
Networks. ArXiv13080850 Cs (2013).
50. Rothschild G, Eban E, Frank LM: A cortical-hippocampal-
cortical loop of information processing during memory
consolidation. Nat. Neurosci. 2017, 20:251-259.
51. Schuster M, Paliwal KK: Bidirectional recurrent neural
networks. IEEE Trans. Signal Process. 1997, 45:2673-2681.
52. van den Oord A et al.: WaveNet: A Generative Model for Raw
Audio. ArXiv160903499 Cs; 2016.
53. Hahn TTG, McFarland JM, Berberich S, Sakmann B, Mehta MR:
Spontaneous persistent activity in entorhinal cortex
modulates cortico-hippocampal interaction in vivo. Nat.
Neurosci. 2012, 15:1531-1538.
54. Gupta AS, van der Meer MAA, Touretzky DS, Redish AD:
Hippocampal replay is not a simple function of experience.
Neuron 2010, 65:695-705.
55. Alexander AS, Nitz DA: Retrosplenial cortex maps the
conjunction of internal and external spaces. Nat. Neurosci.
2015, 18:1143-1151.
56. Jacob P-Y et al.: An independent, landmark-dominated head-
direction signal in dysgranular retrosplenial cortex. Nat.
Neurosci. 2017, 20:173-175 http://dx.doi.org/10.1038/nn.4465.
57. Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL,
Pennartz CMA: Hippocampus leads ventral striatum in replay of
place-reward information. PLoS Biol. 2009, 7:e1000173.
 J, Gaskell MG: Does sleep improve your grammar?
58. Mirkovic
Preferential consolidation of arbitrary components of new
linguistic knowledge. PLoS One 2016, 11.
59. Monaghan P et al.: Sleep promotes analogical transfer in
problem solving. Cognition 2015, 143:25-30.
60. Arnold AEGF, Iaria G, Ekstrom AD: Mental simulation of routes
during navigation involves adaptive temporal compression.
Cognition 2016, 157:14-23.
Current Opinion in Neurobiology 2017, 44:193–201