Pattern Recognition 97 (2020) 107021

Contents lists available at ScienceDirect

Pattern Recognition
journal homepage: www.elsevier.com/locate/patcog

Decomposed slice sampling for factorized distributions

Jiachun Wang, Shiliang Sun∗
Department of Computer Science and Technology, East China Normal University, 3663 Zhongshan Road, Shanghai 200241, PR China

a r t i c l e i n f o a b s t r a c t

Article history: Slice sampling provides an automatical adjustment to match the characteristics of the distribution. Al-
Received 27 September 2018 though this method has made great success in many situations, it becomes limited when the distribution
Revised 19 July 2019
is complex. Inspired by Higdon [1], in this paper, we present a decomposed sampling framework based
Accepted 25 August 2019
on slice sampling called decomposed slice sampling (DSS). We suppose that the target distribution can
Available online 26 August 2019
be divided into two multipliers so that information in each term can be used, respectively. The first mul-
Keywords: tiplier is used in the first step of DSS to obtain horizontal slices and the last term is used in the second
Slice sampling step. Simulations on four simple distributions indicate the effectiveness of our method. Compared with
Markov chain Monte Carlo slice sampling and Hamiltonian Monte Carlo on Gaussian distributions in different dimensions and ten
Decomposed slice sampling real-world datasets, the proposed method achieves better performance.
Hamiltonian Monte Carlo
© 2019 Elsevier Ltd. All rights reserved.

1. Introduction a variety of scenarios, for example, in variable selection [9,10],

Markov chain Monte Carlo (MCMC) [2,3] is a very general and
powerful framework that can be used to sample from a large
class of distributions encountered in pattern recognition and
machine learning. MCMC aims to efficiently draw samples from an
unnormalized density function by utilizing the properties of the
Markov chain. It serves as a fundamental approach for probabilistic
inference in many computational statistical problems.
Markov chains in MCMC methods with auxiliary variables may
mix faster and be easier to sample than standard single variable
algorithms [1]. The Swendsen-Wang (SW) algorithm [4] is the
first formal auxiliary variable methods for MCMC [1,5]. It was
introduced to overcome some of the drawbacks of the Gibbs
sampler [6] for the Ising model and was generalized by Edwards
and Sokal [7]. The further development of auxiliary variables
methods for MCMC is the introduction of slice sampling. The
slice sampling method [8] alternates between uniformly sampling
from the vertical region determined by the density at the current
point and uniformly sampling from the horizontal “slice” deter-
mined by the current vertical position. It is an auxiliary-variable
method where the horizontal slice is the auxiliary variable. Slice
sampling can automatically adjust to match the characteristics
of the distribution, which makes it easy to tune. Thus in some
cases, slice sampling has better performance than Gibbs sampling
and Metropolis methods [8]. Because of the easy implementation
and good mixing properties, the slice sampler can be applied in
∗
Corresponding author.
E-mail address: slsun@cs.ecnu.edu.cn (S. Sun).
spatial models [11,12] and biological models [13–15].
Although slice sampling from univariate distributions is
straightforward, in complex multivariate distributions, effectively
finding an interval of the “slice” is difficult. Therefore, adaptive
methods that consider the dependencies between variables were
proposed, and other upgraded approaches such as random walk
suppression methods, i.e., overrelaxed slice sampling and reflective
slice sampling, were also developed [8]. Recently, Murray et al.
[16] attempted to perform slice sampling in latent variable mod-
els with multivariate Gaussian priors by sampling from a high
dimensional elliptical curve.
Similar in introducing auxiliary variable but different in ap-
proach is the Hamiltonian Monte Carlo (HMC) [17,18]. HMC casts
the probabilistic simulation in the form of a Hamiltonian system.
By introducing momentum as the auxiliary variable, HMC exploits
the gradient information to generate samples. The combination
with the Metropolis algorithm removes any bias associated with
the discretization in HMC [18,19]. With careful tuning of pa-
rameters, HMC can achieve favorable mixing properties in many
cases. However, proper tuning is difficult because HMC is sensitive
to parameter change. As a consequence, adaptive approaches
[20,21] were proposed for the interest of automating parameter
tuning. Recently, HMC with Riemann manifold information was
also proposed [22]. Another attempt towards improving HMC
has exploited non-canonical Hamiltonian dynamics to expand the
range of sampling space [23]. As a representative MCMC method,
HMC will be taken as a compared method in our experiments.
The previous work mentioned above is dedicated to improv-
ing the specific algorithm for solving slice intervals. Unlike the

https://doi.org/10.1016/j.patcog.2019.107021
0031-3203/© 2019 Elsevier Ltd. All rights reserved.
2 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021
Fig. 1. Illustration of slice sampling. (a) For a given value x0 , a vertical level, u, is drawn uniformly from (0, p˜ (x )), and used to define a “slice”, shown by the solid horizontal
lines. (b) Because in practice, solving the exact endpoints of the slice is intractable, a new point is drawn uniformly from the interval [xmin , xmax ] which contains the current
point x0 .
previous work, in this paper, we present a novel sampling frame-
work based on slice sampling called decomposed slice sampling
(DSS) to efficiently sample more complex distributions for the rea-
son that traditional slice sampling becomes less efficient when the
target distribution is complex. We suppose that the unnormalized
target distribution is a factorized distribution which can be decom-
posed into two multipliers so that information in each term can be
used, respectively. Note that for most cases in practice, the target
distribution can be factorized. For example, the posterior distribu-
tion is proportional to a likelihood function multiplied by a prior
distribution. The first multiplier is used in the first step of DSS to
uniformly generate an auxiliary variable and define the horizontal
slice. Then we sample a new point from the second multiplier over
the interval of slices given by the first multiplier. The correctness
of DSS is verified theoretically and practically. The simulated exper-
iments of DSS compared with slice sampling on different dimen-
sions show that DSS performs better in the multi-modal cases. The
result of experiments performed on ten real-world datasets indi-
cates that methods based on the DSS perform better than the slice
sampling and HMC in the case of complex distributions. Though
the DSS is based on the standard slice sampling method in this
paper, other upgraded slice sampling methods which improve the
algorithm for solving slice intervals can also be used in the DSS.
The remainder of this paper is organized as follows.
Section 2 presents related works. Section 3 introduces the slice
sampling method. Section 4 presents the decomposed sampling
framework and show its correctness. Experimental results and
analysis are provided in Section 5. Finally, we conclude this paper
and discuss future work in Section 6.
2. Related works
Recent works about slice sampling mainly focuse on adaptive
multivariate slice sampling. A new multivariate slice sampling
method that uses multiple auxiliary variables to perform multi-
variate updating was proposed in [5]. This method extends the
bivariate slice sampling to the multivariate one, and generates
samples by updating all dimensions at once. Our method is also a
multivariate sampling method but with only one auxiliary variable.
In order to generate effective samples with highly correlated
parameters, some work [24–26] has been put forward. Thompson
and Neal [24] described two slice sampling methods for taking
multivariate steps using the “crumb” framework [8]. These meth-
ods utilize the gradients at rejected samples to accommodate
the local curvature of the log-density surface, which can produce
better samples when the parameters are highly correlated. The
shrinking rank method is also a variation of slice sampling, which
can effectively sample multivariate distributions with highly cor-
related parameters [25]. In this method, the crumbs are Gaussian
random variables centered at the current state. By sampling within
a transformed space, the factor slice sampler [26] can generate
nearly independent draws from a highly correlated, high dimen-
sional target distribution. Though the DSS is based on the standard
slice sampling in this paper, the above mentioned methods are
complementary to DSS (i.e., they can be equipped on the DSS to
meet specific purposes).
Other approaches include parallelizing the multivariate slice
sampling [27] and relaxing the requirement of evaluating the
target distribution up to a constant by bringing together the
advantages of pseudo-marginal methods and slice sampling [28].
It is interesting to consider improving the efficiency of the DSS
through these methods in future work.
3. Slice sampling method
Slice sampling [8] provides an approach to match the distri-
bution by automatically adjusting the step size. It can be seen
as an auxiliary variable method because the sampling process
involves the introduction of the auxiliary variable. Suppose we
wish to sample from a distribution, p(x), which is proportional to
a computable function p˜ (x ). We can first introduce an auxiliary
variable u, sample from the joint distribution pˆ (x, u ) instead, and
then ignore u. This joint distribution is defined as follows:
1
i f 0 < u < p˜ (x ),
pˆ (x, u ) = Zp
(1)
0 otherwise,

where Z p = p˜ (x )dx. The marginal distribution of x is then
  p˜(x )
1 p˜ (x )
pˆ (x, u ) du = du = = p( x ) . (2)
0 Z p Zp
In order to sample from pˆ (x, u ) and then ignore u, we can
alternately sample x and u. According to Eq. (1), fixing x, u is
uniform from 0 to p˜ (x ). Thus, given the initial value of x and p˜ (x ),
we sample u uniformly from (0, p˜ (x )). Then we fix u and sample
x uniformly from the horizontal “slice” defined by {x : p˜ (x ) > u}.
The procedure of slice sampling is illustrated in Fig. 1a. Note that
in practice, solving the exact endpoints of the “slice” is intractable.
Hence, we turn to find an interval xmin ≤ x ≤ xmax that contains
the current x. This interval is hoped to contain as much of the
slices as possible, but not much larger than the slice. The two
most commonly used approaches for finding an appropriate slice
 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021
Algorithm 1 The “stepping out” method.
Input:
Function proportional to the density, f (x );
The current point, xn ;
The vertical level defining the slice, u;
Estimate of the scale of a slice, w;
The integer that limits the maximum size of a slice interval to mw,
m;
Output: The final interval, (L, R );
1: Generate a random number, U ∼ Uni f orm(0, 1 )
2: Set the value of the left bound of the initial interval, L = xn −
w ∗U
3: Set the value of the right bound of the initial interval, R = L + w
4: Generate another random number, V ∼ Uni f orm(0, 1 )
5: Compute the maximum times that the initial interval can be
expanded to the left side, J = F loor (m ∗ V )
6: Compute the maximum times that the initial interval can be
expanded to the right side, K = (m − 1 ) − J
7: while J > 0 and u < f (L ) do
8: Update the left bound of the interval, L = L − w
Update the residual times that expand to the left side, J =
9:
J−1
10: while K > 0 and u < f (R ) do
11: Update the right bound of the interval, R = R + w
12: Update the residual times that expand to the right side, K =
K −1
13: return (L, R )
interval are the “stepping out” procedure and the “doubling”
procedure [8]. In this paper, we use the “stepping out” procedure.
The procedure of slice sampling in one update [8], i.e., from the
current state xn to the next state xn+1 is summarized as follows:
Step 1. Generate a value u uniformly from (0, p˜ (xn )), which
defines a horizontal “slice”: S = {x : u < p˜ (x )}.
Step 2. Randomly choose an initial interval I0 that contains the
current state xn with a size of w, where w is a rough
estimate of the scale of S. Then expand the interval by w
each time using the “stepping out” method (details are
shown in Algorithm 1) until both of the endpoints are
outside the slice region S, or the width of the interval
already reached the given maximum size mw. Notice that
both parameters, m and w, are given manually.
Step 3. Sample one new point xn+1 each time uniformly from
the interval found in step 2 until xn+1 lies in the slice
region. Points that are sampled outside the slice region
S are used to shrink the interval, which is called the
“shrinkage” procedure.
The proof of the correctness of slice sampling is provided in
Appendix A. Though the slice sampling has a very simple form,
it can be applied to a wide variety of distributions and is more
efficient than Gibbs sampling and Metropolis-Hastings in some
simple cases [8]. However, slice sampling from complex distri-
butions becomes difficult because of the time-consuming solving
process of the slice interval.
4. Decomposed slice sampling
Based on the idea of slice sampling, we propose a novel
sampling method called decomposed slice sampling (DSS) which
is able to simulate more complex distributions than the original
slice sampling, especially in high dimensions. Specifically, we
decompose the factorized target distribution into two multipliers.
On the basis of the first part of the distribution, we generate the
3
auxiliary variable and define the horizontal slice. Then we sample
new points from the second part of the distribution over the slice
interval provided by the first part. In this section, we will detail
the decomposed slice sampling and show the correctness of this
proposed framework.
4.1. Decomposed slice sampling
Suppose that the target distribution p(x) can be written in the
form
p( x ) ∝ b 1 ( x ) ∗ b 2 ( x ) , (3)
where b2 (x) is a distribution that is relatively simpler than b1 (x).
In practice, there are many cases that the target distribution
is a factorized distribution which can be decomposed into two
factors. For example, the posterior distribution is usually the
target distribution and it is proportional to a likelihood function
multiplied by a prior distribution.
By introducing an auxiliary variable u, the distribution for u
given x is defined as:
1
p( u | x ) =
, s.t. 0 < u < b1 (x ). (4)
b1 ( x )
Given x, the auxiliary variable u is distributed as Uniform(0,
b1 (x)).
Thus, the distribution for x given u is then
p(x|u ) = b2 (x ), s.t. 0 < u < b1 (x ). (5)
That is, given u, x is distributed according to b2 (x) over {x:
0 < u < b1 (x)}. Notice that b2 (x) is a distribution relatively simpler
than b1 (x) (e.g, a prior), which makes the simulation on p(x|u)
easier.
Similar with the slice sampling, this method also contains three
steps in one update, from the current state xn to the next state
xn+1 :
Step 1. Generate a value u uniformly from (0, b1 (xn )), which
defines a horizontal “slice”: S = {x : u < b1 (x )}.
Step 2. Randomly choose an initial interval I0 that contains the
current state xn with a size of w, where w is a rough esti-
mate of the scale of S. Then expand the interval by w each
time using the “stepping out” method until both of the
endpoints are outside the slice region S, or the width of
the interval already reached the given maximum size mw.
Step 3. Sample one new point xn+1 each time from b2 (x) over the
interval found in step 2, until xn+1 lies in the slice region.
Points that are sampled outside the slice region S are
used in the shrinkage procedure to shrink the interval.
This procedure is illustrated in Fig. 2. Note that in step 3,
the sampling method adopted to sample new points can be any
efficient MCMC method, which depends on the specific practical
situations. In the section of experiments, we will employ slice
sampling and HMC in step 3, respectively.
In fact, related methods have been used in the past. The
Swendsen–Wang (SW) algorithm is an early auxiliary variable
method for MCMC. The original SW algorithm [4] applied only
to the Ising/Potts model for image analysis; in the following year
Edwards and Sokal [7] generalized the SW algorithm to make it
available to other models. Later, Higdon [1] reinterpreted the gen-
eralized SW algorithm from the perspective of auxiliary variables.
In these methods, the target distribution can be written as the
form of multiplication of k + 1 functions: p(x)∝p0 (x)b1 (x)bk (x).
By introducing k auxiliary variables, u1 , . . . , uk , the joint distri-
bution p(x, u1 , . . . , uk ) is defined as a uniform distribution over
the region 0 < ui < bi (x) for i = 1, . . . , k. Then p(x, u1 , . . . , uk ) can
be sampled using Gibbs sampling or some other effective MCMC
4 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021
Fig. 2. Illustration of decomposed slice sampling. The gray curves represent p(x). (a) For a given value xn , a vertical level, u, is drawn uniformly from (0, b1 (xn )), and used
to define a “slice”. The slice interval is shown in bold. (b) Sample one new point xn+1 from b2 (x) over the interval [xmin , xmax ], until xn+1 lies in the slice region. The rejected
point x∗ is used to shrink the interval to [xmin , xmax ].
methods. The slice sampling is actually a special case of the
generalized SW algorithm, where k = 1 and p0 (x ) = 1 [8], and the
decomposed slice sampling procedure described above is also a
special case where k = 1 but p0 (x) = 1.
4.2. Correctness of decomposed slice sampling
To show that the decomposed slice sampling is a correct proce-
dure, it is necessary to prove that the distribution p(x) is invariant
during each step in the Markov chain. To show invariance, we first
suppose that the initial point x0 follows the distribution p(x). Thus
the joint distribution of x0 and u follows:
 Un+1 = Frac(Un + (xn+1 − xn )/w ),
b2 (x )/Z i f 0 < u < b1 (x ),
p(x, u ) =
0 otherwise,

where Z = b1 (x ) ∗ b2 (x )dx. The marginal distribution of x is
obtained by integrating u as follows:
 
b1 ( x )
b1 ( x ) ∗ b2 ( x )
p(x, u ) du = b2 (x )/Z du =
0 Z
Note that in the subsequent update from x0 to x1 , the auxiliary
variable u has the same value. If the joint distribution p(x0 , u) and
p(x1 , u) are identically distributed, then p(x0 ) and p(x1 ) are also
identically distributed by integrating u. So does the subsequent up-
date from x1 to x2 , and so on. Thus, we only need to show that
p(xn , u) and p(xn+1 , u ) are identically distributed in the step of
sampling xn+1 given xn (n = 0, 1, 2, . . .). Because these steps do not
change u, it is equal to prove that the conditional distribution for x
given u is invariant in step 2 and step 3 of the DSS. As introduced
above, p(x|u) is distributed according to b2 (x) over S = {x : u <
b1 (x )}. We can show the invariance of the conditional distribution
by proving that the updates satisfy the detailed balance [29]:
p(xn |u ) p(xn+1 |xn , u ) = p(xn+1 |u ) p(xn |xn+1 , u ).
Denote
p( x n | u ) = b 2 S ( u ) ( x n ) = k n ,
p(xn+1 |u ) = b2S(u ) (xn+1 ) = kn+1 ,
where b2S(u) (x) is the b2 distribution over the slice S determined
by u. It is worth noting that, here, S should be explained as the
accurate analytic solution of the slice region. Now Eq. (8) can be
simplified as:
kn × p(xn+1 |xn , u ) = kn+1 × p(xn |xn+1 , u ).
We can demonstrate that the update procedure satisfies the
detailed balance by proving the following stronger result [8]:
kn × p(xn+1 , r|xn , u ) = kn+1 × p(xn , π (r )|xn+1 , u ), (11)
where r denotes random choices in the sampling procedure, or
more specifically, U and J (or V, K) in the “stepping out” step, and
the sequence of rejected points P in the “shrinkage” step. π (r) is
a first order one-to-one mapping that depends on xn and xn+1 .
Regarding r as a variable, we can obtain Eq. (10) by integrating r
(i.e., considering all the possible choices of r).
Next, we define the mapping π (r). We first define the π (U)
that maps Un to Un+1 as follows:
(12)
(6) where the function Frac(x ) = x − Floor(x ) retains the fractional
part of x. The definition of π (U) makes the left bound Ln+1 of
the initial slice interval In+1 that contains xn+1 align with the
left bound Ln of the initial slice interval In that contains xn , or
explained in formulation, (Ln+1 − Ln ) mod w = 0.
The “stepping out” procedure will generate a random number
= p( x ) . (7)
J ∈ {0, . . . , m − 1}, and so we define the mapping π (J) that maps
Jn to Jn+1 as follows:
Jn+1 = Jn + (xn+1 /w − Un+1 ) − (xn /w − Un ), (13)
where (xn+1 /w − Un+1 ) − (xn /w − Un ) denotes the distance (w
units) between the left bound Ln and Ln+1 .
When the transition between xn and xn+1 is possible, it is easy
to find that the final slice interval will be the same for xn as for
xn+1 with the above definitions. Actually, these two intervals will
be the same when the interval for xn first contains xn+1 . Then in
the following “stepping out” steps, the two intervals will stay the
same. For the situation that the transition between xn and xn+1 is
impossible, Eq. (11) will hold because both sides of it are zero.
Finally, π maps the sequence of rejected points Pn used to
(8) shrink the interval that contains xn to the sequence of rejected
points Pn+1 that are generated from the interval found from xn+1 .
Thus, Pn and Pn+1 are the same sequence.
After the completion of the definition of π , we return to prove
Eq. (11), which can be expanded as:
(9)
kn × p(Un |xn ) p(xn+1 , Jn , Pn |Un , xn )
= kn+1 × p(Un+1 |xn+1 ) p(xn , Jn+1 , Pn+1 |Un+1 , xn+1 ). (14)
Here, the unchanged u is omitted for conciseness. Obviously,
p(Un |xn ) is uniform over (0,1). Although the value of Un+1 depends
on xn and xn+1 , p(Un+1 ) is also uniform over (0,1) because Un+1
(10) is a linear transformation of Un . Therefore, the second term of
 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021
both sides in Eq. (14) can be crossed out and the equation can be
written as:
kn × p(Jn |Un , xn ) p(xn+1 , Pn |Jn , Un , xn )
= kn+1 × p(Jn+1 |Un+1 , xn+1 ) p(xn , Pn+1 |Jn+1 , Un+1 , xn+1 ).
Similarly, p(Jn |Un , xn ) and p(Jn+1 |Un+1 , xn+1 ) are both uniform
over (0,1). Therefore, Eq. (15) can be simplified and transformed
as:
kn × p(Pn |Jn , Un , xn ) p(xn+1 |r, xn )
= kn+1 × p(Pn+1 |Jn+1 , Un+1 , xn+1 ) p(xn |π (r ), xn+1 ).
Here r represents Un , Jn and Pn , while π (r) represents Un+1 ,
Jn+1 and Pn+1 . As already described, the sequences of rejected
points Pn and Pn+1 in the “shrinkage” procedure are the same,
which means that the probability p(Pn |Jn , Un , xn ) is equal to
p(Pn+1 |Jn+1 , Un+1 , xn+1 ). Thus, we only need to prove the following
equation:
kn × p(xn+1 |r, xn ) = kn+1 × p(xn |π (r ), xn+1 ).
As we have demonstrated above, the definitions of π (U) and
π (J) make sure that the final interval containing xn overlaps the
final interval containing xn+1 completely while the definition of
π (P) keeps the shrinkage of these two interval synchronized.
Denote this identical interval as Sˆ and we obtain:
p(xn+1 |r, xn ) b ˆ (xn+1 ) kn+1
= 2S = ,
p(xn |π (r ), xn+1 ) b2Sˆ (xn ) kn
where b2Sˆ (x ) is the b2 distribution over Sˆ.
So far we have proved that Eq. (17) holds, which means that
the distribution p(x) is invariant during each step in the Markov
chain. This completes the proof.
5. Experiments and results
In this section, we demonstrate the performance of the de-
composed slice sampling by simulation experiments and real
data experiments. We compare the proposed DSS methods with
standard slice sampling and Hamiltonian Monte Carlo. For each
compared method, the parameters are allowed to be tuned as
optimal. The estimate of the typical size of a slice, w, is manually
set to be 15, and the integer that limits the maximum size of a
slice interval (or the maximum number of iterations) m is set to
be 300 in all the related experiments.
5.1. Effectiveness test
Though we have theoretically proved the correctness of DSS,
it is more intuitive to verify this proposed method through some
tests. For the convenience of visualization, we evaluate the perfor-
mance of DSS on two univariate distributions and two bivariate
distributions. We employ slice sampling in the third step of DSS to
pick new points and name this specific DSS method as DSS-ss. We
draw 20,0 0 0 samples in each experiment. Histograms of samples
from 1D distributions compared with the corresponding density
contour are shown in Fig. 3. Scatter diagrams of samples taken
from 2D distributions are shown in Fig. 4.
It can be seen that by using the DSS, both histograms of sam-
ples from 1D distributions and scatter diagrams of samples from
2D distributions completely fit the corresponding target density
contours. This means that the resulting samples correctly follow
their target distributions, which provides intuitional evidence for
the effectiveness of DSS.
5
5.2. Simulation experiment
In this subsection, we respectively sample the unimodal and
multi-modal Gaussian distributions from 1 to 60 dimensions
(15) by using DSS-ss and the slice sampling, and then compare the
CPU-time and the maximum mean discrepancy (MMD) of these
two methods.
5.2.1. Evaluation criteria
In order to evaluate the time performance of different methods,
we take the CPU time consumed during sampling as one of the
(16) evaluation criteria. For judging the correctness of the sampling
results, we use the MMD as the second criterion.
MMD [30,31] was first proposed for the two-sample test prob-
lem to determine whether two distributions p and q are identical.
It supposes that taking all the sample spaces generated by distri-
butions as the input of the function f, if the means of the corre-
sponding function values are equal, then the two distributions can
be considered to be the same. It is now generally used to measure
(17) the similarity between two distributions. In this experiment, we
choose the function f as the kernel function κ (x, x ) = 1 + xT ∗ x .
Thus the square of MMD can be expressed as:
MMD2 [F, p, q] = E x,x [κ (x, x )] − 2E x,y [κ (x, y )] + E y,y [κ (y, y )],
(19)
where x is a random variable obeying the distribution p, and y is
a random variable obeying distribution q.
(18)
In our experiments, x represents samples picked by the pro-
posed methods or comparison methods, and y represents random
vectors from Gaussian distribution generated by the library func-
tion mvnrnd in Matlab. The similarity between x and y is measured
by calculating the MMD. Smaller values of MMD mean that x is
more similar to y, that is, the distribution of x is closer to the
target Gaussian distribution. A value of zero for MMD indicates
that x and y are equally distributed.
5.2.2. Settings
For experiments of sampling unimodal Gaussian, we set
b1 (x ) = N (μ1 , 1 ) and b2 (x ) = N (μ2 , 2 ). For the multi-
modal case, we set b1 (x ) ∝ N (μ3 , 3 ) + N (μ4 , 3 ) and
b2 (x ) ∝ N (μ5 , 4 ) + N (μ6 , 4 ). As for these hyper-parameters, μi
is an n-dimensional vector randomly generated from [0,4] and di-
agonal covariance matrix j obtains n diagonal elements randomly
from [0,30], where i = 1, . . . , 6, j = 1, . . . , 4 and n = 1, . . . , 60.
We respectively sample the unimodal and multi-modal Gaussian
distributions from 1 to 60 dimensions by using DSS-ss and the
slice sampling. We draw 10,0 0 0 samples for each experiment,
and perform experiments for 10 times on randomly assigned
hyper-parameters. Note that in order to ensure the repeatability of
the experiments, we fixed all the random seeds as constants.
5.2.3. Results and analysis
The average CPU time and its standard deviations in 10 re-
peating experiments for sampling 10,0 0 0 points by DSS and slice
sampling in different dimensions are shown in Fig. 5. From the
figures, we can see that in the case of unimodal distributions, DSS
is always more time-consuming than slice sampling, but in the
multi-modal situation it turns out to be the opposite. A reasonable
explanation is that in the case of unimodal Gaussian, slice sam-
pling only needs to sample one unimodal Gaussian distribution,
while DSS needs to obtain slice interval from b1 (x) and perform
a full slice sampling on b2 (x), which takes more than the time to
only simulate one unimodal distribution. For the case of multi-
modal Gaussian, if the likelihood and the prior are mixed Gaussian
of two components, the slice sampling needs to sample a mixed
6 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021

Fig. 3. Histograms of samples taken by DSS from 1D distributions compared with the corresponding density contour: (a) 1D unimodal Gaussian, b1 (x ) = N (3, 1 ) and b2 (x ) =
N (1, 1 ) and (b) 1D bimodal Gaussian, b1 (x ) ∝ N (0, 4 ) + N (10, 4 ) and b2 (x ) = N (4, 4 ).

Fig. 4. Scatter diagrams of samples taken by DSS from 2D distributions compared with the corresponding density contour: (a) 2D unimodal Gaussian, b1 (x ) = N (3, I ) and
b2 (x ) = N (1, I ) and (b) 2D bimodal Gaussian, b1 (x ) ∝ N (0, I ) + N (8, I ) and b2 (x ) = N (4, I ).

Fig. 5. The averaged CPU-time (s) of DSS and slice sampling in different dimensions: (a) Unimodal Gaussian distributions and (b) Multi-modal Gaussian distributions.
 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021 7

Fig. 6. The averaged MMDs of DSS and slice sampling: (a) Unimodal Gaussian distributions and (b) Multi-modal Gaussian distributions.

Gaussian with four components, while DSS samples the mixed Table 1
Dataset description.
Gaussian with only two components in each step. The results of
CPU time indicates DSS is more time-saving than slice sampling Dataset # of instances # of features
when the target distribution is not a simple unimodal distribution. Iris 150 4
We also record the average MMD of the samples produced in Mammographic 830 5
every 10 repeating experiments; thus both methods have recorded Brestcancer 683 9
60 MMDs. Taking the MMD of DSS as the horizontal axis and the Fertility 100 9
Heart 270 13
MMD of slice sampling as the vertical axis, we obtain Fig. 6. Note
Pabomovie 2000 20
that the closer the point is to the line y = x, the more similar Parkinsons 195 22
the MMDs of DSS and slice sampling are. We can see that points Germannuber 1000 24
are close to the line in both subfigures, especially in the case of Chen-2002 179 85
Gordon-2002 181 1626
multi-modal. This means that DSS has similar performance with
slice sampling in MMD, especially in complex cases.
From the simulation experiments, we can conclude that as the
target distribution becomes complex, the proposed DSS is more compared with HMC, the decomposed slice sampling employs
time-saving than slice sampling, but has similar performance with HMC (DSS-hmc) in its third step.
slice sampling in MMD.
5.3.1. Datasets and setups
5.3. Real data experiment Table 1 shows the information about the datasets used in the
classification tasks. The first eight datasets can be found in the UCI
In this subsection, we apply the proposed method to the data repository [34], and the last two datasets are cancer gene ex-
Bayesian logistic regression model (BLR) [32] for classification pression datasets that are available at [35]. Feature dimensions are
tasks. BLR is a famous linear discriminant model for binary between 4 to 1626, and the total data instances range from 100
classification. The model has a great deal of theoretical support, to 20 0 0. All the datasets contain 2 classes except for the classical
and performs prominently well in terms of predictive accuracy Iris dataset which has 3 classes. Thus, for Iris, we experimented
[32,33]. BLR uses the sigmoid function to describe the relationship with the instances of label “Versicolour” and “Virginica” because
between data and prediction, i.e., y = σ (wT φ ), where y is the pre- the classification accuracies of the other combinations are always
dictive probability, φ is the feature vectors, and w is parameters. 100% for each method.
By considering a Gaussian prior and a likelihood function with After normalization, all of the datasets are randomly split into
a multiplicative form, the non-normalized form of the posterior the training set and test set by a ratio of 7:3. Since the Bayesian
distribution of w can be obtained by the product of prior and logistic regression seeks a Gaussian representation of the posterior
likelihood function. In the following experiments, we employ distribution, for simplicity, we choose Gaussian priors N (0, I ). We
four MCMC methods to approximate the integral of the posterior draw 50 0 0 samples for each experiment, and perform experiments
distribution, i.e., two methods that are based on the decomposed for 10 times on randomly assigned datasets. The average accura-
slice sampling framework, standard slice sampling and HMC. The cies in percentage with the corresponding standard deviations are
DSS methods applied on BLR models set b1 (x) proportional to the reported in Table 2.
likelihood function and b2 (x) as the prior.
In order to evaluate the performance of the decomposed slice 5.3.2. Classification performances
sampling, we design two groups of comparison experiments on ten From Table 2, we can see that methods based on the DSS per-
real-world datasets. The first comparison is DSS-ss and the original form better than the slice sampling and HMC in higher dimension
slice sampling (SS). As mentioned in Section 5.1, here DSS-ss cases. For the first group, DSS-ss outperforms SS on the last five
represents the decomposed slice sampling that employs the slice datasets whose feature dimensions are no less than twenty. On the
sampling in step 3, which avoids the comparison being interfered lower-dimensional datasets (Iris and Brestcancer) where the tar-
by other sampling methods. Similarly, in the second group, in get distributions are simple, DSS-ss performs slightly inferior to
order to be constant with the method being compared, when SS. For the second group, the performance of DSS-hmc is never
8 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021

Table 2
Averaged classification accuracies and standard deviations (%) for all datasets.

Dataset slice sampling HMC methods

DSS-ss SS DSS-hmc HMC

Iris 89.00 ± 0.1610 90.00 ± 0 90.00 ±0 85.33 ± 0.1146

Mammographic 73.49 ±0 73.49 ± 0 73.29 ± 0 73.53 ± 0.0013
Brestcancer 85.02 ± 0.0024 85.07 ± 0.0025 84.93 ± 0.0043 85.07 ± 0.0034
Fertility 96.67 ±0 96.67 ± 0 96.67 ±0 96.67 ±0
Heart 87.65 ±0 87.65 ± 0 87.78 ± 0.0070 87.41 ± 0.0052
Pabomovie 81.53 ± 0.0028 81.33 ± 0.0030 81.38 ± 0.0025 81.32 ± 0.0018
Parkinsons 82.76 ±0 81.55 ± 0.0116 81.03 ±0 80.35 ± 0.0121
Germannuber 76.27 ± 0.0041 75.97 ± 0.0037 75.80 ± 0.0029 74.80 ± 0.0311
Chen-2002 90.74 ± 1.9520 90.19 ± 1.9618 90.93 ± 1.3664 89.63 ± 1.7891
Gordon-2002 96.53 ± 1.8383 95.99 ± 2.1673 99.21 ± 0.9887 98.68 ± 0.9027

overtaken by HMC unless on two low-dimensional datasets, Mam-
mographic and Brestcancer. For clarifying the performance differ-
ences between different methods in higher dimension cases, we
run paired t-test on averaged accuracies of datasets whose feature
dimensions are no less than twenty. The p-value for the first group,
DSS-ss vs SS, is 0.0335, and the p-value for the second group, DSS-
hmc vs HMC, is 0.0276. Both p-values are less than 0.05, which
indicates the significant improvements of the proposed method.
In short, the experiment results again reveal the superior per-
formance of the proposed method, especially in high-dimensional
cases where the target distributions are complex.
6. Conclusions and future work
In this paper, we have proposed a novel sampling framework,
namely, the decomposed slice sampling method, which is used to
efficiently sample from unnormalized factorized distributions. This
slice sampling based method divides the target distribution into
two multipliers, alternating between drawing an auxiliary variable
from the first multiplier to define the horizontal slice and sampling
a new point from the second term over the interval given by the
first multiplier. The correctness and effectiveness of the proposed
MCMC framework have been verified theoretically and experimen-
tally. The experimental results have demonstrated that the DSS
based sampling methods outperform the slice sampling and the
HMC, especially in complex situations. Thus, it is recommended
to use DSS for sampling when the target distribution is complex.
Though the traditional slice sampling method is suitable for simple
unimodal distributions, in practice, most target distributions are
more complex. Therefore, the DSS proposed is a practical sampling
method, and can be more widely applied to sampling from factor-
ized distributions. In the future work, we will consider upgrading
our method by using some improved algorithms for solving slice
intervals.
Acknowledgments
This work is supported by the National Natural Science Foun-
dation of China under Project 61673179, Shanghai Knowledge Ser-
vice Platform Project (No. ZF1213), and the Fundamental Research
Funds for the Central Universities.
each step in the Markov chain. To show invariance, we first sup-
pose that the initial point x0 follows the distribution p(x). Thus the
joint distribution of x0 and u follows:
1
i f 0 < u < p˜ (x ),
p(x, u ) = Zp
(A.1)
0 otherwise,

where Z p = p˜ (x )dx and p˜ (x ) is a computable function that is pro-
portional to p(x). The marginal distribution of x is obtained by in-
tegrating u as follows:
  p˜(x )
1 p˜ (x )
p(x, u ) du = du = = p( x ) . (A.2)
0 Z p Zp
Note that in the subsequent update from x0 to x1 , the auxiliary
variable u is unchanged. If the joint distribution p(x0 , u) and p(x1 ,
u) are identically distributed, then p(x0 ) and p(x1 ) are also identi-
cally distributed by integrating u. So does the subsequent update
from x1 to x2 , and so on. Thus, we only need to show that p(xn , u)
and p(xn+1 , u ) are identically distributed in the step of sampling
xn+1 given xn (n = 0, 1, 2, . . .). Because these steps do not change
u, it is equal to prove that the conditional distribution for x given
u is invariant in step 2 and step 3 described in Section 3. As in-
troduced in Section 3, p(x|u) is uniform over S = {x : u < p˜ (x )}. We
can show the invariance of the conditional distribution by proving
that the updates satisfy the detailed balance:
p(xn |u ) p(xn+1 |xn , u ) = p(xn+1 |u ) p(xn |xn+1 , u ). (A.3)
For a uniform distribution, this reduces to proving that
p(xn+1 |xn , u ) = p(xn |xn+1 , u ), (A.4)
for any xn and xn+1 in S.
We can demonstrate that the update procedure satisfies the de-
tailed balance by proving the following stronger result:
p(xn+1 , r|xn , u ) = p(xn , π (r )|xn+1 , u ), (A.5)
where r denotes random choices in the sampling procedure, or
more specifically, U and J (or V, K) in the “stepping out” step, and
the sequence of rejected points P in the “shrinkage” step. π (r) is a
first order one-to-one mapping that depends on xn and xn+1 . We
can obtain Eq. (A.4) by integrating all the possible choices of r.
Next, we define the mapping π (r). We first define the π (U) that
maps Un to Un+1 as follows:
Un+1 = Frac(Un + (xn+1 − xn )/w ), (A.6)

Appendix A. Proof of slice sampling
This proof is based on the idea in Section 4.8 of [8], and we pro-
vide a more detailed explanation of the correctness of slice sam-
pling. For the natural language description is difficult to under-
stand in [8], we use the form of formulas to explain it rigorously.
The proof is as follows.
To show that slice sampling is a correct procedure, it is neces-
sary to prove that the target distribution p(x) is invariant during
where the function Frac(x ) = x − Floor(x ) retains the fractional
part of x. The definition of π (U) makes the left bound Ln+1 of
the initial slice interval In+1 that contains xn+1 align with the left
bound Ln of the initial slice interval In that contains xn , or ex-
plained in formulation, (Ln+1 − Ln ) mod w = 0.
The “stepping out” procedure will generate a random number
J ∈ {0, . . . , m − 1}, and so we define the mapping π (J) that maps Jn
to Jn+1 as follows:
Jn+1 = Jn + (xn+1 /w − Un+1 ) − (xn /w − Un ), (A.7)
 J. Wang and S. Sun / Pattern Recognition 97 (2020) 107021
where (xn+1 /w − Un+1 ) − (xn /w − Un ) denotes the distance (w
units) between the left bound Ln and Ln+1 .
When the transition between xn and xn+1 is possible, it is easy
to find that the final slice interval will be the same for xn as for
xn+1 with the above definitions. Actually, these two intervals will
be the same when the interval for xn first contains xn+1 . Then in
the following “stepping out” steps, the two intervals will stay the
same. For the situation that the transition between xn and xn+1 is
impossible, Eq. (A.5) will hold because both sides of it are zero.
Finally, π maps the sequence of rejected points Pn used to
shrink the interval that contains xn to the sequence of rejected
points Pn+1 generated from the interval found from xn+1 . Thus, Pn
and Pn+1 are the same sequence.
After the completion of the definition of π , we return to prove
Eq. (A.5), which can be expanded as:
p(Un |xn ) p(xn+1 , Jn , Pn |Un , xn )
= p(Un+1 |xn+1 ) p(xn , Jn+1 , Pn+1 |Un+1 , xn+1 ). (A.8)
Here, the unchanged u is omitted for conciseness. Obviously,
p(Un |xn ) is uniform over (0,1). Although the value of Un+1 depends
on xn and xn+1 , p(Un+1 ) is also uniform over (0,1) because Un+1 is a
linear transformation of Un . Therefore, the first term of both sides
in Eq. (A.8) can be crossed out and the equation can be written
as:
p(Jn |Un , xn ) p(xn+1 , Pn |Jn , Un , xn )
= p(Jn+1 |Un+1 , xn+1 ) p(xn , Pn+1 |Jn+1 , Un+1 , xn+1 ). (A.9)
Similarly, p(Jn |Un , xn ) and p(Jn+1 |Un+1 , xn+1 ) are both uniform
over (0,1). Therefore, Eq. (A.9) can be simplified and transformed
as:
p(Pn |Jn , Un , xn ) p(xn+1 |r, xn )
= p(Pn+1 |Jn+1 , Un+1 , xn+1 ) p(xn |π (r ), xn+1 ). (A.10)
Here r represents Un , Jn and Pn , while π (r) represents Un+1 , Jn+1
and Pn+1 . As already described, the sequences of rejected points Pn
and Pn+1 in the “shrinkage” procedure are the same, which means
that p(Pn |Jn , Un , xn ) is equal to p(Pn+1 |Jn+1 , Un+1 , xn+1 ). Thus, we
only need to prove the following equation:
p(xn+1 |r, xn ) = p(xn |π (r ), xn+1 ). (A.11)
As we have demonstrated above, the definitions of π (U) and
π (J) make sure that the final interval containing xn overlaps the
final interval containing xn+1 completely while the definition of
π (P) keeps the shrinkage of these two interval synchronized. Con-
sequently, xn and xn+1 always obey the same uniform distribution,
i.e., Eq. (A.11) holds, which means that the target distribution p(x)
is invariant during each step in the Markov chain. This completes
the proof.
References
[1] D.M. Higdon, Auxiliary variable methods for Markov chain Monte Carlo with
applications, Publ. Am. Stat. Assoc. 93 (1998) 585–595.
[2] C. Andrieu, N. De Freitas, A. Doucet, M.I. Jordan, An introduction to MCMC for
machine learning, Mach. Learn. 50 (2003) 5–43.
[3] S. Sun, A review of deterministic approximate inference techniques for
Bayesian machine learning, Neural Comput. Appl. 23 (2013) 2039–2050.
[4] R.H. Swendsen, J.-S. Wang, Nonuniversal critical dynamics in Monte Carlo sim-
ulations, Phys. Rev. Lett. 58 (1987) 86–88.
[5] M.W. Liechty, J. Lu, Multivariate normal slice sampling, J. Comput. Graph. Stat.
19 (2010) 281–294.
[6] S. Geman, D. Geman, Stochastic relaxation, Gibbs distributions, and the
Bayesian restoration of images, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984)
721–741.
9
[7] R.G. Edwards, A.D. Sokal, Generalization of the Fortuin-Kasteleyn-Swend-
sen-Wang representation and Monte Carlo algorithm, Phys. Rev. D 38 (1988)
2009–2012.
[8] R.M. Neal, Slice sampling, Ann. Stat. (2003) 705–741.
[9] D.J. Nott, D. Leonte, Sampling schemes for Bayesian variable selection in gen-
eralized linear models, J. Comput. Graph. Stat. 13 (2004) 362–382.
[10] S.K. Kinney, D.B. Dunson, Fixed and random effects selection in linear and lo-
gistic models, Biometrics 63 (2007) 690–698.
[11] D.K. Agarwal, A.E. Gelfand, Slice sampling for simulation based fitting of spatial
data models, Stat. Comput. 15 (2005) 61–69.
[12] T.D. Kulkarni, P. Kohli, J.B. Tenenbaum, V. Mansinghka, Picture: a probabilistic
programming language for scene perception, in: IEEE Conference on Computer
Vision and Pattern Recognition, Vol. 1, 2015, pp. 4390–4399.
[13] P.O. Lewis, M.T. Holder, K.E. Holsinger, Polytomies and Bayesian phylogenetic
inference, Syst. Biol. 54 (2005) 241–253.
[14] B. Shahbaba, R.M. Neal, Gene function classification using Bayesian models
with hierarchy-based priors, BMC Bioinform. 7 (2006) 448–456.
[15] S. Sun, C.M. Greenwood, R.M. Neal, Haplotype inference using a Bayesian hid-
den Markov model, Genet. Epidemiol. 31 (2007) 937–948.
[16] I. Murray, R.P. Adams, D.J.C. Mackay, Elliptical slice sampling, J. Mach. Learn.
Res. 9 (2009) 541–548.
[17] S. Brooks, A. Gelman, G.L. Jones, X.L. Meng, Handbook of Markov chain Monte
Carlo, Chance 25 (2011) 53–55.
[18] S. Duane, A.D. Kennedy, B.J. Pendleton, D. Roweth, Hybrid Monte Carlo, Phys.
Lett. B 195 (1987) 216–222.
[19] C.M. Bishop, Pattern Recognition and Machine Learning (Information Science
and Statistics), Springer-Verlag, New York, 2006.
[20] Z. Wang, S. Mohamed, N.D. Freitas, Adaptive Hamiltonian and Riemann mani-
fold Monte Carlo samplers, Statistics (2013) 1462–1470.
[21] M.D. Hoffman, A. Gelman, The no-U-turn sampler: adaptively setting path
lengths in Hamiltonian Monte Carlo, J. Mach. Learn. Res. 15 (2014) 1593–1623.
[22] M. Girolami, B. Calderhead, Riemann manifold Langevin and Hamiltonian
Monte Carlo methods, J. R. Stat. Soc. 73 (2015) 123–214.
[23] N. Tripuraneni, M. Rowland, Z. Ghahramani, R. Turner, Magnetic Hamiltonian
Monte Carlo, in: Proceedings of the 34th International Conference on Machine
Learning, Vol. 70, 2017, pp. 3453–3461.
[24] M. Thompson, R.M. Neal, Covariance-adaptive slice sampling, Statistics (2010).
arXiv: 1003.3201
[25] M.B. Thompson, R.M. Neal, Slice sampling with adaptive multivariate steps: the
shrinking-rank method (2010). arXiv:1011.4722.
[26] M.M. Tibbits, C. Groendyke, M. Haran, J.C. Liechty, Automated factor slice sam-
pling, J. Comput. Graph. Stat. 23 (2014) 543–563.
[27] M.M. Tibbits, M. Haran, J.C. Liechty, Parallel multivariate slice sampling, Stat.
Comput. 21 (2011) 415–430.
[28] I. Murray, M. Graham, Pseudo-marginal slice sampling, in: Artificial Intelli-
gence and Statistics, Vol. 51, 2016, pp. 911–919.
[29] R.M. Neal, Probabilistic inference using Markov chain Monte Carlo methods,
Technical Report CRG-TR-93-1, Department of Computer Science, University of
Toronto, 1993.
[30] A. Gretton, K.M. Borgwardt, M.J. Rasch, B. Schölkopf, A. Smola, A kernel
two-sample test, J. Mach. Learn. Res. 13 (2012) 723–773.
[31] A. Rozantsev, M. Salzmann, P. Fua, Beyond sharing weights for deep domain
adaptation, IEEE Trans. Pattern Anal. Mach. Intell. 41 (2019) 801–814.
[32] A. Genkin, D.D. Lewis, D. Madigan, Large-scale Bayesian logistic regression for
text categorization, Technometrics 49 (2007) 291–304.
[33] G. Kumar, V. Govindaraju, Bayesian background models for keyword spotting
in handwritten documents, Pattern Recognit. 64 (2017) 84–91.
[34] D. Dheeru, E. Karra Taniskidou, UCI machine learning repository, 2017.
[35] Cancer gene expression data sets, http://schlieplab.org/Static/Supplements/
CompCancer/datasets.htm. Accessed July 12, 2019.
Jiachun Wang is a student in the Pattern Recognition and Machine Learning Re-
search Group, East China Normal University, Shanghai, China. Her research interests
include Markov chain Monte Carlo sampling, probabilistic models, and their appli-
cations.
Shiliang Sun received the Ph.D. degree in pattern recognition and intelligent sys-
tems from Tsinghua University, Beijing, China, in 2007.
He is a Professor with the Department of Computer Science and Technology and
the Head of the Pattern Recognition and Machine Learning Research Group, East
China Normal University, Shanghai, China. From 2009 to 2010, he was a Visiting
Researcher with the Department of Computer Science, Centre for Computational
Statistics and Machine Learning, University College London, London, U.K. In 2014, he
was a Visiting Researcher with the Department of Electrical Engineering, Columbia
University, New York, NY, USA. His current research interests include approximate
inference, sequential modeling, multi-view learning, deep learning, and their appli-
cations. His research results have expounded in 100+ publications at peer-reviewed
journals and conferences.
Prof. Sun is on the Editorial Board of multiple international journals, including Pat-
tern Recognition and IEEE Transactions on Neural Networks and Learning Systems.