4 - Theory of SPs

Lecture Notes: RANDOM VIBRATION OF STRUCTURES / SPRING 2009 / Page: 1
Lecturer: Prof. APOSTOLOS S. PAPAGEORGIOU

PART (4): THEORY OF RANDOM (STOCHASTIC) PROCESSES
RANDOM (STOCHASTIC) PROCESSES
A stochastic process, or a random process, is a mathematical model of a dynamical

process whose dependence on a parameter is governed by probabilistic laws.
The theory of Stochastic Processes (SP) has evolved as a generalization of the

concept of Random Variables (RV). In order to apply probabilistic methods we must
have an experiment which can be repeated many times under similar conditions and
which can lead to different outcomes. The set Ω (i.e. the Sample Space) of all possible
outcomes of a trial / experiment plays a basic role in contemporary axiomatic
formulation of Probability Theory. A RV can be used to model random events for which
the outcomes of repeated trials are a real or complex number. In many problems the
outcome of a trial / experiment is not a number but a function of one or more
parameters, such as time or space or both. In such cases the outcome of each trial /
experiment is called a realization or a sample function and the collection of all
possible sample functions is called the ensemble of the random process.
FIGURE: Ensemble of a Stochastic Process
The Figure shows the realizations of a SP where is the parameter. Thus a SP is a

function of two variables: the parameter and the sample point . A more complete
notation of a stochastic process is thus , , ∈ , ∈Ω.
 For a fixed , , ∙ is a function of the Sample Space Ω and is
thus a random variable (RV).
Last update: MARCH 2011

 On the other hand, for a fixed , ∙, defines a function of the

variable and is a realization, or sample function of the stochastic process.
For expediency, the dependence of a SP on is not explicitly shown.
Thus, a SP , can be regarded either:

 as a (parameterized) family of RV depending on the parameter , or
 as a family of realizations depending on the sample point .
[NOTE: In the latter case, to specify the SP we have to provide the PDF (or pdf) of
occurrence of the various realizations. This leads to the definition of a particular
measure (the probability measure) on the function space of realizations, and by
specifying this measure, we specify the SP. This approach to the concept of a SP,
originated in Kolmogorov’s work (KOLMOGOROV, 1956) on the foundations of
probability theory, has been very fruitful in the further development of the theory. It
can be shown that specifying a SP by defining a probability measure on the set of
realizations of the SP includes as special cases all the other ways of specifying a SP
(DOOB, 1953).]
DOOB, J.L. (1953). Stochastic Processes, John Wiley and Sons, Inc., New York
KOLMOGOROV, A.N. (1956). Foundations of the Theory of Probability, 2nd English Edition,
Translation edited by N. Morrison, with an added bibliography by A.T. Bharucha-Reid,
CHELSEA Publishing Co., New York
Since , ∈ , at a fixed can be either a discrete or a continuous RV, a SP

,
∈ , will be called:
 discrete-valued when at each is a discrete RV;
 continuous-valued when at each is a continuous RV;
A SP can also be classified according to the property of its index set :
 If is discrete (e.g. 0,1,2, ⋯ ), the SP is said to be a discrete-
parameter SP;
 If is continuous (e.g. : 0 ), the SP is said to be a continuous-
parameter SP.
We will be primarily concerned with continuous-parameter, continuous-valued SP’s that

are real-valued.

CLASSIFICATION OF STOCHASTIC PROCESSES
Stochastic Process
Continuous Random Sequence or
STATE SPACE Random Function
Discrete Parameter Continuous Parameter
Discrete
Chain Chain
Discrete Continuous
PARAMETER SET

CHARACTERIZATION OF STOCHASTIC PROCESSES
There are several ways of characterizing a SP. The most versatile for our purposes
exploits the following two facts:
1) can be interpreted as a (parameterized) family of RV indexed by , and
2) A finite set of RVs is completely specified by its joint distribution function.
Therefore, a SP is specified as follows:
Definition/Specification of a Stochastic Process
If for every finite set , ,⋯, of ∈ , there corresponds a set of RVs,

, , ⋯ , , having a well-defined Joint Probability
Distribution Function (JPDF)
,⋯, ; ,⋯, ∩⋯∩
called the nth distribution function of , then this family of joint distribution
function defines the SP , ∈ , provided it satisfies the following two
compatibility conditions, referred to as Kolmogorov compatibility conditions:
a) The condition of consistency, i.e., for any ,
,⋯, , ∞, ⋯ , ∞ ; ,⋯, ,⋯, ,⋯, ; ,⋯,
which indicates that marginal distributions can be consistently generated from higher
dimensional distributions, and
b) The symmetry property in that it is invariant under an arbitrary permutation of

the indices 1, ⋯ , , i.e.,
,⋯, ; ,⋯, ,⋯, ; ,⋯,
where , ,⋯, is an arbitrary permutation of 1, ⋯ , .
The corresponding nth density function of , is given by
,⋯, ; ,⋯,
,⋯, ; ,⋯,
⋯

MOMENTS OF STOCHASTIC PROCESSES
Some of the most important properties of a SP are characterized by its moments,

particularly those of the first and second order.
The nth moment, , of a SP at a given ∈ , is defined by
≝ ,
 The first moment, , is the mean of the SP at ∈ , and is sometimes

denoted as or simply .
 The mean square value of at ∈ is given by .
 is referred to as the root mean square of at ∈ .
The nth central moment of a SP at a given ∈ , is defined by
 The second central moment, , is the variance of the SP at

∈ , and is denoted as .
An important measure of linear independence between and is contained in

the auto-correlation function , which plays a central role in the theory of
SPs. The auto-correlation function is defined as follows:
, ≝ , ; ,
It is a function of and in general.
The corresponding central moment, referred to as the auto-covariance function

, , is defined as follows:

, ≝
, ; ,
It can easily be verified that
, ,
When , then
, ,
that is, , .
The normalized auto-covariance function is called correlation-coefficient function and

is denoted by
,
, ≝
It is analogous to the correlation coefficient defined in the theory of RVs with the
same “measure of linear independence” interpretation.
It can be demonstrated (in the same fashion as for the correlation coefficient ) that
| , | 1
When , 0 for all , the SP , ∈ is called an

uncorrelated process.
Similar definitions may be provided for the cross-correlation and cross-covariance

functions of two different RPs, say and , specifically,
, ≝
and
, ≝
,
In the definitions of the second moment functions, “auto-” refers to the same SP while
“cross-” refers to a different SP.

The mean and correlation function (or covariance function) of a SP provide a partial
characterization of the process, referred to as second-moment characterization. It is
clear that SPs with the same second-moment characteristics can have very
different sample properties.
Some Important Properties of Second-Moment Characteristics
We note some important properties associated with the correlation functions. Properties
of the covariance functions can be similarly deduced.
(a) Symmetry:
It follows easily from the definition of , and , that, for
every , ∈ , the following relations are valid:
, ,
, ,
, , ,
(b) For every , ∈ , the following relations follow as consequences of the

Cauchy-Schwarz inequality:
, , ,
, , ,
(c) The function , [ , ] is non-negative definite on ;

that is , for every , ,⋯, ∈ and for any arbitrary function defined
on ,
, 0
The above inequality is demonstrated as follows: Consider a RV defined by

We get
(d) Consider two RPs, and . It is clear that
| | 0
2 0
2 0
, , ,
Therefore
, , ∓2 ,
(e) The addition of a deterministic function to a SP does not alter its covariance
function.
Consider
where is a SP with mean and covariance function , ,

and is a deterministic function.
Then
and

In particular if , then , which has zero

mean and possesses the same covariance function as .
o Thus when we investigate the covariance function of a SP, there is no

loss of generality in assuming that it has a zero mean. It is convenient to
make such an assumption to simplify the algebra.
o In view of the abovementioned property [regarding the equality of the

covariances of the processes and ], we shall
assume the mean to be zero, unless specified otherwise, and drop the
distinction between correlation and covariance functions.
__________________
NOTE: CAUCHY-SCHWARZ INEQUALITY
The Cauchy-Schwarz Inequality [named after Augustin-Louis Cauchy and the German
mathematician Hermann Amandus Schwarz (1843-1921)] is the following:
in which and are real RVs with finite second moments.
Proof: Consider the function
in which is real. Now 0 no matter what may be the value of .

Thus it follows that
2 0
Such a function is said to be a nonnegative quadratic form in . A plot of versus

must either not intercept the -axis at all or it must touch the -axis at one and only one
point.
The minimum value of is attained for ⁄ .
 If 0 (i.e. no intersection of with the -axis) then

Therefore,
 If 0 (i.e. the plot of touches the -axis) then, from the above
expressions we conclude that
SIDE NOTE:
In order to apply the Cauchy-Schwarz Inequality to derive the inequalities involving the
auto-(or cross-) correlation functions stated above, we assume that one of the two RVs
(involved in the Cauchy-Schwarz Inequality) is while the other one is (or )
__________________

STATIONARY STOCHASTIC PROCESSES
In terms of statistical regularity, SPs can be grouped into two classes:
 Stationary Stochastic Processes

 Non-stationary Stochastic Processes
As we have seen, the nth probability distribution function, ,⋯, ; ,⋯, of a

SP , ∈ , is in general a function of , ⋯ , as well as , ⋯ , . A non-
stationary process is one whose distribution functions depend upon values of the time
parameters explicitly. Its statistical behavior thus depends upon the absolute origin
of time.
A SP , ∈ , is said to be stationary, or strictly stationary, if its associated

probability distributions stay invariant under an arbitrary translation of the time
parameter; that is for each and for an arbitrary , we have
,⋯, ; ,⋯, ,⋯, ; ,⋯,
∈ ∈ , 1,2, ⋯ ,
Setting in the above expression, we see that the probability distribution

depends upon the time parameters only through their differences. In other words,
statistical properties of a stationary process are independent of the absolute time
origin.
We can easily see that a stationary SP , ∈ , possesses the following important

properties for its moments, if they exist:
 Since the first distribution , is not a function of , we have
for any 1,2, ⋯ .
 Since
, ; , , ;
we have
 In view of the symmetry property , , of the auto-

correlation (auto-covariance) function, it follows that
The auto-correlation (auto-covariance) function of a stationary process is thus an

even function of .
 always exists and is finite. Furthermore,
| | 0
i.e., the auto-correlation function attains its maximum at zero lag (i.e.
0 ).
This result is a direct consequence of
, , ,
which for stationary SPs takes the form
0 0
or
| | 0
Notice that since

0
we also have
0 0
NOTE: It readily follows that if and exist (there existence is

guaranteed for a SP that is differentiable in the mean square), then 0
and 0.
 If | | is continuous at the origin, then it is uniformly continuous in

.
In order to show this, let us consider the difference

Using the Cauchy-Schwarz Inequality (stated and proved above) we obtain
| | 2 0 0
Since is continuous at the origin, then 0 → 0 as

→0.
Hence, the difference tends to zero with uniformly in
.
 We have already demonstrated that the auto-correlation function , of

any SP (not necessarily stationary) is non-negative definite. The same applies
for the particular case of a stationary SP, i.e., for every , , ⋯ , ∈ and for
any arbitrary function defined on ,
NOTE: A theorem due to S. BOCHNER asserts that every nonnegative-definite

function (i.e. a function that satisfies the above inequality) has a nonnegative Fourier
Transform, if such a transform exists, i.e.
1
Φ 0
2
The corresponding inverse Fourier Transform is
Observe that the mean-square value of the stationary SP may

be expressed as follows:
0 Φ
As it will become evident in later sections, the function Φ plays a

dominant role in the theory of stationary SPs. The physical significance of the
function Φ is revealed by the above equation which expresses the mean-
square value of a stationary SP as a sum of infinitesimal components

Φ . Thus Φ describes the distribution of the total mean-square

value over the frequency domain, hence the name Mean-Square Spectral
Density Function, or, Spectral Density of the SP . An alternative name
for the function Φ , the Power Spectrum, is also used in the literature
since is often a measure of average energy. For example, if
represents the random displacement of a single-degree-of-freedom (SDOF)
mechanical system, then is proportional to the average potential energy
in the system. If represents the random velocity, then is
proportional to the average kinetic energy. The Power Spectral Density will be
studied extensively below.
An alternative proof of the fact that Φ attains only nonnegative values

will be given in a later section.
The two equations above (appearing in boxes), expressing Φ and

in terms of each other give a mathematical statement of the well-known
WIENER-KHINTCHINE Theorem, which asserts that for a weekly stationary
SP the autocorrelation function R and the spectral density function Φ form a
Fourier transform pair, i.e. ↔Φ .
The importance of the autocorrelation function R and the spectral

density function Φ is because they provide average amplitude and frequency
information about the sample histories and that
(i) they can be measured with available data-processing techniques;

(ii) they are closed with respect to linear time invariant operations in the
sense that if these statistics are known for the excitation, then it is
possible to obtain the corresponding statistics for the response of a linear
time invariant dynamic system; and
(iii) they often provide adequate information about the response for making
engineering decisions concerning the severity of the vibration and the
reliability of the system.
Given a physical problem, it is often difficult to ascertain whether the stationary property
holds since ,⋯, ; ,⋯, ,⋯, ; ,⋯, must hold
for all . For practical purposes, we are often interested in a wider class of stationary
SP, referred to as weakly stationary.

_________________________
HISTORICAL NOTE:
The Mean-Square Spectral Density Function was defined by WIENER (1930) and independently by
KHINTCHINE (1934). Wiener attributed the fundamental idea underlying spectral density to
SCHUSTER (1899; 1900; 1905). He wrote: ‘‘The germs of the generalized harmonic analysis of this paper are already
in the work of Schuster, but only the germs. To make the Schuster theory assume a form suitable for extension and
generalization, a radical recasting is necessary.’’
However, some of the ideas underlying spectral density preceded Schuster’s 1899 paper. RAYLEIGH
(1889) introduced the idea that a random process has an autocorrelation. RAYLEIGH (1903; 1912;
1919a,b) wrote many other papers that considered the spectral representation and probability distributions
of random signals.
For an extensive discussion on the subject see PAEZ (2006).
WIENER, N. (1930). Generalized harmonic analysis, Acta Mathematica 55 (118)
KHINTCHINE, A.Y. (1934). Korrelationstheorie der stationären stochastischen Prozesse, Math.

Ann., 109, 604-615.
SCHUSTER, A. (1899). The periodogram of magnetic declination, Cambridge Philosophical

Transactions 18, 108
SCHUSTER, A. (1900). The Periodogram of magnetic declination, Transactions of the Cambridge

Philosophical Society, 107–135
SCHUSTER, A. (1905). The periodogram and its optical analogy, Proceedings of the Royal
Society 77, 136–140
L. RAYLEIGH (1889). On the character of the complete radiation at a given temperature,

Philosophical Magazine 27, 460–469
L. RAYLEIGH (1903). On the spectrum of an irregular disturbance, Philosophical Magazine 5,

238–243
L. RAYLEIGH (1912). Remarks concerning Fourier’s theorem as applied to physical problems,

Philosophical Magazine 24, 864–869.
L. RAYLEIGH (1919a). On the problem of random vibrations, and of random flights in one, two,
or three dimensions, Philosophical Magazine 37, 321–347.
L. RAYLEIGH (1919b). On the resultant of a number of unit vibrations, whose phases are at
random over a range not limited to an integral number of periods, Philosophical Magazine 37,
498-515
PAEZ, Th., L. (2006). The history of random vibrations through 1958, Mechanical Systems and
Signal Processing, 20, 1783–1818
__________________________

DEFINITION: WEAKLY STATIONARY PROCESS
A SP , ∈ , is weakly stationary (also referred to as second-order stationary,

wide-sense stationary, or covariance stationary) if
| | ∞
and
∞ ,
 It should be clear that a strictly stationary SP whose moments are finite is also
weakly stationary, but the converse is not true in general. An important exception
is the Gaussian SP. As we shall see, a Gaussian SP is completely specified by its mean
and covariance functions. Hence a weakly stationary Gaussian SP is also strictly stationary.

MEANS AND CORRELATION FUNCTIONS OF MEAN SQUARE DERIVATIVES
The m.s. derivative of , ∈ and higher order ones, if they exist, are stochastic
processes. It can be demonstrated that their correlation functions and cross-correlation
functions are determined simply in terms of the correlation function of .
For easy reference, the Table that follows gives a summary of the most important m.s.
differentiation properties.
Non-stationary Processes Weakly Stationary Processes
, ,
. .
0, 1
1 ,
Special Case: 1, 0 , and ,

0
When is weakly stationary,

and are uncorrelated RVs. If, in
addition, is Gaussian SP, and
are independent RVs.
NOTE:
Φ 1 Φ ω

Following are several examples of stationary and weakly stationary processes.
EXAMPLE
Consider a SP , ∈ , defined by
cos Θ
where and are real constants and Θ is a RV uniformly distributed over the
interval 0,2 . We easily obtain
cos Θ
cos θ
cos θ
2
1
, cos Θ cos Θ cos
2
 Therefore, the above defined SP , ∈ , is weakly stationary (notice

the dependence of , on the difference ; | | 0
∞ ; and ∞ ).
 It is evident that the realizations (sample functions) of the SP are all cosine
curves with frequency and amplitude . They differ from each other only
through their possible phase differences.
EXAMPLE
Consider a SP , ∈ , defined by
in which is a complex-valued RV, and denotes a deterministic complex-valued

function of . We wish to determine the properties of and such that is
weakly stationary.
It is easily seen that

∗ ∗ ∗
where the superscript “*” denotes complex conjugate. Notice that the second equation
is a generalization of the definition of correlation function for the case of complex-
valued stochastic processes.
In order that has a constant mean and variance and its correlation function
depends only on , we see from the above two equations that the following conditions
must be satisfied:
(a) 0
∗ | |
(b) , giving , √ 1 , is a real
constant and a real function of , and
∗
(c) 0 , or , and being constants.
Thus, is a weakly-stationary SP if and only if it has the form
with 0.
The correlation function of is then
| |
The real part of this weakly stationary SP is
cos Θ
This is a slightly more general process than the SP cos Θ discussed

previously.
EXAMPLE
Consider a linear combination of two weakly stationary processes, i.e.
in which and are zero mean, complex-valued RVs.

The mean and correlation function of are:
0
∗ ∗ ∗
∗ ∗
∗ ∗
We see that in this case, the SP is weakly stationary if 0.
The corresponding correlation function is
| | | |
EXAMPLE
For the weakly stationary process defined above, suppose that
∗
1
2
Where and are real-valued RVs, and suppose .
Then clearly, based on the above assumptions
cos sin
∗
We note that, in view of the conditions 0 and
∗
0, the RVs and have zero means, are uncorrelated, and have the same
variance, say .
To summarize, the SP cos sin is a zero-mean, real-valued,

weakly stationary SP with correlation function
cos
Note that an alternate form of the SP is the following:
| | | |
| |
2| | cos Θ
cos Θ

where | | √ and Θ tan .
It should be evident that the SP is a monochromatic function with random amplitude and
random phase.
EXAMPLE
Now let us generalize the analysis presented in connection with the above examples and
consider the following SP
in which are zero-mean, complex-valued RVs. Following the same approach as used
in the previous example, we can show that is a zero mean SP and that it is weakly
stationary if
∗
0 ,
In this case, the correlation function of is
| |
∗
The SP takes on real values if 2 with and for
∗
1,2, ⋯ , . As in the previous example, we may write ,
where and are real-valued, uncorrelated RVs with zero means and
. Then, can be written as
cos sin

where ∙ denotes the real part of the expression contained in the square brackets.
The correlation function of is
| |
| | | |
| | | | | | | |
| | | |
cos
As in the previous exercise, may be written as
| | cos Θ
in which Θ tan .
 The above expression of represents a superposition of harmonics with

random amplitudes | | and random phase Θ .
NOTE: The weakly stationary process ∑ defined above with auto-

correlation function ∑ cos (where 2 ) has a Power
Spectral Density Φ given by the following expression:
Φ
2

__________________________
NOTE:
Recall that based on the following (‘standard’) definition of Fourier Transform:
1
⟷
2
we obtain the following results:
cos ↔
sin ↔
However, the (‘non-standard’) definition of Fourier Transform that we conventionally

adopt in Random Vibration is the following:
1
⟷
2
Based on the above (‘non-standard’) definition, the Fourier Transform of the harmonic
functions would be the following:
cos ↔
sin ↔
i.e., the ‘standard’ Fourier Transforms can be converted to ‘non-standard’ Fourier

Transforms by multiplication with the factor .
__________________________

SPECTRAL REPRESENTATION OF STOCHASTIC PROCESSES
Harmonic analysis provides a very powerful tool for the analytical and experimental
treatment of a large class of physical problems. To carry out the harmonic analysis, it is
necessary to determine the spectral properties of the functions involved.
o If the functions are periodic, their spectral properties may be determined through
Fourier series representation in terms of a discrete set of frequencies that are
integral multiples of the fundamental frequency.
o A non-periodic, if absolutely integrable, admits a Fourier integral
representation in terms of a continuous band of frequencies.
In discussing the spectral representation of SP there are two approaches:
(1) ‘Intuitive’ approach: In this approach we start by considering the Fourier

Transform (FT) of each sample function of a SP, assuming that such a Fourier
representation exists. We soon realize that such a representation does not exist
for weakly stationary processes. We deal with such processes by considering the
truncated FT of each sample, taking the square amplitude of the FT, normalizing
by the duration and taking the limit as →∞.
(2) Fourier-Stieltjes Integral approach: In this approach we make use of the

Fourier-Stieltjes Integral to represent any stochastic process, stationary and
non-stationary. This approach, although mathematically more involved, is far
more general and it can deal with both stationary and non-stationary processes.
NOTE: A very readable introduction to ‘Stieltjes Integral’, requiring only a

background on basic Calculus, may be found in
WIDDER, D.V. (1989). Advanced Calculus, 2nd Edition, DOVER PUBLICATIONS, Inc., NY
A mathematically more demanding exposition of ‘Stieltjes Integral’ may be found in
APOSTOL, T.M. (1974). MATHEMATICAL ANALYSIS (2nd Edition), ADDISON-WESLEY

Publishing Company, Reading Massachussetts
See also:
CARTER, M. and B. van BRUNT (2000). The Lebesgue-Stieltjes Integral: A Practical

Introduction, SPRINGER

The symbol of such an integral is , and the case of Riemann integral (that
we are familiar from elementary Calculus) occurs as the special case in which .
When has a continuous derivative, the definition is such that the Stieltjes integral
becomes the Riemann integral . However, the Stieltjes
integral still makes sense when is not differentiable or even when is
discontinuous. In fact, it is in dealing with discontinuous that the
importance of the Stieltjes integral becomes apparent. By a suitable choice of a
discontinuous , any finite or infinite sum can be expressed as a Stieltjes integral, and
summation and ordinary Riemann integration then become special cases of this more
general process. Problems in Physics which involve mass distributions that are partly
discrete and partly continuous can also be treated by using Stieltjes integrals. In the
mathematical theory of Probability the Stieltjes integral is a very useful tool that
makes possible the simultaneous treatment of continuous and discrete RVs.

(1) ‘Intuitive’ approach

Let us consider the SP , and the transformed random process defined as
follows:
1
2
The above integral exists in the mean square if and only if
∗
1
,
2
is bounded for all values of and .
NOTE: This condition is not met by the weakly stationary SPs as will be shown
later. One can understand this intuitively by recalling that a SP may be visualized as an
ensemble of realizations, say . For the Fourier transform of any such realization
to exist, the realization must be absolutely integrable, i.e. | | ∞ . For a weakly
stationary process every sample function extends from the infinite past to the infinite
future. Since the distribution of values of the sample functions over the ensemble of
sample functions is the same at any time , we can state with confidence of probability 1
that every sample function does not vanish at ∞ and ∞ on the time axis.
Therefore, almost all sample functions are not absolutely integrable, and their
transforms do not exist.
For the sake of the argument let us assume that is such that the transformed SP
exists. The inverse transformation may be expressed in the mean-square sense as
wherever is mean-square continuous. Intuitively, the above equation states that

may be replaced, up to a mean-square equivalence, by a sum of harmonic
components, and that is the random complex amplitude of the components
with frequency .
∗
Similarly, interpreting the above expression for as a 2-D Fourier
transform of the autocorrelation function , , we may express the inverse
transformation as follows [PAPOULIS, A. 1968. Systems and Transforms with
Applications in Optics, McGraw-Hill]:

∗
,
∗
The validity of the transform pair , ↔ requires the
absolute integrability, piece-wise continuous and bounded variation of
, over the domain and .
To demonstrate that does not exist when is weakly stationary, we first

consider the truncated Fourier transform
1
,
2
The existence of , in mean square requires that the following be bounded for
all and :
∗
1
, ,
2
It suffices to show that in the special case of the right-hand side of the
above expression diverges as approaches infinity.
Let and . Then, the above equation becomes
1
| , |
2
The domain of integration for the right-hand side of the above expression is shown in
the Figure (below). Reversing the order of integration,
1
| , |
2
1
2 | |
2
The last integral diverges as → ∞. Therefore does

not exist if is weakly stationary.

Now if the last expression derived above is multiplied by prior to taking the limit,
we obtain
1 | |
lim | , | lim 1
→ → 2 2
1 | |
Φ lim
→ 2 2
Where we have assumed that the auto-correlation function is absolutely integrable from
∞ to ∞ ; therefore the mean-square spectral density
Φ exists.
| |
It will be demonstrated that lim → 0
Then,
1
lim | , | Φ
→ 2
Thus, the spectral density function, as we have previously defined it, corresponds to the
ensemble average of the squared moduli of the Fourier transforms of the realizations
. This result suggests that we may estimate the spectral density function from a
“large number” of “long” sample realizations of roughly as follows:

 Obtain sample realizations of with length .

 For each realization, take its finite Fourier transform and compute its squared
modulus.
 Compute the average squared modulus at each frequency .
 Multiply by to get the proper normalization and units.
Note that taking longer records does not necessarily improve the estimate. The record
length needs to only be long enough for the limiting operation above to be valid. To
improve the estimate, one must increase the number of sample realizations that are
averaged.
o The existence of the Spectral Density function requires only that the auto-
correlation function be absolutely integrable, piecewise continuous
and bounded, conditions met by most weakly stationary random processes of
interest.
o Note that on the left-hand site of the equation lim → | , |
Φ the operations of expectation and taking the limit cannot be
interchanged. Furthermore, this limit is one of a sequence of nonnegative
numbers. Therefore, this equation gives an alternative proof of the non-
negativity of an arbitrary spectral density function of a weakly stationary
process.
______________________________
| |
NOTE: To prove that lim → 0 , we have to
demonstrate that for an arbitrarily small number 0 there exists a such that
| |
2
First we note that since is absolutely integrable, there exists a 0 such

that
| |
2
Secondly,
| | | |
| |
2
The integral on the right-hand side of the above inequality can be split into

1 | | 1
| || | | | | || |
2
1 | | 1
| || | | | | || | 2
2
The proof is complete if we choose , where
2
| || |
____________________________
NOTE: The concept of Spectral Density as described above does not apply in the case
of a non-stationary SP since the auto-correlation function is no longer a function of
. However, when exists in mean-square, it is
convenient to call the following expression Generalized Spectral Density of :
∗
Φ , ≝
∗
Then the Fourier transform pair , ↔ may be re-written as
, ↔Φ , . The corresponding equations are analogues to the
Wiener-Khintchine theorem in the stationary case. Note that , and
Φ , are deterministic functions. Thus, Φ , exists if , is
absolutely integrable, piecewise continuous and bounded. The Generalized Spectral
Density of retains some of the manipulative advantages of a spectrum.
_________________________
Let be a real weakly stationary SP with auto-correlation function and

Spectral Density Φ . Then, in view of the fact that is real and
symmetric we obtain
1
Φ
2
1
cos
Clearly, the Spectral Density Φ of a real SP is real, symmetric, and positive.
Therefore,

2Φ cos
Incidentally, in practice (e.g. experimental measurements) the one-sided Spectral

Density G 2Φ is commonly used as opposed to the two-sided Spectral
Density Φ .
_____________________
NOTE: In mechanical engineering applications the Spectral Density function is

commonly described in terms of octaves (oct) and decibels (dB).
 As in music, an octave (oct) is doubling of frequency. The increase in octaves

from to is defined as
log
1
log
log 2
 The decibel (dB) is the ratio of two measurements of power, given by
log 0.1 ⟺ 10 log 1
The bel is the ratio of two measurements of power, given by
log 1 ⟺ 10 10
The bel (and decibel) is named after ALEXANDER GRAHAM BELL (1847-
1922).
Based on the above definition, a doubling of Spectral Density corresponds to an

increase of approximately 3 , since 10 log 2 3.01 .
___________________________

(2) Fourier-Stieltjes Integral Approach

Let be the real-valued SP ∑ cos sin (where
and are real-valued, uncorrelated RVs with zero means and
) that we investigated in one of the examples above.
We demonstrated that the process is weakly stationary with auto-correlation

function ∑ cos and PSD function
Φ ∑ .
An alternative representation of , its auto-covariance function , and its

PSD function Φ is the following:
2 2
, Φ
2 2
where: , , 1,2, ⋯ , , the random coefficients are

∗
complex-valued with mean values 0 and correlations ,
0 , and 0.
The above representation shows that consists of a superposition of harmonics

with frequencies . The coefficients of these harmonics are
orthogonal, have mean value zero, and variance ⁄ .
This observation suggests the representation for an arbitrary

weakly stationary process , where the random complex-valued coefficients
associated with the harmonics of frequencies are such that 0 and
∗
Φ .
As it is demonstrated by the Spectral Representation Theorem, it turns out that our

intuition is correct.

SPECTRAL REPRESENTATION THEOREM – Complex-Valued SP’s: If

is a complex-valued, weakly stationary, and m.s. continuous SP with spectral
distribution and spectral density Φ , then there exists a complex-valued SP
∗
with stationary orthogonal increments [i.e., 0, ]
defined up to an additive constant such that the m.s. integral
exists at any time and, for ∞ 0 , we have
0 , | | , | | Φ
The mean and auto-correlation function of the SP are:
∗
0 , 1 2 Φ
The above representation is reduced to the following one for the case of a real-valued
SP:
SPECTRAL REPRESENTATION THEOREM – Real-Valued SP’s: If is a

real-valued, weakly stationary, and m.s. continuous SP with one-sided spectral
density , then there exist two real-valued SPs , with stationary orthogonal
increments [i.e.,
0, ] such that the m.s. integral
cos sin
exists at any time and we have
0 , 0
2 2
for , , 0
The mean and auto-correlation function of the SP are:
0 , cos
Clearly, in this case the auto-correlation function takes real values.

The properties of the SP’s , are similar to the properties of the coefficients
, of the real-valued SP ∑ cos sin discussed
above.
Since is m.s. continuous, its auto-correlation function is continuous so that the

BOCHNER Theorem can be applied.
Bochner’s (or Bochner - Khintchine) Theorem:
A continuous function ∶ → is nonnegative definite if and only if it has the

representation
where is a real-valued, increasing, and bounded function.
The proof of BOCHNER’s Theorem can be found in:

BOCHNER, S. (1933). ‘Monotone Funktionen Stieltjessche Integrale und harmonishe Analyse,’
Math. Ann. 108, 376-385.
KHINTCHINE, A.Y. (1934). Korrelationstheorie der stationären stochastischen Prozesse, Math.

Ann., 109, 604-615.
CRAMER, H. (1939). ‘On the representation of a function by certain Fourier integrals,’ Trans.
Am. Math. Soc. 46, 191-201.
CRAMER, H. and M.R. LEADBETTER (1967). Stationary and Related Stochastic Processes,
John Wiley & Sons, Inc., New York, [Section 7.5].
GNEDENKO, B.V. (1962). The Theory of Probability, translated by B.D. Seckler, Chelsea
Publishing Co., New York, [Section 39].
LOEVE, M. (1960). Probability Theory (2nd Edition), D. Van Nostrand Co., Inc., Princeton, NJ,
[page 207].
The reader may consult also:

YAGLOM, A.M. (1962). An Introduction to the Theory of Stationary Random Functions,
PRENTICE-HALL, INC., Englewood Cliffs, NJ
SVESHNIKOV, A. A. (1966). Applied Methods of the Theory of Random Functions, Pergamon

Press

Here we only show that ∙ given by the expression τ is in

fact nonnegative definite.
We recall that function is said to be nonnegative definite if the following relation

∑ ∑ ∗
0 is valid for any integer 1 , arguments ∈ ,
∗
and complex constants , 1,2, ⋯ , , where denotes the complex conjugate of
∈ .
Note that
∗ ∗
is nonnegative for any integer 1 , arguments , and complex constants .
Real-valued SP’s:
Based on the above results, one can make the following statements:
 If is a real-valued weakly stationary process, its correlation function τ

is a nonnegative definite function.
 The BOCHNER Theorem states that, if the correlation function ∶ →

is continuous, then there exists a real-valued, non-decreasing, and bounded
function , called the spectral distribution function of , such that has the
representation τ . The spectral distribution function is
defined up to a constant that can be eliminated by setting ∞ 0 . If is
absolutely continuous, there exists a function Φ ⁄ , ∈
, called the spectral density or the mean power spectral density function of
.
 The correlation function and the spectral density are Fourier Transform
pairs. That is:

1
Φ ⟷ Φ
2
NOTE: If 0 , then Γ , from which follows

that
1
Φ Φ where Φ Γ
2
In engineering applications the one-sided spectral density function
2Φ , 0
because one does not have to deal with negative frequencies.
By analogy with the spectral distribution of , we can define the one-sided

spectral distribution function of this process by
2 Φ 2 Φ , 0
where the notation indicates that the integral does not include the
value of the integrand at the origin (i.e. at 0 ).
We summarize now some properties of the spectral and the one-sided spectral density
functions of real-valued SP’s:
 Φ and are positive functions and Φ Φ for all ∈ .

[Clearly this is the case because is an increasing function.]
 An alternative to
Φ ⟷ Φ
is
cos ⟷ cos

 The area under the spectral densities Φ and is 0 . If

0 , this area is the variance of .
_________________________
NOTE: Consider the zero-mean, stationary SP . Then, the

auto-covariance function is expressed as follows:
∗
, 1 1
1
2 2
2
2 2 1 1 ∗
1 2
In view of the fact that the process is stationary , , that is
2 1 1 1 ∗
1 2
Comparing the above expression of the auto-covariance function with the expression in
terms of the PSD function
it becomes evident that the two expressions are equivalent iff
∗
Φ 2
________________________
HISTORICAL NOTE: The basic proposition enabling us to pass to the spectral

representation of a stationary random function is the BOCHNER-KHINTCHINE
Theorem. The proof of τ and also of the fact that every function
τ which can be represented in the form: τ , (where is
non-decreasing and bounded), is the correlation function of some stationary SP, was derived
by KHINTCHINE (1934) as a consequence of two facts, taken together:
(a) The class of functions which are correlation functions of stationary SPs
coincides with the class of non-negative definite functions of the variable ;

(b) A function τ is non-negative definite if and only if it can be represented in

the form τ , (where is non-decreasing and
bounded).
Shortly before the appearance of KHINTCHINE’s (1934) paper, the proposition (b),
above, had been published by BOCHNER (1933), but it was discovered independently
by KHINTCHINE at about the same time. Hence, this result is referred to as the
BOCHNER Theorem or the BOCHNER-KHINTCHINE Theorem.
The possibility of representing an arbitrary stationary SP as

was first shown by KOLMOGOROV in 1940 in the context of Hilbert space geometry.
Later, this representation was justified and interpreted in a probability theory context
(e.g. BARTLETT, 1955; DOOB, 1953; LOEVE, 1963).
It is interesting to point out that the spectral representation, , of a

stationary process was proved after the appearance of KHINTCHINE’s result, τ
, and in fact its proof was based on KHINTCHINE’s result. A more
direct proof of the spectral representation of , (i.e., without making use of the
BOCHNER-KHINTCHINE Theorem) was proposed by LOEVE (1945).[LOEVE’s
direct proof is presented also by BARTLETT (1955), YAGLOM (1962) and PAPOULIS
(1965).]
BARTLETT, M.S. (1955). Stochastic Processes, Cambridge University Press, London.
DOOB, J.L. (1953). Stochastic Processes, Wiley, New York.
LOEVE, M. (1945). “Sur les fonctions aléatoires stationnaires de second ordre,” Rev. sci.,
Paris, 83, 297.
YAGLOM, A.M. (1962). An Introduction to the Theory of Stationary Random Functions,
Revised English Edition Translated and Edited by Richard A. Silverman, PRENTICE-
HALL, INC., Englewood Cliffs, NJ
LOÉVE, M. (1963). Probability Theory, Van Nostrand-Reinhold, Princeton, NJ
See also:
PAPOULIS, A. (1965). Probability, Random Variables, and Stochastic Processes,
McGraw-Hill, New York.

Evolutionary Spectral Density for Non-stationary Processes

The PSD function gives a frequency domain description of a stationary SP, and is a
powerful tool for RV analysis. It is desirable to look for a similar function that can
describe frequency-domain properties of a non-stationary SP.
We use the Fourier-Stieltjes integral representation, in connection with the concept

of ‘Evolutionary (i.e. time dependent) Power Spectral Density’ function introduced
by PRIESTLEY (1965, 1967), to describe the evolving (with time) distribution of power
over frequency. Such a representation is used to describe a special class of non-stationary
processes referred to as ‘oscillatory processes’. It can be demonstrated that an
‘oscillatory process’ admits the following representation:
where is again (as in the representation of stationary SP’s) a SP with stationary

∗
orthogonal increments [i.e., 0, ] , and ,
is a deterministic ‘oscillatory’ function. If , varies slowly with , then
, physical significance as an amplitude-modulated harmonic function.
Clearly, if , 1 then the process becomes identical to the stationary
process .
PRIESTLEY, M. B. (1965). “Evolutionary spectra and non-stationary processes,” J. R. Stat. Soc.,

Ser. B (Methodology), Series B, 27, 204–237.
PRIESTLEY, M. B. (1967). “Power spectral analysis of non-stationary random processes,” J.

Sound Vib. , 6, 86–97.
Let us consider the non-stationary (oscillatory) process , .
[As always, we assume that the mean value has been subtracted from the process and the
resulting process has mean zero; as a result the auto-covariance function , and
the auto correlation function , become identical.]
Then,

∗ ∗
, , 1 1
1 , 2 2
2
∗ ∗
, , 2 2 1 1
1 2
1 2 1 2
∗
, , 2 1 Φ
where Φ is the PSD function of the associated stationary SP .
The mean-square of the process is easily computed by setting in the above

expression of the auto-covariance function ,
| , | Φ
,
where: Φ , | , | Φ is the evolutionary PSD function of the SP.
In many applications, , ≝ , i.e. the deterministic modulating function is

independent of frequency . Such a process is referred to as ‘uniformly modulated
process’ and results from a stationary SP as follows: .

_____________________
NOTE: GENERALIZED HARMONIC ANALYSIS
The following notes are based on the following references:
PRIESTLEY, M. B. (1967). “Power spectral analysis of non-stationary random

processes,” J. Sound Vib., 6, 86–97.
PRIESTLEY, M. B. (1981). Spectral Analysis and Time Series, Volumes 1 &2, Academic
Press, Harcourt Brace Jovanovich, Publishers
SVESHNIKOV, A. A. (1966). Applied Methods of the Theory of Random Functions,

Pergamon Press
Broadly speaking, the general subject of power spectral analysis may be described as a
method of representing time-varying quantities (such as radio signals, vibration records,
seismograph records, etc.) as the combination of a number of sine and cosine waves with
different amplitudes, frequencies, and phases. Looked at from this point of view, power
spectral analysis may be thought of simply as a generalization of Fourier analysis.
Consider a deterministic function :
(1) If the time-varying quantity is a periodic function it can be represented by a

Fourier series:
where the coefficients are constants which are uniquely determined by

the form of .
(2) If is non-periodic, but (roughly speaking) decays to zero as → ∞

(mathematically speaking is absolutely integrable, i.e. | |
∞), we may represent it as:
where is a function of the frequency which is also uniquely

determined by the form of .
For each frequency, the corresponding squared amplitude is proportional to the

“power” (or “energy per unit time”) contributed by that component, and a graph of
squared amplitudes plotted against frequency is called a “power spectrum”.

Accordingly,
in the case (1) we say that has a “discrete” power spectrum (or “line
spectrum”),
whereas
in the case (2) we say that has a “continuous” power spectrum, i.e. that
the total power of is distributed over a continuous range of
frequencies.
[REMARK: It should be remarked that when is non-periodic each amplitude is

, i.e. is infinitesimal, and the energy distribution can strictly be described only in
terms of a “power-density” distribution over frequency.]
However, there are many cases where is neither strictly periodic nor decays to
zero, but has (in a sense) a “steady-state” form, i.e. does not become arbitrarily large
or small as → ∞ . Remarkably, it is still possible to obtain a “Fourier-type”
representation for such functions, the pioneering work in this direction being due to
WIENER (1930, 1933).
WIENER, N. (1930). Generalized Harmonic Analysis, Acta Math. Stockh., 55:117-258.
WIENER, N. (1933). The Fourier Integral and Certain of Its Applications, Cambridge
University Press
WIENER developed an ingenious theory called “Generalized Harmonic Analysis”,

in which he defined a more general type of Fourier expansion which includes both
Fourier series and Fourier integrals as special cases. That is, he considered those
functions which could be represented as a “generalized” Fourier integral of the form
where the function is uniquely determined by the form of , but need not
be necessarily differentiable.
 If is periodic, turns out to be a step function so that equation

reduces to a Fourier Series, i.e. to equation
∑ .

 If decays to zero as → ∞ , then will be differentiable (for all ),

so that equation reduces to a regular Fourier
Integral, i.e. to equation with / .
The essentially new feature of generalized harmonic analysis is that functions of the
“steady-state” type can also be represented in the form , but
now the function is no longer differentiable. In fact, equation
can be used to represent a large class of functions of this type,
provided
so that the order of magnitude of the increment in over an interval is much

larger than . Clearly, in this case is not differentiable, (i.e. its derivative will
not be finite at any point), but our main interest is in the energy distribution of ,
i.e. we are interested in the quantity | | [rather than itself], and according
to equation √ ,| | is of order , so that, in general, the function
| | ⁄ , say , will be finite for all values of . The function ,
represents the density of energy over frequency, and is called the “power spectral
density function”. If consists of a periodic function say, superimposed
on a “steady-state” function, say [i.e. + ] then | |
will be finite at the natural frequencies of and, correspondingly, will have a
-function form at these points.
These ideas are extended to Random Functions. The basic result underlying the
spectral analysis of stationary processes is the fact that each realization can be
represented in the same form as , where, of course, the
function will change from realization to realization. In other words, for each ,
is itself a random variable, and (regarded as a function of ) is now also a
random process. Furthermore, it may also be shown that, if is stationary,
is an orthogonal process, in the sense that the increments ,
(at any two distinct points , ) are uncorrelated random variables. Conversely, if
a process has a representation of the form with
orthogonal, it may be shown that is necessarily stationary.

Finally, the random function , being a function with independent increments,

cannot possess a derivative and, accordingly, the retention of the Stieltjes integral is
unavoidable.
In order to visualize how a random function, say , which has no derivatives

(anywhere) in spite of its continuity arises, consider the following generation
mechanism: The ordinate of the random function is obtained as the result of the
summation of an infinitely great number of infinitely small mutually independent
increments. Over a time increment ∆ the random function receives independent
random increments , possessing zero mathematical expectations (i.e.
0) and equal dispersions . We assume that the instants of the increments of the
function are uniformly distributed in time, with a rate (i.e. increments arriving per
unit time). We assume that the number is so great that even for an extremely small time
interval ∆ the number of increments ⋅ ∆ can be regarded as an integer. Then the
increment of the function after a time ∆ can be represented as the sum
By increasing the number and simultaneously decreasing the variance of each term
in the above expression it is possible to consider the function as continuous in the
limit, however its increment will not be proportional to the increment of the argument
∆ , as would be the case for ordinary (non-random) continuous functions.
In fact, finding the variance of both sides of the above equality and taking into account
the independence of the ’s , we obtain
|∆ | ⋅Δ
that is, the first power of Δ is proportional (in the mean) to the square of the
increment of the function, and not to the increment itself, as would have been the
case had the function possessed a derivative.
The above example explains the mathematical mechanism of the occurrence of non-
differentiable random functions.

4 - Theory of SPs

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

4 - Theory of SPs

Uploaded by

Copyright:

Available Formats

Lecture Notes: RANDOM VIBRATION OF STRUCTURES / SPRING 2009 / Page: 1

Lecturer: Prof. APOSTOLOS S. PAPAGEORGIOU

RANDOM (STOCHASTIC) PROCESSES

A stochastic process, or a random process, is a mathematical model of a dynamical

The theory of Stochastic Processes (SP) has evolved as a generalization of the

FIGURE: Ensemble of a Stochastic Process

The Figure shows the realizations of a SP where is the parameter. Thus a SP is a

Last update: MARCH 2011

 On the other hand, for a fixed , ∙, defines a function of the

For expediency, the dependence of a SP on is not explicitly shown.

Thus, a SP , can be regarded either:

Since , ∈ , at a fixed can be either a discrete or a continuous RV, a SP

We will be primarily concerned with continuous-parameter, continuous-valued SP’s that

Last update: MARCH 2011

CLASSIFICATION OF STOCHASTIC PROCESSES

Last update: MARCH 2011

CHARACTERIZATION OF STOCHASTIC PROCESSES

Therefore, a SP is specified as follows:

Definition/Specification of a Stochastic Process

If for every finite set , ,⋯, of ∈ , there corresponds a set of RVs,

,⋯, ; ,⋯, ∩⋯∩

a) The condition of consistency, i.e., for any ,

,⋯, , ∞, ⋯ , ∞ ; ,⋯, ,⋯, ,⋯, ; ,⋯,

b) The symmetry property in that it is invariant under an arbitrary permutation of

,⋯, ; ,⋯, ,⋯, ; ,⋯,

where , ,⋯, is an arbitrary permutation of 1, ⋯ , .

The corresponding nth density function of , is given by

Last update: MARCH 2011

MOMENTS OF STOCHASTIC PROCESSES

Some of the most important properties of a SP are characterized by its moments,

The nth moment, , of a SP at a given ∈ , is defined by

 The first moment, , is the mean of the SP at ∈ , and is sometimes

The nth central moment of a SP at a given ∈ , is defined by

 The second central moment, , is the variance of the SP at

An important measure of linear independence between and is contained in

It is a function of and in general.

The corresponding central moment, referred to as the auto-covariance function

Last update: MARCH 2011

It can easily be verified that

The normalized auto-covariance function is called correlation-coefficient function and

When , 0 for all , the SP , ∈ is called an

Similar definitions may be provided for the cross-correlation and cross-covariance

Last update: MARCH 2011

Some Important Properties of Second-Moment Characteristics

(b) For every , ∈ , the following relations follow as consequences of the

(c) The function , [ , ] is non-negative definite on ;

The above inequality is demonstrated as follows: Consider a RV defined by

Last update: MARCH 2011

(d) Consider two RPs, and . It is clear that

where is a SP with mean and covariance function , ,

Last update: MARCH 2011

In particular if , then , which has zero

o Thus when we investigate the covariance function of a SP, there is no

o In view of the abovementioned property [regarding the equality of the

NOTE: CAUCHY-SCHWARZ INEQUALITY

in which and are real RVs with finite second moments.

Proof: Consider the function

in which is real. Now 0 no matter what may be the value of .

Such a function is said to be a nonnegative quadratic form in . A plot of versus

The minimum value of is attained for ⁄ .

 If 0 (i.e. no intersection of with the -axis) then

Last update: MARCH 2011