IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 16, NO.
5, SEPTEMBER/OCTOBER 2010
Mitigation of Fiber Nonlinearity Using
a Digital Coherent Receiver
David S. Millar, Student Member, IEEE, Sergejs Makovejs, Student Member, IEEE,
Carsten Behrens, Student Member, IEEE, Stephan Hellerbrand, Student Member, IEEE,
Robert I. Killey, Member, IEEE, Polina Bayvel, Fellow, IEEE, and Seb J. Savory, Member, IEEE
Abstract—Coherent detection with receiver-based DSP has re-
cently enabled the mitigation of fiber nonlinear effects. We inves-
tigate the performance benefits available from the backpropaga-
tion algorithm for polarization division multiplexed quadrature
amplitude phase-shift keying (PDM-QPSK) and 16-state quadra-
ture amplitude modulation (PDM-QAM16). The performance of
the receiver using a digital backpropagation algorithm with vary-
ing nonlinear step size is characterized to determine an upper
bound on the suppression of intrachannel nonlinearities in a single-
channel system. The results show that for the system under in-
vestigation PDM-QPSK and PDM-QAM16 have maximum step
sizes for optimal performance of 160 and 80 km, respectively.
Whilst the optimal launch power is increased by 2 and 2.5 dB for
PDM-QPSK and PDM-QAM16, respectively, the Q-factor is corre-
spondingly increased by 1.6 and 1 dB, highlighting the importance
of studying nonlinear compensation for higher level modulation
formats.
Index Terms—Coherent detection, digital backpropagation,
nonlinearity compensation, quadrature amplitude modulation
(QAM).
I. INTRODUCTION
XPONENTIAL growth in capacity requirements in re-
E cent years has led to rapid improvements in the spectral
efficiency of optical communications systems [1]. While this
growth was previously sustained by the introduction of wave-
length division multiplexing (WDM), with on-off-keyed sys-
tems, this approach yields a theoretical maximum of 1 bits/s/Hz
over the bandwidth of the optical channel. By utilizing coher-
ent detection with phase and polarization diversity, it becomes
possible to detect the full 4-D signal space of amplitude and
phase on two orthogonal polarizations rather than the single di-
mension of total power used with direct detection (DD). As all
four dimensions of the optical field are detected by a phase and
Manuscript received December 21, 2009; revised February 9, 2010; accepted
March 4, 2010. Date of publication May 20, 2010; date of current version
October 6, 2010. This work was supported by the Engineering and Physical
Sciences Research Council, “Building the Future Optical Network in Europe,”
a Network of Excellence funded by the European Commission through the
7th Information and Communications Technology-Framework Programme,
Yokogawa Electric Corporation, Oclaro, and The Royal Society.
D. S. Millar, S. Makovejs, C. Behrens, R. I. Killey, P. Bayvel, and S. J. Savory
are with the Department of Electronic and Electrical Engineering, Optical Net-
works Group, University College London, London, WC1E 7JE, U.K. (e-mail:
d.millar@ee.ucl.ac.uk; s.makovey@ee.ucl.ac.uk; c.behrens@ee.ucl.ac.uk;
r.killey@ee.ucl.ac.uk; p.bayvel@ee.ucl.ac.uk; s.savory@ee.ucl.ac.uk).
S. Hellerbrand is with the Institute for Communications Engineering,
Technische Universität München, München, D-80290, Germany (e-mail:
stephan.hellerbrand@mytum.de).
Digital Object Identifier 10.1109/JSTQE.2010.2047247
1077-260X/$26.00 © 2010 IEEE
1217
polarization diverse coherent receiver, all four of these can be
used for modulation and improving the achievable spectral effi-
ciency. The detection of all four dimensions of the optical field
also enables the equalization of previously limiting linear trans-
mission impairments, such as group velocity dispersion (GVD)
and polarization mode dispersion (PMD). With the elimination
of linear transmission impairments, attention has turned to the
mitigation of nonlinear impairments, which digital coherent re-
ceivers cannot completely compensate stimulating research into
nonlinearity mitigating receiver subsystems.
Recently, theoretical research has been undertaken to assess
the ultimate nonlinear capacity of optical fibers [2], where both
high-level modulation formats and intrachannel nonlinearity
compensating DSP has been assumed. High-level modulation
formats have been a topic of much research, resulting in spec-
tral efficiency for PDM-QAM16 in excess of 7 bits/s/Hz [3]. As
higher level modulation formats have relatively lower optical
signal to noise ratio (OSNR) tolerance; therefore, higher launch
powers are required, resulting in greater nonlinear penalties.
Although research has focused on the study into both the tech-
niques for high-order modulation and nonlinearity compensat-
ing DSP, little work has been done into the intersection of these
two areas: the comparative benefits achievable with nonlinearity
compensation when the order of modulation is increased.
While the mitigation of interchannel fiber nonlinearities re-
mains an active research topic [4], this paper describes the study
of a digital coherent receiver and associated algorithms for the
intrachannel nonlinearity compensation. To maximize the ef-
ficacy of the nonlinear DSP, a phase and polarization diverse
digital coherent receiver is employed, allowing the full opti-
cal field within the receiver bandwidth to be reconstructed in
the digital domain. This allows us to exploit our knowledge of
the physical nature of the optical channel, and design our DSP
accordingly.
This investigation is performed by a series of single-channel
experiments and simulations to determine both the possible ben-
efits and the necessary spatial resolution when using the digital
backpropagation algorithm to mitigate intrachannel fiber non-
linearity. We compare the effects of nonlinearity compensa-
tion on two widely investigated high-level modulation formats:
polarization division multiplexed quaternary phase-shift key-
ing (PDM-QPSK), which yields 4 bits/symbol; and PDM 16-
state quadrature amplitude modulation (PDM-QAM16), which
yields 8 bits/symbol. Both modulation formats are investigated
at 10.7 GBd, such that the benefits of nonlinearity compensation
may be compared with a doubling of modulation density, while
the mitigating effects of dispersion and the signal bandwidth
remain the same.
1218 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 16, NO. 5, SEPTEMBER/OCTOBER 2010

Section II gives an introduction to digital backpropagation, ∂ α jβ2 ∂ 2

the use of the Manakov equation in forming split-step solutions E Y = − EY + EY
∂z 2 2 ∂t2
and forms algorithms for different step sizes and two possi-

2 jγ ∗ 2
ble nonlinear cascade models for backpropagation. A detailed − jγ |EY | + |EX | EY −
2 2
E E (1)
description of other DSP algorithms used in the receiver is 3 3 Y X
given in Section III. This is followed by a description of the ∂ α jβ2 ∂ 2 8
experimental setup used in Section IV, and simulations per- E X = − EX + EX − jγ (|EX |2 + |EY |2 )EX
∂z 2 2 ∂t2 9
formed in Section V, both of which we used to emulate non-
linear distortion occurring in transmission. Results are given in ∂ α jβ2 ∂ 2 8
E Y = − EY + EY − jγ (|EY |2 + |EX |2 )EY .
Section VI with accompanying discussion, and conclusions in ∂z 2 2 ∂t2 9
Section VII. (2)
Equation (2) may be split into its constituent linear and non-
II. DIGITAL BACKPROPAGATION
linear parts, resulting in
A. Nonlinear Channel Models
∂E
Digital backpropagation is a method of nonlinearity compen- = (D̂ + N̂ )E, where
∂z
sation, which has generated much interest recently [5]–[8]. It
exploits the knowledge of the physical behavior of the opti- jβ2 ∂ 2
D̂ = ,
cal fiber as a nonlinear channel, by approximating the inverse 2 ∂t2
nonlinear channel, most commonly described by the nonlin- 8 α
N̂ = −jγ EH E −
ear Schrödinger equation (NLSE). The solution of the NLSE 9 2
is approximated by the split-step Fourier method (SSFM),
E = [ EX EY ] T (3)
commonly used to simulate nonlinear transmission in optical
fibers. While this algorithm may be used to approximate an From (3), we may derive an exact solution of the Manakov
inverse channel, the channel is inherently limited by additive equation, which forms the basis of the split-step type solutions:
Gaussian white noise (AGWN), which stems from amplified
spontaneous emission (ASE) due to inline optical amplification. E(z + h, T ) = exp(h(D̂ + N̂ ))E(z, T ). (4)
This noise cannot be removed by any filter due to its random
nature, and will additionally result in nonlinear phase noise (due B. Split-Step Methods
to the Gordon–Mollenauer effect). Receiver-based digital chan-
nel inversion may be performed with many channels within the The following approximation is central to all SSFM numerical
receiver bandwidth [5], thus also mitigating interchannel non- solutions to both the coupled polarization nonlinear NLSE (1)
linear effects. It should be noted, however, that the required and the Manakov equation (2), which we will investigate here.
nonlinear step size in previous experimental work [8] was much We may say that for sufficiently small step h
smaller in this case and the discussion of large step sizes in this exp(h(D̂ + N̂ ))E(z, T ) ≈ exp(hD̂) exp(hN̂ )E(z, T ). (5)
paper may not be applicable.
Rather than the coupled polarization NLSE (1) [9], in this pa- While the approximation in (5) is very basic, this approxima-
per, we have used the Manakov equation (2) [10] for the inverse tion of the exact solution in (4) forms the basis of all split-step
channel model. The Manakov equation is more applicable for type solutions. A common refinement to (5) is to evaluate the
long distance simulations, where total transmission distance is nonlinear part of the solution with a constant envelope pro-
greater than 1000 km [11]. This model accounts for the residual file and varying intensity. This modification allows larger step
birefringence of the fiber, and the effect that this has on the sizes to be used as the solution does not imply constant power
state of polarization and nonlinearity within the fiber. Since the throughout the step as (5) does.
residual birefringence scatters the signal state of polarization on By assuming that the only change in the electric field over
a significantly smaller scale than the fiber nonlinear length, fiber the nonlinear step is loss (and consequently that dispersion may
nonlinearity acts on both polarizations equally as described by be considered to act separately), we may normalize the solution
the rightmost term in (2). Here, E = [EX , EY ] is the optical of the nonlinear part of the Manakov equation to the varying
field in delayed time, subscripts X and Y denote orthogonal power profile within the step and remove the loss term
linear polarization states, α is the fiber loss parameter, β2 is the  z +h
GVD parameter, and γ is the nonlinearity parameter. Though   
N̂ (E , z ) = N̂ (E , z  )dz 
the Manakov equation is widely used for simulation of fibers z
with residual birefringence, application of this equation for dig- 1 − exp (−αh)
ital backpropagation algorithms has only recently been made = N̂ (E, z  )
α
in [12] and formalized in [13]
= LEff N̂ (E, z  ), where
∂ α jβ2 ∂ 2
E X = − EX + EX 
E (z + h, T ) = exp(−αh/2)E(z, T ). (6)
∂z 2 2 ∂t2


2 jγ ∗ 2 The approximation in (6) gives us a nonlinear step, which
− jγ |EX | + |EY | EX −
2 2
E E
3 3 X Y includes loss and the total nonlinear phase shift over the spatial
MILLAR et al.: MITIGATION OF FIBER NONLINEARITY USING A DIGITAL COHERENT RECEIVER 1219

the order of the blocks is reversed, and the bulk-step model,

therefore, refers to a cascade of Wiener systems.

D. Channel Models and Accuracy Considerations

Fig. 1. Wiener model.
The improvement in accuracy that the symmetric split-step
method offers over the “bulk” step method given in (5) may
be quantified by examining the error term generated by the
approximations in (5) and (7), by use of the Baker–Campbell–
Hausdorff formula [15] for the commutability of operators (8).
This formula gives us an analytical insight into both the rela-
tive accuracy of the symmetric-step and bulk-step variants of
Fig. 2. Hammerstein model. the nonlinear channel model, and the effects of step size and
channel model on the accuracy of the digital backpropagation
algorithm

exp(hD̂) exp(hN̂ )


h2
= exp hD̂ + hN̂ + (D̂N̂ − N̂ D̂) + . . . . (8)
Fig. 3. Wiener–Hammerstein model. 2

In the Baker–Campbell–Hausdorff formula (8), for conve-

nience, we have included the dominant (first) term only of the
step. This is effectively a multiplication by the effective nonlin-
error series. This is equivalent to the error in the bulk-step ap-
ear length LEff , as defined in [9].
proximation given in (5). The error term resulting from the
The accuracy of this solution to the Manakov equation can
symmetric split-step method may be described as follows:
be improved by applying the dispersion operator in two equal
parts, before and after the nonlinear operator. This leads to the



symmetric SSFM [9] h h h
exp D̂ exp(hN̂ ) exp D̂ = exp D̂ exp(Θ)
2 2 2
exp(h(D̂ + N̂ ))E(z, T ) where


  
h h h2
≈ exp D̂ exp(LEff N̂ ) exp D̂ E(z, T ). (7) h
Θ = hN̂ + D̂ + N̂ ,
D̂
+ ...
2 2 2 2 2


C. Two- and Three-Block Nonlinear Models h
exp D̂ exp(Θ)
The two variations of the split-step method may be described 2
    
in terms of two- and three-block nonlinear models from nonlin- h3 D̂ D̂
ear systems theory [14], for each short length of fiber over which = exp hD̂ + hN̂ + N̂ + , N̂ , + ... . (9)
a split step is taken. Nonlinear models of particular interest are: 6 2 2
the Wiener model, which consists of a linear block followed by
a memoryless nonlinear block (see Fig. 1), the Hammerstein We note from (8) that the dominant error term is propor-
model, consisting of a memoryless nonlinear block followed by tional to h2 , while in (9), the dominant is proportional to h3 .
a linear block (see Fig. 2), and the Wiener–Hammerstein model, This indicates (a fact well known by those familiar with optical
which represents the concatenation of the Wiener and Hammer- fiber simulations using the SSFM) that the symmetric split-step
stein models, that is, a linear block followed by a memoryless method will give greater accuracy with an identical spatial step
nonlinear block, followed by a second linear block (see Fig. 3). size.
It is immediately apparent that the Wiener–Hammerstein An essential factor in the application of this solution, however,
model (see Fig. 3) is used to represent the behavior of a sec- is that the solution of the Manakov equation is to be performed
tion of fiber in the symmetric split-step method. These three at the receiver with a noisy signal and DSP. It is therefore in our
block systems are then cascaded to form an approximation of interest to perform not the most accurate reverse propagation,
the entire channel. Similarly, the bulk-step approach refers to a but the least complex with sufficient accuracy. Much recent
cascaded two-block (Hammerstein model) approximation of the research [12], [16], [17] has demonstrated that for receiver-
channel. The difference between the two- and three-block non- based digital backpropagation, a single nonlinear step per span
linear models stems largely from their relative accuracy, which with the bulk-step method may be considered a sufficiently
is analyzed in the Section II-D. It should be noted that while the accurate solution of the Manakov equation. In this paper, we
bulk-step approximation of the forward channel corresponds to will extend the ideas proposed in [5] and [6] by examining both
a cascade of Hammerstein systems, when this model is used for two- and three-block nonlinear models for step sizes ranging
compensation (that is, approximation of the inverse channel), from subspan to an entire link.
1220 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 16, NO. 5, SEPTEMBER/OCTOBER 2010
Fig. 4. Digital back end of phase and polarization diverse digital coherent
receiver.
E. Digital Backpropagation Algorithms
The coherent receiver-based nonlinearity compensation al-
gorithms used in this investigation may be categorized as two
variations of digital backpropagation. These are the Wiener and
Wiener–Hammerstein reverse-channel approximations (corre-
sponding to the bulk-step and symmetric-step methods), each
of which may be applied to step sizes of less than one span, or
one span or more.
Dispersive steps were performed in the frequency domain,
using the dispersion operator defined in (3). This method de-
scribes a circular convolution of the dispersion operator (3) and
the time-domain signal, and is given in (10), where F represents
the Fourier transform
exp(hD̂)E = F −1 {exp(hF{D̂})F{E}}.
Nonlinear operations were performed in the time domain with
the nonlinear operator normalized to both nonlinear step-size
and launch power. For nonlinear step sizes of a single span or
greater, the nonlinear operator is defined as follows:
N̂ (t, zNL ) = jϕzNL (|EX (t)|2 + |EY (t)|2 )PL .
Here, PL represents launch power in milliwatt, zNL is the
nonlinear step size, t represents the retarded time frame, and ϕ
is the nonlinear phase-shifting coefficient, which is a constant
to be optimized. All other symbols retain their conventional
meaning or those defined previously.
In the case of more than one nonlinear step per span, the
nonlinear operator is modified to account for the exponentially
varying power profile within the span. This leads to a nonlinear
operator, which is defined as follows:
N̂ (t, zNL ) = jϕ10(sL /10n ) zNL (|EX (t)|2 + |EY (t)|2 )PL
(12)
where n is the number of steps per span, s is the index of the
step within the span, and L is the fiber loss per span in decibels.
III. RECEIVER DSP STRUCTURE
The structure of the digital back end of our coherent receiver
is as described in Fig. 4.
Received digitized data is resampled to a rate of two samples
per symbol, and then, prepared for processing by deskewing
Fig. 5. Recirculating loop setup used for transmission experiments, with op-
tical front end of the phase and polarization diverse digital coherent receiver.
and normalizing the signals to have unit power per polarization.
The signal is then filtered with a stationary approximation of
the inverse of the optical channel. In most of the previous work,
this inverse channel model has compensated for dispersion only,
which may be modeled as a finite-impulse response (FIR) fil-
ter [18]. Here, we investigate the different inverse nonlinear
channel models described in Section II for this block. Adaptive
equalization is performed using the constant modulus algorithm
(CMA) equalizer for blind deconvolution of the signal and chan-
nel, and separation of the two orthogonal polarization states for
PDM-QPSK signals, and the radially directed CMA equalizer
for PDM-QAM16 [19]. Intradyne frequency offset is removed
using a fourth-power nonlinearity on the symbol sequence to
remove the effects of phase modulation, and the resulting com-
plex signal is then examined in the frequency domain to find the
frequency of the resulting intradyne frequency tone as in [20].
Phase recovery is then performed using the well-known Viterbi
and Viterbi algorithm for PDM-QPSK [21] modified from the
(10) normal fourth-power nonlinearity by raising the phase of each
symbol to the power of four and the magnitude to the power of
1.3. This modification reduces the weighting in the averaging
filter given to the highest power samples, which have undergone
the most nonlinear distortion. A decision-directed digital phase
locked loop was used in the case of PDM-QAM16 modulated
(11) signals [19]. Symbol estimation is then performed, followed by
determination of bit error rate (BER) for quantifying system
performance.
IV. EXPERIMENTAL TRANSMISSION SETUP
To characterize the functionality of our digital coherent re-
ceiver with nonlinearity compensation, we performed a set of
transmission experiments to examine the effects of linear and
nonlinear impairments. A particular focus was to investigate the
effectiveness of nonlinear compensation techniques for PDM-
QPSK and PDM-QAM16 modulation formats and the impact
of varying DSP complexity on the transmission performance.
The optical signals were transmitted multiple times through a
single-span recirculating fiber loop, with the following coher-
ent detection, digitization, and offline DSP, as shown in Fig. 5.
The loop consisted of 80.2 km single mode fiber (SMF) with an
overall chromatic dispersion of 1347 ps/nm and loss of 15.4 dB.
The experimental procedure was similar to that of [17] and [22].
The polarization-multiplexed QPSK signal was generated us-
ing an I–Q modulator, which was driven over 2Vπ with respect
to the minimum bias point of its transfer function. For the data,
MILLAR et al.: MITIGATION OF FIBER NONLINEARITY USING A DIGITAL COHERENT RECEIVER
Fig. 6. Transmitter structure for PDM-QPSK, with optional QAM16 stage
highlighted. Inset: optical eye-diagrams at the output of the transmitter for (top)
PDM-QPSK and (bottom) PDM-QAM16.
two decorrelated 212 pseudorandom binary sequences were used
from the output of the pulse pattern generator (PPG), which
were subsequently amplified to 7Vp−p (2Vπ ) to separately drive
the I and Q arms of the modulator. The transmitter DFB laser
linewidth, wavelength, and output optical power were 1 MHz,
1554 nm, and 8 dBm, respectively. To emulate polarization
multiplexing, we used a passive delay-line fiber interferometer,
where two data were again decorrelated, time and amplitude
aligned, and finally, recombined via a polarization beam splitter
(PBS), as shown in Fig. 6.
To synthesize a PDM-QAM16 signal, we employed a re-
cently developed method based on the interferometric optical
processing of a QPSK signal [23]. To aid carrier phase estima-
tion, an external cavity laser (ECL) with a linewidth of 100 kHz
was used in the QAM16 transmitter. The initial QPSK signal
is launched into a phase-stabilized fiber interferometer, where
the two signals are decorrelated, time-aligned, and attenuated
with respect to each other by 6 dB (highlighted, Fig. 6). The
phase between two arms was set to 90◦ and maintained utilizing
a feedback circuit. For the feedback circuit, we used a ditherless
bias control circuit, alternatively, a circuit design described in
[24] may be used. Even though this method cannot be used to in-
dependently modulate different streams of data, this can be used
to investigate transmission performance of PDM-QAM16 sig-
nals. In addition, this generation method allows suppression of
the transfer of noise between the electrical and optical domains
in the transmitter, owing to the nonlinear transfer function of the
modulator. Polarization multiplexing emulation was performed
as in the PDM-QPSK case.
V. SIMULATION OF NONLINEAR EQUALIZER PERFORMANCE
To verify experimental results, a 215 long symbol sequence
has been simulated for 10.7 GBd PDM-QPSK and PDM-
QAM16 transmission. Laser phase noise was modeled as a
Wiener process, leading to a transmitter laser linewidth of 1 MHz
for PDM-QPSK and 100 kHz for PDM-QAM16, while the influ-
ence of relative intensity noise (RIN) was neglected throughout
the simulations. An electrical fifth-order Bessel filter with a
3 dB bandwidth of 26 GHz was used to emulate the limited
transmitter bandwidth.
To ensure accurate simulation of the experimental setup, the
transmission link shown in Fig. 5 has been modeled in as much
detail as possible, while making the following assumptions.
Each acousto-optic modulator (AOM) is assumed to introduce
a loss of 3 dB, while erbium-doped fiber amplifiers are oper-
ated in saturation to give a fixed output power of 17 dBm, and
1221
TABLE I
FIBER AND LINK PARAMETERS
add noise power to the signal corresponding to a noise figure
of 4.5 dB. Further attenuation is then applied to attain the de-
sired launch power into the fiber. We simulated a span length
of 80.2 km, modeling propagation inside the transmission fiber
with a symmetrical SSFM covering linear effects, the Kerr ef-
fect, PMD, and nonlinear polarization scattering. The fiber pa-
rameters, which were used are detailed in Table I. The optical
loop filter was modeled as a second-order Gaussian filter with a
3 dB bandwidth of 100 GHz.
After transmission, the signal was detected with a single-
ended coherent receiver assuming a local oscillator (LO) to
signal ratio of 24 dB and an LO linewidth of 100 kHz. Limited
receiver bandwidth was largely determined by the p-i-n photo-
diodes, which are modeled with fifth-order Bessel filters using a
3 dB bandwidth of 7 GHz. Receiver-side A/D converters intro-
duce additional quantization noise and are modeled as having an
effective resolution of 4 bits. Subsequent DSP was performed
as described in Sections II and III.
The residual implementation penalty is assumed to stem
mainly from electrical noise in transmitter and receiver, and is
modeled by adding additional electrical noise at the receiver to
give back-to-back performance similar to the measured receiver
sensitivity.
VI. PDM-QPSK AND PDM-QAM16 TRANSMISSION RESULTS
To examine the variation of system performance with vari-
ous implementations of digital backpropagation, we performed
experiments with PDM-QPSK and PDM-QAM16 near to their
maximum reach without nonlinearity compensating DSP. A
symbol rate of 10.7 GBd was chosen to exploit the full receiver
bandwidth when using T /2 sampling for processing. The dig-
ital backpropagation algorithm was then investigated, in terms
of both the nonlinear step size and the use of both Wiener and
Wiener–Hammerstein models.
The performance of the Wiener cascade model backpropa-
gation was experimentally characterized for PDM-QPSK and
PDM-QAM16, and the results shown in Figs. 7 and 8, respec-
tively. In Figs. 7 and 8, Q-factor in decibel is plotted as a contour
graph against nonlinear step size in kilometers on the horizontal
axis and launch power in dBm on the vertical axis. Algorithm
performance was examined over 97 spans (7780 km) for PDM-
QPSK and 20 spans (1600 km) for PDM-QAM16. As these dis-
tances are close to maximum reach for the modulation formats
investigated, both nonlinear effects and the possible benefits of
nonlinearity compensation are more significant than for shorter
distances.
1222 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 16, NO. 5, SEPTEMBER/OCTOBER 2010
Fig. 7. Contour plot of experimentally determined Q-factor in decibels against
launch power and nonlinear step size for Wiener cascade compensation of 97
spans transmission PDM-QPSK at 10.7 GBd. Nonlinear step size of a single
span lies at 80 km.
Fig. 8. Contour plot of experimentally determined Q-factor in decibels against
launch power and nonlinear step size for Wiener cascade compensation of 20
spans transmission PDM-QAM16 at 10.7 GBd. Nonlinear step size of a single
span lies at 80 km.
In Figs. 7 and 8, we observe that for lower powers, perfor-
mance is limited by the accumulated optical noise. For launch
powers of below –5 dBm, there is an insignificant improvement
in performance for either modulation format with any step size.
As launch power is increased, we note that an improvement in
performance is available for reduced step sizes up to 160 km.
In the case of PDM-QPSK, the optimum launch power is im-
proved by some 2 dB: from approximately −4.5 to −2.5 dBm.
In the case of PDM-QAM16, optimum launch power increases
by 2.5 dB: from approximately −3.5 to −1 dBm. For PDM-
QPSK, the benefits available with decreasing nonlinear step
size become saturated at 160 km, while with PDM-QAM16,
the improvement in performance becomes saturated at approx-
imately 80 km. A common characteristic of these curves is that
the improvement in maximum Q-factor may be modest (in the
region of 1 dB in each case), but the increase in input dynamic
range (that is, the range of launch powers for which the BER
is less the forward error correction limit) may be dramatic: ap-
proximately 4 dB for PDM-QPSK and greater than 5 dB for
PDM-QAM16.
Fig. 9. Q-factor in decibels for transmission of 10.7 GBd PDM-QPSK over 97
spans. Broken lines denote experimental data, while solid markers correspond
to simulated results. Various nonlinear step sizes are demonstrated.
Fig. 10. Q-factor in decibels for transmission of 10.7 GBd PDM-QAM16
over 20 spans. Broken lines denote experimental data, while points correspond
to simulated results. Various nonlinear step sizes are demonstrated.
Figs. 9 and 10 show the variation of the Q-factor against
launch power with varying nonlinear step sizes for PDM-QPSK
and PDM-QAM16, respectively. Experimental data is denoted
by broken lines, while Monte Carlo simulations (as described in
Section IV) are denoted by points in Figs. 9 and 10. In both cases,
there is a very good agreement between the experimental and
simulated results. Linear compensation denotes compensation
for chromatic dispersion only, while 1 step per span and 16
steps per span correspond to nonlinear step sizes of 80 and
5 km, respectively. One step per link and four steps per link
correspond to a first-order and a fourth-order nonlinear cascade,
respectively. For PDM-QPSK over 97 spans, this corresponds
to nonlinear step sizes of 7780 and 1945 km, respectively. For
PDM-QAM16 over 20 spans, this corresponds to nonlinear step
sizes of 1600 and 400 km, respectively.
The influence of the distribution of dispersion between the
two linear blocks in the Wiener–Hammerstein cascade was
MILLAR et al.: MITIGATION OF FIBER NONLINEARITY USING A DIGITAL COHERENT RECEIVER 1223

Fig. 11. Variation of experimental Q-factor in decibels with distribution of

dispersion in Wiener–Hammerstein cascade compensation of 97 spans trans- Fig. 12. Variation of Q-factor in decibels with launch power for 20 spans
mission PDM-QPSK at 10.7 GBd. For the purposes of illustration, 0.5 dBm PDM-QAM16 for different nonlinear step sizes with Wiener model compensa-
launch power and an 80 km nonlinear step size are shown. tion. Experimental data shown as points and fit shown as solid lines.

investigated. This corresponds to varying the position of the

nonlinear block within the section of fiber, which is approxi-
mated by each three block system. The optimum was found to
be 85% of the dispersion in the first block and 15% in the sec-
ond block. This corresponds to applying the nonlinearity at half
of the effective nonlinear length of the fiber section. A graph
illustrating this is shown in Fig. 11, which shows the variation
of Q-factor with the length of the first dispersive step. Fig. 11
shows an 80 km nonlinear step size for 0.5 dBm launch power.
Other launch powers and nonlinear step sizes also exhibit a max-
imum when the split of dispersion is 85%–15%, although the
maximum improvement in Q-factor is smaller for lower launch
powers and smaller nonlinear step sizes. This dispersive split
was used for all subsequent uses of Wiener–Hammerstein model
Fig. 13. Plot of improvement in inferred maximum Q-factor against mean
backpropagation. dispersive block length for PDM-QPSK at 10.7 GBd using Wiener and Wiener–
To effectively represent and compare the results between the Hammerstein model nonlinearity compensation.
Wiener and Wiener–Hammerstein models, we used fitted curves
to quantify improvements in launch power and Q-factor with re-
duction in step size. A least-squares fit was used to produce the
polynomial approximation of Q-factor as a function of launch which we define as follows:1
power given in [24]. The quality of the polynomial fit is illus- Total link length
trated for PDM-QAM16 with several orders of Wiener model Mean dispersive block length = .
Number of dispersive blocks
nonlinearity compensation in Fig. 12. This fitting process was
carried out for both the Wiener and Wiener–Hammerstein mod- The characterization of PDM-QPSK with varying mean dis-
els of nonlinearity compensation, varying the nonlinear step size persive block length is given later in Figs. 13 and 14. Here, we
between 5 km and the entire link for each modulation format. plot improvement in Q-factor at the optimum launch power (see
Using this fitting process, we were able to extrapolate the Figs. 13 and 14) using the polynomial fitting process with the
optimum launch power for different models and spatial reso- experimental measurements as previously described.
lutions of nonlinearity compensation. This inferred optimum It is noted that in both Figs. 13 and 14 both nonlinear compen-
launch power is useful as the experimental measurements have sation models offer similar potential benefits for a given mean
a granularity of 1 dB in launch power, and the change in opti- dispersive block length. Maximum Q-factor is improved by ap-
mum launch power may be less than this, i.e., between different proximately 1.6 dB, and optimum launch power is increased
nonlinear step sizes. From the polynomial approximation of
variation in Q-factor with launch power, we are also able to 1 Where there are two adjacent dispersive blocks in the Wiener–Hammerstein

infer Q-factor at the optimum launch power.
To account for the difference in complexity between the two
nonlinear models and allow a more direct comparison, perfor-
mance is characterized for the mean dispersive block length,
system, we assume that they may be incorporated into a single block. Therefore,
for a very high-order cascade, the mean dispersive block length of the Wiener–
Hammerstein system will approach that of the Wiener system. Conversely, for
a first-order nonlinear model, the Wiener model will have a mean dispersive
block length, which is twice that of the Wiener–Hammerstein model.
1224 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 16, NO. 5, SEPTEMBER/OCTOBER 2010

Fig. 14. Plot of improvement in inferred optimum launch power against mean Fig. 15. Plot of improvement in inferred maximum Q-factor against mean dis-
dispersive block length for PDM-QPSK at 10.7 GBd using Wiener and Wiener– persive block length for PDM-QAM16 at 10.7 GBd using Wiener and Wiener–
Hammerstein model nonlinearity compensation. Hammerstein model nonlinearity compensation.

by approximately 1.9 dB for both compensation models. The

improvement in both Q-factor and launch power saturates for a
mean dispersive block length of approximately 160 km, or two
spans. The similarity between the models when characterized
in terms of mean dispersive block length may be considered
with reference to the discussion of the accuracy of the models
presented in Section II. When the step size is small, the accu-
racy of both models is good and there is agreement in accuracy
between the two models as the error due to the commutability
of the linear and nonlinear blocks is insignificant. As the non-
linear step size increases, the benefit in accuracy of the Wiener–
Hammerstein model becomes more significant. This difference
is offset, however, due to the difference in mean dispersive block
length between the two models. This becomes most noticeable
for a nonlinear step size corresponding to the link length L: for
this case, the mean dispersive block length is L for the Wiener
cascade, but L/2 for the Wiener–Hammerstein cascade. Fig. 16. Plot of improvement in inferred optimum launch power against
The performance of PDM-QAM16 is characterized at 20 mean dispersive block length for PDM-QAM16 at 10.7 GBd using Wiener
spans transmission for the Wiener and Wiener–Hammerstein and Wiener–Hammerstein model nonlinearity compensation.
models for nonlinearity compensation later in Figs. 15 and 16.
Again, we plot improvement in Q-factor at the optimum launch
power (see Figs. 15 and 16) using the polynomial fitting process
with the experimental measurements.
We note that similarly to the PDM-QPSK case, the perfor-
mance of the Wiener and Wiener–Hammerstein models are
remarkably similar. Benefits in performance are saturated for
a mean dispersive block length of less than 80 km, with a max-
imum improvement in Q-factor of approximately 1 dB, and an
increase in optimum launch power of 2.5 dB. This reflects the
greater improvement in input dynamic range noted for PDM-
QAM16 in comparison to PDM-QPSK.
The reduced number of required nonlinear blocks for signif-
icant improvement in performance for PDM-QAM16 in com-
parison with PDM-QPSK is shown in Fig. 17. For a 1.5 dB
increase in optimum launch power, it is noted that PDM-QPSK
over 97 spans requires approximately 25 nonlinear blocks over
7780 km, while PDM-QAM16 requires approximately 7 over Fig. 17. Plot of improvement in inferred optimum launch power against num-
1600 km. This corresponds to a slightly smaller nonlinear step ber of nonlinear blocks for Wiener model nonlinearity compensation of PDM-
size for PDM-QAM16 (230 km compared to 310 km), as may QPSK and PDM-QAM16.
MILLAR et al.: MITIGATION OF FIBER NONLINEARITY USING A DIGITAL COHERENT RECEIVER
Fig. 18. Plot of improvement in inferred optimum launch power against non-
linear step size for Wiener model nonlinearity compensation of PDM-QPSK
and PDM-QAM16.
be seen in Fig. 18, though this difference is dwarfed by the
reduction of nonlinear blocks due to the reduced transmission
distance. This is also in agreement with our earlier observa-
tion that PDM-QAM16 requires a smaller nonlinear step size
than PDM-QPSK to gain the maximum available benefits from
backpropagation. Again, the data used in Figs. 17 and 18 is
from polynomial fitting of the experimental measurements as
previously described.
VII. CONCLUSION
We have investigated the performance of a coherent receiver
with nonlinearity compensating DSP and have shown that it
can be successfully used to mitigate intrachannel nonlinearities
in both PDM-QPSK and PDM-QAM16 over distances of 7780
and 1600 km, respectively. The impact of the key receiver DSP
parameter, namely the nonlinear step size was investigated. It
was shown that significant improvements in performance may
be achieved with resolution significantly coarser than a sin-
gle span. While performance in this long-step region may be
improved with the use of a three block Wiener–Hammerstein
model rather than the more commonly used Wiener model, the
increased computational effort this model requires offsets any
benefit when performance is examined in terms of the mean dis-
persive block length. For the examined receiver bandwidth and
symbol rate, the benefit of nonlinearity compensation saturated
for a nonlinear step size of 160 km for PDM-QPSK and 80 km
for PDM-QAM16. Additionally, nonlinear backpropagation ap-
pears to offer a greater benefit for PDM-QAM16, which may be
attributed to this format’s higher susceptibility to fiber nonlinear
effects. This leads us to infer that nonlinearity compensation of
this kind is considerably more attractive for modulation formats,
which are highly spectrally efficient, and transmitted over short
links, where the reduced memory due to dispersion and the in-
creased benefit available combine to produce greater benefits
from fewer blocks.
REFERENCES
[1] A. Chraplyvy, “The coming capacity crunch,” in Proc. ECOC 2009, Paper
Mo1.0.2.
1225
[2] R. J. Essiambre, G. J. Foschini, G. Kramer, and P. J. Winzer, “Capacity
limits of information transport in fiber-optic networks,” Phys. Rev. Lett.,
vol. 101, p. 163901, Oct. 2008.
[3] P. J. Winzer, A. H. Gnauck, C. R. Doerr, M. Magarini, and L. L.
Buhl, “Spectrally efficient long-haul optical networking using 112-Gb/s
polarization-multiplexed 16-QAM,” J. Lightw. Technol., vol. 28, no. 4,
pp. 547–556, Feb. 2010.
[4] C. Xie, “Suppression of inter-channel nonlinearities in WDM coher-
ent PDM-QPSK systems using periodic-group-delay dispersion compen-
sators,” in Proc. ECOC 2009, pp. 1–2, Paper P4.08.
[5] X. Li et al., “Electronic post-compensation of WDM transmission impair-
ments using coherent detection and digital signal processing,” Opt. Exp.,
vol. 16, no. 2, pp. 880–888, Jan. 2008.
[6] E. Ip and J. Kahn, “Compensation of dispersion and nonlinear effects using
digital backpropagation,” J. Lightw. Technol., vol. 26, no. 20, pp. 3416–
3425, Oct. 2008.
[7] G. Charlet et al., “72 × 100 Gb/s transmission over transoceanic distance,
using large effective area fiber, hybrid Raman-erbium amplification and
coherent detection,” in Proc. OFC/NFOEC 2009, Paper PDPB6.
[8] G. Goldfarb, M. G. Taylor, and G. Li, “Experimental demonstration of fiber
impairment compensation using the split-step finite-impulse-response fil-
tering method,” IEEE Photon. Tech. Lett., vol. 20, no. 22, pp. 1887–1889,
Nov. 2008.
[9] G. P. Agrawal, Nonlinear Fiber Optics, 3rd ed. New York: Academic,
2001, pp. 45–51.
[10] D. Marcuse, C. R. Menyuk, and P. K. A. Wai, “Application of the Manakov-
PMD equation to studies of signal propagation in optical fibers with
randomly varying birefringence,” J. Lightw. Technol., vol. 15, no. 9,
pp. 1735–1746, Oct. 1997.
[11] C. R. Menyuk, “Application of multiple-length-scale methods to the study
of optical fiber transmission,” J. Eng. Math., vol. 36, no. 1–2, pp. 113–136,
1999.
[12] S. Oda et al., “112Gbps DP-QPSK transmission using a novel nonlinear
compensator in digital coherent receiver,” presented at the OFC/NFOEC
2009, San Diego, CA, Mar. 22–26, Paper OThR6.
[13] F. Yaman and G. Li, “Nonlinear impairment compensation for polariza-
tion multiplexed WDM transmission using digital backward propagation,”
IEEE Photon. J., vol. 1, no. 2, pp. 143–152, Aug. 2009.
[14] R. Haber and H. Unbehaben, “Structure identification of nonlinear dy-
namic systems – A survey on input/output approaches,” Automatica,
vol. 26, no. 4, pp. 651–677, 1990.
[15] G. H. Weiss and A. A. Maradudin, “The Baker-Hausdorff formula and a
problem in crystal physics,” J. Math. Phys., vol. 3, no. 4, pp. 771–777,
Jul./Aug. 1962.
[16] E. Yamazaki et al., “Multi-staged nonlinear compensation in coherent
receiver for 16340-km transmission of 111-Gb/s no-guard-interval Co-
OFDM,” presented at the ECOC 2009, Vienna, Austria, Paper Th9.4.6.
[17] D. S. Millar, S. Makovejs, V. Mikhailov, R. I. Killey, P. Bayvel, and S. J.
Savory, “Experimental comparison of nonlinear compensation in long-
haul PDM-QPSK transmission at 42.7 and 85.4 Gb/s,” in Proc. ECOC
2009, pp. 1–2, Paper Th9.4.4.
[18] S. J. Savory, “Digital filters for coherent optical receivers,” Opt. Exp.,
vol. 16, pp. 804–817, Jan. 2008.
[19] I. Fatadin, D. Ives, and S. J. Savory, “Blind equalization and carrier phase
recovery in a 16-QAM optical coherent system,” J. Lightw. Technol.,
vol. 27, no. 15, pp. 3042–3049, Aug. 2009.
[20] D. C. Rife and R. C. Boorstyn, “Single-tone parameter estimation from
discrete-time observations,” IEEE Trans. Inf. Theory, vol. 20, no. 5,
pp. 591–598, Jul. 1974.
[21] A. Viterbi and A. Viterbi, “Nonlinear estimation of PSK modulated carrier
phase with application to burst digital transmission,” IEEE Trans. Inf.
Theory, vol. 29, no. 4, pp. 543–544, Jul. 1982.
[22] S. Makovejs et al., “Experimental investigation of PDM-QAM16 trans-
mission at 112 Gbit/s over 2400 km,” in Proc. OFC/NFOEC 2010, Paper
OMJ6.
[23] S. Makovejs et al., “Novel method of generating QAM-16 signals at 21.3
Gbaud and transmission over 480 km,” IEEE Photon. Tech. Lett., vol. 22,
no. 1, pp. 36–38, Nov. 2009.
[24] S. Makovejs, G. Gavioli, V. Mikhailov, R. I. Killey, and P. Bayvel,
“Experimental and numerical investigation of bit-wise phase-control
OTDM transmission,” Opt. Exp., vol. 16, no. 23, pp. 18725–18730, Oct.
2008.
[25] S. J. Savory, “Optimum electronic dispersion compensation strategies for
nonlinear transmission,” Electron. Lett., vol. 42, no. 7, pp. 407–408,
2006.
1226 IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, VOL. 16, NO. 5, SEPTEMBER/OCTOBER 2010
David S. Millar (S’07) was born in Manchester,
U.K., in 1982. He received the M.Eng. degree in
electronic and communications engineering from
the University of Nottingham, Nottingham, U.K., in
2007. He is currently working toward the Ph.D. de-
gree in digital signal processing for coherent optical
communication in the Optical Networks Group, Uni-
versity College London, London, U.K.
His current research interests include signal-
processing techniques for high-level modulation for-
mats, and nonlinearity compensation.
Mr. Millar was a Reviewer for several IEEE publications, including IEEE
PHOTONICS TECHNOLOGY LETTERS and the IEEE JOURNAL OF LIGHTWAVE
TECHNOLOGY.
Sergejs Makovejs (S’08) was born in Volgograd,
Russia, in 1983. He received the B.Sc. and M.Sc.
degrees in telecommunications from Riga Technical
University, Riga, Latvia, in 2004 and 2006, respec-
tively. Since 2007, he has been working toward the
Ph.D. degree from the University College London,
London, U.K., where his research has been focused
on high-speed optical fibre communication systems.
In 2006, he was at Siemens, Munich, Germany,
where he was involved in designing of rail automation
and signalling systems. He has authored and coau-
thored five peer-reviewed conference and journal papers.
Mr. Makovejs was nominated for the Corning Outstanding Student Paper
Competition at OFC/NFOEC 2010.
Carsten Behrens (S’09) was born in Potsdam,
Germany, in 1981. He received the Dipl.-Ing. de-
gree in electronic and electrical engineering from the
Technische Universität Berlin, Berlin, Germany, in
2007. He is currently working toward the Ph.D. de-
gree in the Optical Networks Group, University Col-
lege London, London, U.K.
His current research interests include coherent de-
tection of higher order modulation formats and non-
linear effects in optical fibers.
Mr. Behrens was the recipient of the Erwin-
Stephan Prize for excellent results in 2008.
Stephan Hellerbrand (S’03) was born in Munich,
Germany, in 1980. He received the B.Sc. degree
in electrical engineering in 2003 from Technische
Universität München (TUM), Munich, Germany,
with a thesis on joint source-channel coding. In the
same year he spent two terms at the University of
Melbourne, Melbourne, Australia. In 2005, he re-
ceived the Dipl.-Ing. degree in electrical engineering
from TUM with a thesis on data compression, which
was the result of a research visit to Lund Institute of
Technology, Lund, Sweden.
Since October 2005, he has been working towards the Dr.-Ing. degree as a
full-time member of the research and teaching staff at the Institute for Commu-
nications Engineering of TUM. His research focus is on system modeling and
signal processing for the compensation of transmission impairments in fiber-
optic communication links.
Robert I. Killey (M’00) received the B.Eng. degree
in electronic and communications engineering from
the University of Bristol, Bristol, U.K., in 1992, the
M.Sc. degree in microwaves and optoelectronics from
University College London (UCL), London, U.K., in
1994, and the D.Phil. degree from the University of
Oxford, Oxford, U.K., in 1998. His doctoral work
was on InGaAsP Fabry–Pérot optical modulators
and their applications in soliton communications, in
collaboration with Alcatel Submarine Systems Ltd.,
Greenwich, U.K.
He was a Research Fellow in the Department of Electronic and Electrical
Engineering, Optical Networks Group, UCL, and became a Lecturer in 2000.
He has authored or coauthored more than 100 journal and conference papers.
His current research interests include nonlinear fiber effects in wavelength di-
vision multiplexing transmission systems, wavelength-routed optical networks,
and applications of electronic signal processing in optical communications.
Dr. Killey is a member of the IEEE Photonics Society and the Institution
of Engineering and Technology. He has been involved in the Technical Pro-
gram Committees for European Conference on Optical communication, IEEE
Photonics Society Annual Meeting, and Optoelectronics and Communications
Conference.
Polina Bayvel (S’87–M’89–SM’00–F’09) received
the B.Sc. (Eng.) and Ph.D. degrees in electronic
and electrical engineering from University College
London (UCL), London, U.K., in 1986 and 1990, re-
spectively. Her Ph.D. research focused on nonlinear
fiber optics and their applications.
In 1990, she was at the Fiber Optics Labora-
tory, General Physics Institute, Moscow (Russian
Academy of Sciences) under the Royal Society Post-
doctoral Exchange Fellowship. She was a Principal
Systems Engineer at STC Submarine Systems, Ltd.,
London, U.K., and Nortel Networks (Harlow, U.K., and Ottawa, Canada), where
she was involved in the design and planning of optical fiber transmission net-
works. During 1994–2004, she held a Royal Society University Research Fel-
lowship at UCL, and in 2002, she became a Chairperson in Optical Commu-
nications and Networks. She is currently the Head of the Optical Networks
Group, UCL. She has authored/coauthored more than 260 refereed journal and
conference papers. Her research interests include optical networks, high-speed
optical transmission, and the study and mitigation of fiber nonlinearities.
Prof. Bayvel is a member of the Technical Program Committee (TPC) of
a number of conferences, including European Conference on Optical Commu-
nication (ECOC) and Co-Chairperson of the TPC for ECOC 2005. She is the
2002 recipient of the Institute of Physics Paterson Prize and Medal for her con-
tributions to research on the fundamental aspects of nonlinear optics and their
applications in optical communications systems. In 2007, she was the recipient
of the Royal Society Wolfson Research Merit Award. She is a Fellow of the
Royal Academy of Engineering (FREng), Optical Society of America, the U.K.
Institute of Physics, and Institute of Engineering and Technology.
Seb J. Savory (M’07) was born in Stirling, U.K.,
in 1973. He received the M.Eng., M.A., and Ph.D.
degrees in engineering from the University of
Cambridge, Cambridge, U.K., in 1996, 1999, and
2001, respectively, and the M.Sc. degree in math-
ematics from the Open University, Milton Keynes,
U.K., in 2007.
His interest in optical communications began in
1991, when he joined Standard Telecommunications
Laboratories, Harlow, U.K., prior to being sponsored
though his undergraduate and postgraduate studies by
Nortel. On completion of his Ph.D. degree in 2000, he joined Nortel’s Harlow
Laboratories as Senior Research Engineer, where he was engaged in research
into digital signal processing and advanced optical transmission systems. In
2005, he joined the Optical Networks Group at University College London
(UCL), London, U.K., where he held a Leverhulme Trust Early Career Fellow-
ship from 2005–2007, being appointed as a University Lecturer in 2007. From
June 2009–June 2010, he was also a Visiting Professor at the Politecnico di
Torino, Italy. His research interests include digital coherent transceivers, optical
transmission systems and subsystems, digital signal processing and nonlinear
systems.
Dr. Savory is a Chartered Engineer and an Associate Editor for IEEE PHOTON-
ICS TECHNOLOGY LETTERS. He also serves on the technical program committee
for the Optical Fiber Communication Conference, the European Conference on
Optical Communication and the IEEE Photonics Society Annual Meeting.