Chapter 5: Linear Block Codes: Vahid Meghdadi

Outline
Basic principles
Linear Block Coding
Chapter 5: Linear Block Codes
Vahid Meghdadi
University of Limoges
meghdadi@ensil.unilim.fr
Vahid Meghdadi Chapter 5: Linear Block Codes

Outline
Basic principles
Linear Block Coding
Basic principles
Introduction
Block codes
Error detection
Error correction
Linear Block Coding

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Introduction
I Transmission through noisy channel

I Transmission errors can occur, 1’s become 0’s and 0’s become
1’s
I To correct the errors, some redundancy bits are added to the
information sequence, at the receiver the correlation is
exploited to locate transmission errors
I Here, only binary transmission is considered.
I In this chapter we try to optimally add this redundancy to the
information bits.

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Block codes (1/3)
Definition of block coding:

I A message m is a binary sequence of size k: m ∈ Ak
I To each message m correspond a codeword which is a binary
sequence c of size n : n > k.

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Block codes (2/3)
I The code space contains 2n points but only 2k of them are

valid codewords.
I A code must be a one-to-one relation (injection)
Example:
m ∈ {00, 01, 10, 11}
c ∈ {000, 011, 101, 110}
This is a one parity check code. 001,010,100,111 ∈
/C

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Block code (3/3)
Definition:
The rate of a code is defined as k/n. For example for the previous
code n = 3, k = 2 so the rate is 2/3.

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Linear codes
Definition:
A binary code is linear if the following condition is satisfied:
∀c1 ∈ C , ∀c2 ∈ C ⇒ c1 + c2 ∈ C
Example: With the previous example
011 + 101 = 110 ∈ C
011 + 110 = 101 ∈ C

101 + 110 = 110 ∈ C

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Hamming distance and Hamming weight (1/2)
Definition :
Hamming weight of a binary sequence is defined as the number of
1 in the sequence.
w ([101]) = 2 w ([11101]) = 4
Definition:
Hamming distance between two binary sequences v and w is the
number of places where they differ.
Example
v = [110101]
⇒ d(v, w) = 2
w = [110011]

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Hamming distance and Hamming weight (2/2)
Property 1: d(v, w) + d(w, x) ≥ d(v, x) (triangle inequality)

Property 2: d(v, w) = w (v + w)
Example:
v = [110101] w = [110011]
v + w = [000110] d(v, w) = w (v + w) = 2

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Minimum distance of block code
dmin = min{d(v, w) : v, w ∈ C and v 6= w}

v,w
If the code is linear:
dmin = min{w (v + w) : v, w ∈ C and v 6= w}

= min{w (x) : x ∈ C and x 6= 0}
= wmin
Theorem: The minimum distance of a linear block code is equal to

the minimum weight of its non zero codeword.

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Good codes
A good code is one whose minimum distance is large.
For the previous example dmin = 2:
{00, 01, 10, 11} ⇒ {000, 011, 101, 110}

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Error detection (1/3)
I Noisy channel introduces an error-pattern: c is sent, r is

received where r = c + e
I w (e) =the number of errors that occur during the
transmission
I e is called error pattern
I For a code with dmin as minimum distance, any two distinct
codewords differ in at least dmin places.
I No error pattern of weight dmin − 1 or less can change one
codeword into another.

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
I Any error pattern of size at most dmin − 1 can be detected.

I Some error patterns of size dmin or larger can be detected.
Example:
{000, 011, 101, 110} ∈ C
For this code dmin = 2:
I any one error is detected
I but any 3 errors can be detected as well.

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
I For c + e ∈ C, and e 6= 0, the error cannot be detected.

Because the code is linear, only for e as a codeword this
constraint is satisfied.
I So, for a (n, k) code, there are exactly 2n − 2k error patterns
that can be detected.
Example: for the previous code, all the following error pattern can
be detected:
{[001], [010], [100], [111]}
For this code, n = 3, k = 2, 2n − 2k = 4

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Error Detection Capacity
I The error detection fails only when the error pattern is

identical to a non-zero codeword, otherwise c + e ∈ C and the
error is not detected.
Definition: Weight distribution
I Let Ai be the number of codewords of weight i,
I A0 , A1 , · · · , An are called the weight distribution of the code.

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Code example
For this example:

A0 = 1, A1 = A2 = 0, A3 = 7, A4 = 7, A5 = A6 = 0, A7 = 1
Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Undetected Error Probability
Definition : Pu (E ) probability of undetected error
Pu (E ) = prob(e ∈ C)
In a BSC channel with p as the error transmission probability

n
X
Pu (E ) = Ai p i (1 − p)n−i
i=1

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Example
Considering previous code (7,4)
Pu (E ) = 7p 3 (1 − p)4 + 7p 4 (1 − p)3 + p 7
For example if p = 0.01 then Pu (E ) = 7 × 10−6
0 1-p
0
p
p
1 1
1-p

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Error correction 1/2

How many errors can be corrected ?
Let t be a positive number such that 2t + 1 ≤ dmin < 2t + 2
Let v be the transmitted and r the received sequences, and w any
codeword. We can write:
d(v, r) + d(w, r) ≥ d(v, w)
Suppose that t 0 errors occur: d(v, r) = t 0 , and for any v, w ∈ C
d(w, v) ≥ dmin ≥ 2t + 1
So d(w, r) ≥ 2t + 1 − t 0 . If t 0 ≤ t then:
d(w, r) > t
Conclusion: For any error pattern of t or fewer errors, r is closer to
real codeword v than any other codeword. So any t errors or fewer
can be corrected.
Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Error correction 2/2
Number
of vectors (n-tuples) around a codeword with a distance of
n
u is: .
u
Definition: Hamming Sphere of radius r is the number of vectors
around a codeword with a Hamming distance up to t:
t
X n
V (n, t) =
j
j=0

Introduction
Outline
Block codes
Basic principles
Error detection
Linear Block Coding
Error correction
Conclusion
I A block code with minimum distance dmin guarantees
correcting all error patterns of t = b(dmin − 1)/2c or fewer.
I t = b(dmin − 1)/2c is called error correcting capability of the
code.
I Some patterns of t + 1 or more can be corrected as well.
I In general, for a t-error-correcting (n, k) linear code, 2n−k
error patterns can be corrected.
I For a t-error-correcting (n, k) linear code and using an optimal
error correcting algorithm at the receiver, the probability that
the receiver make a bad correction is upper bounded by
n
X n
P(E ) < p i (1 − p)n−i
i
i=t+1

Outline
Basic principles
Linear Block Coding
Basic principles
Introduction
Block codes
Error detection
Error correction
Linear Block Coding

Outline
Basic principles
Linear Block Coding
Introduction
I Given n and k, how the code can be designed ?

I How an information sequence can be mapped to the
corresponding code word ?
I We are interested in systematic design.

Outline
Basic principles
Linear Block Coding
Code subspace
I An (n, k) linear code is a k-dimensional subspace of the vector

space of all the binary n-tuples, so it is possible to find k
linearly independent code words g0 , g1 , · · · , gk−1 to span this
space.
I So any code word can be written as a linear combination of
these base vectors:
c = m0 g0 + m1 g1 + · · · + mk−1 gk−1

Outline
Basic principles
Linear Block Coding
Generator matrix
We can arrange these k linearly independent code words (vectors)

as the rows of a k × n matrix as follows:
   
g0 g00 g01 ··· g0,n−1
 g1   g10 g11 · · · g1,n−1 
G= . =
   
. . . ..
 ..   .. .. .. 
. 
gk−1 gk−1,0 gk−1,1 · · · gk−1,n−1 k×n
c = mG

Outline
Basic principles
Linear Block Coding
Equivalent code
I One can obtain the same code (same codewords) but with a
different generator matrix. For example, we can exchange any
two rows of G.
I The only difference is that we obtain another mapping from
information sequences to codewords.
I We can replace each row by a linear combination of that row
with another row.
I All the codes obtained are equivalent.

Outline
Basic principles
Linear Block Coding
Systematic codes
Definition:
I If in all the codewords we can find exactly the corresponding
information sequence, the code is called systematic.
It is convenient to group all these bits either at the end or at the
beginning of the code word.
I In this case the generator matrix can be divided into two sub
matrices [P|I ].

Outline
Basic principles
Linear Block Coding
Systematic codes
   
g0 p00 p01 ··· p0,n−1 1 0 ···0
 g1   p10 p11 ··· p1,n−1 0 1 ···0 
G= =
   
.. .. .. .. .. .. .. .. .. 
 .   . . . . . . . . 
gk−1 pk−1,0 pk−1,1 · · · pk−1,n−1 0 0 ···1
G = [Pn×n−k Ik×k ]
So for a given message m the codeword will be
c = mG = [mP m]1×n . The first part of the codeword is the
parity bits and the second is the systematic part of the codeword.
Note:Any linear code can be transformed into systematic code.

Outline
Basic principles
Linear Block Coding
Example
Example: u = [1101] and

 
1 1 1 1 0 0 0
 1 0 1 0 1 0 0 
G=
 1

1 0 0 0 1 0 
0 1 1 0 0 0 1
So the codeword will be: [0011101]. The codeword is the sum of

the first, second and forth lines of the G.
The last four bits are the systematic part of the codeword.

Outline
Basic principles
Linear Block Coding
Parity check matrix
For any k × n matrix G with k linearly independent rows, there

exists an (n − k) × n matrix H with (n − k) linearly independent
rows such that any vector in the row space of G is orthogonal to
the rows of H and vice versa. So we can say:
I An n-tuple c is a codeword generated by G if and only if
cHT = 0
Matrix H is called Parity check matrix.
I The rows of H span a code space called dual code of G.
I GHT = 0

Outline
Basic principles
Linear Block Coding
Parity check matrix of systematic codes
For the systematic code G = [Pn×n−k Ik×k ], the parity check

T
matrix is simply H = [I(n−k)×(n−k) P(n−k)×k ]. (Why?)
Exercise: For the following generator matrix, give the generator
matrix of its dual code.
 
1 1 1 1 0 0 0
 1 0 1 0 1 0 0 
G=  1 1 0 0 0 1 0 

0 1 1 0 0 0 1
Note: A non systematic linear code can be transformed into

systematic and then the parity check matrix can be calculated.
This parity check matrix can also be used for the original code.

Outline
Basic principles
Linear Block Coding
Minimum distance of the code
Theorem: The minimum distance of a code is equal to the

smallest positive number of columns of H which are linearly
independent. That is, all combination of dmin − 1 columns of H are
linearly independent, so there is some set of dmin columns which
are linearly dependent.

Outline
Basic principles
Linear Block Coding
Singleton bound
For an (n, k) linear code
dmin ≤ n − k + 1
Proof: the rank(H) = n − k so any n − k + 1 columns of H must

be linearly dependent (the row rank of a matrix is equal to its
column rank). So dmin cannot be larger than n − k + 1 (according
to the previous theorem).
Definition: A code for which dmin = n − k + 1 is called
”maximum distance separable” code.

Outline
Basic principles
Linear Block Coding
Systematic Error Detection
I An n-tuple vector r is a codeword iff
rHT = 0
I The product S = rHT is a (1 × n − k) vector and is called the

syndrome of r.
I If the syndrome is not zero, a transmission error is declared.

Outline
Basic principles
Linear Block Coding
Systematic Error Correction
Maximum likelihood error correction:
ĉ = arg min dH (c, r)

c∈C
Should we test all the possibilities ?

For a (256,200) binary code, there are 2200 = 1.6 × 1060
possibilities to be verified !
Solution: Syndrome decoding which decreases the number of
possibilies.

Outline
Basic principles
Linear Block Coding
Syndrome Decoding
r is the received sequence

r =c+e
S = rHT = (c + e)HT = cHT + eHT = eHT
Conclusion: The syndrome is not a function of the transmitted
codeword but a function of error pattern. So we can construct only
a matrix of all possible error pattern with corresponding syndrome.

Outline
Basic principles
Linear Block Coding
Syndrome Table Construction
I There are 2n possible received vectors

I There are 2k valid codewords
I There are 2n−k possible syndromes
First we generate all the error patterns of length 1, and calculate
the corresponding syndrome and put all of them into a matrix.
Then, we increase the error pattern weight and we do the same
thing as before.
Each time if the new syndrome had already been saved, it will be
thrown away and we continue with the next error pattern.
When all the 2k syndromes are found, the table is complete.
For the previous example (256,200) the size of the matrix is
256 = 7.2 × 1016 , which is still too big.

Outline
Basic principles
Linear Block Coding
Exercise
  Error pattern Syndrome

1 0 0 0 0 1 1 0000000 0000
 0 1 0 0 1 0 1 
H=  1000000 1000
 0 0 1 0 1 1 0  .. ..
0 0 0 1 1 1 1 . .
0000001 1101
It is a (7,3) code with 1100000
dmin = 4. Complete the 1010000
syndrome table which has 16 .. ..
. .
rows: 1110000 1110

Outline
Basic principles
Linear Block Coding
Exercise
Suppose r = [0, 0, 1, 1, 0, 1, 1], calculate the syndrome and then

correct it in maximum likelihood sense. Use the same H as the
previous example.

Outline
Basic principles
Linear Block Coding
Hamming Code
Definition: If the syndrome table becomes complete for all
w (e) ≤ t the code is called perfect.
For example for the code (7, 4) with t = 1, the syndrome table is
completed for all the error patterns of weight 1. These codes with
t = 1 are called Hamming codes.
For Hamming codes, there are 2n−k − 1 non-zero syndromes. This
number must be equal to the number of error patterns of weight 1.
It means that 2n−k − 1 = n. Calling the number of parities
m = n − k, we can write for a Hamming code that:
k = 2m − m − 1
So the valid Hamming codes are:

(3, 1), (7, 4), (15, 11), ..., (2m − 1, 2m − m − 1).
Outline
Basic principles
Linear Block Coding
Error Detection Performance Example
As we saw before the probability that a received sequence is taken

for a codeword by false is given by:
n
X
Pu (E ) = Ai p i (1 − p)n−i
i=1
For a (7,4) code the weight distribution is given by

[1, 0, 0, 7, 7, 0, 0, 1] and for a (15,11) code it is
[1, 0, 0, 35, 105, 168, 280, 435, 435, 280, 168, 105, 35, 0, 0, 1]. Using
a BSC channel
p derived from a BPSK system with
p = Q( 2REb /N0 ), we can draw the probability that a received
sequence is taken for a codeword erroneously.

Outline
Basic principles
Linear Block Coding
Pu curves
I R = 4/7 0
10
(15,11)
I Eb /N0 = 6dB, −2
10
(7,4)
p = 0.0165
I Eb /N0 = 10dB, −4
10
p = 3.6 × 10−4 −6
Pu
10
I R = 11/15
−8
I Eb /N0 = 6dB, 10
p = 0.0078 −10
10
I Eb /N0 = 10dB,
p = 6.41 × 10−5 −12
10
0 2 4 6 8 10
SNR(dB)
It means that at 10 dB of Eb /N0 , from 1011 received sequences,

just one codeword error is not detected (it is not related to the bits
but to the codewords).
Outline
Basic principles
Linear Block Coding
Error correction probability for perfect codes
Up to t errors are corrected in a perfect code where

t = b(dmin − 1)/2c. So the probability that a received sequence go
out of Hamming sphere of radius t can be calculated easily. It is in
fact the probability of failure for a correcting receiver.
t
X n
P(F ) = 1 − p j (1 − p)n−j
j
j=0

Outline
Basic principles
Linear Block Coding
Error correction probability for perfect codes

0
10
(15,11)
(7,4)
−1
10
−2
10
−3
10
P(F)
−4
10
−5
10
−6
10
−7
10
0 1 2 3 4 5 6 7 8 9 10
Eb/N0(dB)
Note that this is not the bit error rate probability but word error
rate probability.
Outline
Basic principles
Linear Block Coding
Soft Decision Decoding

√ √
I A BPSK modulation is used to send the codeword (− Ec , + Ec )
I The Hamming distance of two codewords is √related to their
Euclidean distance by the relation: dE = 2 Ec dH
I With a (n, k, dmin ) block code, and an AWGN with σ 2 = N0 /2, a
transmitted codeword will be a point in n-dimensional space.
I Ec and Eb are related by Ec = REb where R = n/k is the code rate.
I Suppose that there are on the average K codewords at the distance
dmin from a codeword, the probability of block decoding (by the
union bound) is
r !
2Rdmin Eb
P(E ) ≈ KQ
N0
I Neglecting K , the asymptotic coding gain in comparison with
uncoded system is Rdmin .
Outline
Basic principles
Linear Block Coding
Modifications to Linear Codes: Extension

An (n + 1, k, d + 1) code can be constructed from an (n, k, d)
code by:
1. Add to the H a last all-zero column
2. Add to the new H a last all-one row
3. Use linear operation to obtain an equivalent systematic code
4. Find the generator matrix G
Exercise: Give the systematic generator matrix of (8, 4, 4) from the
following (7,4,3) code:
 
1 0 0 1 0 1 1
H= 0 1 0 1 1 1 0 
0 0 1 0 1 1 1

Outline
Basic principles
Linear Block Coding
Modifications to Linear Codes: Puncturing
A code is punctured by deleting one column in the H matrix. So a

(n, k) code is transformed into an (n − 1, k) code. The minimum
distance of the code can be decreased by one.

Chapter 5: Linear Block Codes: Vahid Meghdadi

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter 5: Linear Block Codes: Vahid Meghdadi

Uploaded by

Copyright:

Available Formats

Outline

Chapter 5: Linear Block Codes

Vahid Meghdadi Chapter 5: Linear Block Codes

Linear Block Coding

Vahid Meghdadi Chapter 5: Linear Block Codes

I Transmission through noisy channel

Vahid Meghdadi Chapter 5: Linear Block Codes

Block codes (1/3)

Definition of block coding:

Vahid Meghdadi Chapter 5: Linear Block Codes

Block codes (2/3)

I The code space contains 2n points but only 2k of them are

Vahid Meghdadi Chapter 5: Linear Block Codes

Block code (3/3)

Vahid Meghdadi Chapter 5: Linear Block Codes

Example: With the previous example

011 + 101 = 110 ∈ C

011 + 110 = 101 ∈ C

Vahid Meghdadi Chapter 5: Linear Block Codes

Hamming distance and Hamming weight (1/2)

Vahid Meghdadi Chapter 5: Linear Block Codes

Hamming distance and Hamming weight (2/2)

Property 1: d(v, w) + d(w, x) ≥ d(v, x) (triangle inequality)

Vahid Meghdadi Chapter 5: Linear Block Codes

Minimum distance of block code

dmin = min{d(v, w) : v, w ∈ C and v 6= w}

If the code is linear:

dmin = min{w (v + w) : v, w ∈ C and v 6= w}

Theorem: The minimum distance of a linear block code is equal to

Vahid Meghdadi Chapter 5: Linear Block Codes

{00, 01, 10, 11} ⇒ {000, 011, 101, 110}

Vahid Meghdadi Chapter 5: Linear Block Codes

Error detection (1/3)

I Noisy channel introduces an error-pattern: c is sent, r is

Vahid Meghdadi Chapter 5: Linear Block Codes

Error detection (2/3)

I Any error pattern of size at most dmin − 1 can be detected.

Vahid Meghdadi Chapter 5: Linear Block Codes

Error detection (3/3)

I For c + e ∈ C, and e 6= 0, the error cannot be detected.

Vahid Meghdadi Chapter 5: Linear Block Codes

Error Detection Capacity

I The error detection fails only when the error pattern is

Vahid Meghdadi Chapter 5: Linear Block Codes

For this example:

Undetected Error Probability

Definition : Pu (E ) probability of undetected error

In a BSC channel with p as the error transmission probability

Vahid Meghdadi Chapter 5: Linear Block Codes

Considering previous code (7,4)

For example if p = 0.01 then Pu (E ) = 7 × 10−6

Vahid Meghdadi Chapter 5: Linear Block Codes

Error correction 1/2

Error correction 2/2

Vahid Meghdadi Chapter 5: Linear Block Codes

Vahid Meghdadi Chapter 5: Linear Block Codes

Linear Block Coding

Vahid Meghdadi Chapter 5: Linear Block Codes

I Given n and k, how the code can be designed ?

Vahid Meghdadi Chapter 5: Linear Block Codes

I An (n, k) linear code is a k-dimensional subspace of the vector

Vahid Meghdadi Chapter 5: Linear Block Codes