# Dr Jacqueline John George

Information Theory and Coding

Linear Block Codes
Lecture 3
Dr Jacqueline John George
Linear Block Code

The generation and detection of error correcting codes
make extensive use of Modulo-2 mathematics:

0 + 0 = 0
0 + 1 = 1
1 + 0 = 1
1 + 1 = 0

…which is effectively the same as Exclusive-OR function

Note: “+” indicates addition, not Logical „OR‟
Dr Jacqueline John George
Naming conventions

Lower case bold to denote a vector:

v = n x 1 vector, i.e. [v
0
, v
1
, v
2
, …, v
n-1
]

Where v
i
= 0 or 1 for 0 ≤ i ≤ n

Upper case bold to denote a matrix:
G is a k x n
matrix:
(
(
(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
(
(
¸
(

¸

=
÷ ÷ ÷ ÷ ÷
÷
÷
÷ 1 1,n k 1,2 k 1,1 k 1,0 k
1 1,n 1,2 1,1 1,0
1 0,n 0,2 0,1 0,0
g . . . g g g
. . . . . . .
. . . . . . .
. . . . . . .
g . . . g g g
g . . . g g g
1 k
1
0
g
.
.
.
g
g
G
Dr Jacqueline John George
Example

Consider the following (7, 4) error correcting code v with
message bits v6, v5, v4, v3 and redundant parity check
bits v2, v1, v0 defined by the following (modulo 2 sums)
equations:

v
2
= v
4
+ v
5
+ v
6
v
1
= v
3
+ v
4
+ v
5
v
0
= v
3
+ v
5
+v
6

Form the codewords, what is the error detection and error
correction
Dr Jacqueline John George
Solution

Messages
Code
words
[v
0
- v
6
]
0000 0000000
1000 1101000
0100 0110100
1100 1011100
0010 1110010
1010 0011010
0110 1000110
1110 0101110

Messages
Code
words
[v
0
- v
6
]
0001 1010001
1001 0111001
0101 1100101
1101 0001101
0011 0100011
1011 1001011
0111 0010111
1111 1111111
Hamming distance = 3
Error detection capability = 2
Error correction capability = 1
Dr Jacqueline John George
Defining a “Linear Block Code”

What defines a linear block code?

A block code is linear if the modulo-2 sum of two code
words is also a code word

Dr Jacqueline John George
Defining a “Linear Block Code”

Messages
Code
words
[v
0
- v
6
]
0000 0000000
1000 1101000
0100 0110100
1100 1011100
0010 1110010
1010 0011010
0110 1000110
1110 0101110

Messages
Code
words
[v
0
- v
6
]
0001 1010001
1001 0111001
0101 1100101
1101 0001101
0011 0100011
1011 1001011
0111 0010111
1111 1111111
1011100
+ 1110010
0101110
Dr Jacqueline John George
Defining a “Linear Block Code”

What defines a linear block code?

A block code is linear if and only if the modulo-2 sum
of two code words is also a code word

What properties do linear block codes possess?

Simple mathematical structure
Simple structure can be exploited in design of real-
world encoders/decoders
Best described using a Generator Matrix (Encoder)
and a Parity check matrix (Decoder)
Dr Jacqueline John George
The Generator Matrix

For an (n, k) linear block code C it is possible to find k linearly
independent code words, g
0
, g
1
, . . ., g
k-1
such that every codeword
v in C is a linear combination of these k codewords. i.e.

v = u
0
g
0
+ u
1
g
1
+ . . . . + u
k-1
g
k-1
where u
i
= 0 or 1 for 0 ≤ i ≤ k

The codewords g
0
- g
k-1
can be represented by a k x n matrix
(
(
(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
(
(
¸
(

¸

=
÷ ÷ ÷ ÷ ÷
÷
÷
÷ 1 1,n k 1,2 k 1,1 k 1,0 k
1 1,n 1,2 1,1 1,0
1 0,n 0,2 0,1 0,0
g . . . g g g
. . . . . . .
. . . . . . .
. . . . . . .
g . . . g g g
g . . . g g g
1 k
1
0
g
.
.
.
g
g
G
G is called the Generator Matrix
n columns
k rows
Dr Jacqueline John George
Using the generator matrix to encode messages

If u = [u
0
, u
1
, . . ., u
k-1
] is the message to be encoded. The
corresponding codeword can be written as:
v = u.G

| |
(
(
(
(
(
(
(
(
¸
(

¸

· =
÷
÷
1 k
1
0
g
.
.
.
g
g
v
1 k 1 0
u . . . u u
v = [ u
0
g
0
+ u
1
g
1
+ u
2
g
2
+ . . . + u
k-1
g
k-1
]
Dr Jacqueline John George
Generator matrix - notes

An (n, k) linear code is completely specified by the k rows
of its generator matrix G
It is not necessary to use all n rows -
Any k linearly independent code words can be used to
form G
Care must be taken to ensure linear independence
(code words must not be multiples of each other)
An encoder for such a code needs only to store the k
rows of the generator matrix G and to form linear
combinations of these k rows, based on the input
message u, to be able to produce a valid codeword v
Dr Jacqueline John George
Example

Given the following generator matrix encode the following message
u = (1 1 0 1)
.
(
(
(
(
¸
(

¸

=
(
(
(
(
¸
(

¸

=
1 0 0 0 1 0 1
0 1 0 0 1 1 1
0 0 1 0 1 1 0
0 0 0 1 0 1 1
3
2
1
0
g
g
g
g
G

If u = [ 1 1 0 1 ] then:
v = u.G
= 1.g
0
+ 1.g
1
+ 0.g
2
+ 1.g
3

= [ 1 1 0 1 0 0 0 ] + [ 0 1 1 0 1 0 0 ] + [ 1 0 1 0 0 0 1 ]
= [ 0 0 0 1 1 0 1 ]

Dr Jacqueline John George
Dr Jacqueline John George
Systematic codes
Dr Jacqueline John George
Systematic codes

Definition: Codes in which the codeword is divided into two
parts:
the message, and
the redundant checking part

For an (n, k) linear systematic code the code consists of k
information (or message) digits and n-k parity check digits
which are linear sums of the information digits.
Message Redundant checking bits
Valid codeword
n - k digits k digits
Dr Jacqueline John George
Simplifying the generator matrix
(
(
(
(
¸
(

¸

=
(
(
(
(
¸
(

¸

=
1 0 0 0 1 0 1
0 1 0 0 1 1 1
0 0 1 0 1 1 0
0 0 0 1 0 1 1
3
2
1
0
g
g
g
g
G
A linear systematic (n,k) code can be defined by a k x n matrix G of the
form:
(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
¸
(

¸

=
÷ ÷ ÷ ÷ ÷ ÷
÷ ÷
÷ ÷
÷
1 . . 0 0 p . . . p p p
. . . . . . . . . . . .
. . . . . . . . . . . .
0 . . 1 0 p . . . p p p
0 . . 0 1 p . . . p p p
1 k 1,n k 1,2 k 1,1 k 1,0 k
1 k 1,n 1,2 1,1 1,0
1 k 0,n 0,2 0,1 0,0
1 k
1
0
g
g
g
G
.
.
If I
k
= k x k unity matrix then G = [ P I
k
]
Dr Jacqueline John George
Simplifying the generator matrix
If I
k
= k x k unity matrix then G = [ P I
k
]
and if u = [u
0
, u
1
, . . . ., u
k-1
]
then v = [v
1
, v
2
, . . . v
n-1
]
= [u
0
, u
1
, . . . u
k-1
] . G

therefore the components of v are:
v
n-k+i
= u
i
for 0 ≤ i ≤ k
and v
j
= u
0
p
0j
+ u
1
p
1j
+ . . . +u
k-1
p
k-1,j
for 0 ≤ j ≤ n-k

The n-k equations for v
j
are called the
parity check equations for the code
Dr Jacqueline John George
Impelmenting the Encoder
Dr Jacqueline John George
Encoding linear block codes – generic form
p
0,0
p
1,0
p
k-1,0

p
0,
n-k-1

p
1,
n-k-1

p
k-1,
n-k-1

p
0,1
p
1,1
p
k-1,1

+ + +
u
0
u
1
u
k-1
u
0
u
1
u
k-1
u
0
u
1
u
k-1
v
0
v
1
v
n-k-1
To
channel

u
0
u
2
u
1
u
k-1
To
channel

Message register

Input u

Dr Jacqueline John George
Encoding circuit for previous example
+ + +
v
0
v
1
v
2
v
3
v
5
v
4
v
6
To
channel

Message register

Input u

v
0
= v
3
+ v
5
+v
6
v
1
= v
3
+ v
4
+v
5
v
2
= v
4
+ v
5
+v
6
[1101]

1
0
0 0
1
1
1 1
1
0 0 0
0 0 1
1
1 1 1 1
Dr Jacqueline John George
Dr Jacqueline John George
The parity check matrix
Dr Jacqueline John George
Parity check matrix
The mathematical properties of the generator matrix G
can be used to develop an alternative definition of a
linear code based on H, the parity check matrix.

For an n x k generator matrix of the form G there exists
an (n-k) x k matrix H such that:

G . H
T
= 0

Furthermore, it can be shown that an n-tuple
v = [v
0
, v
1
, . . . v
n-1
]
is a code word generated by G, if and only if
v . H
T
= 0
Dr Jacqueline John George
How do I generate the Parity Check Matrix?

For the (n x k) generator matrix G of the previous
example this parity check matrix has the form:
H = [I
n-k
P
T
]
(
(
(
(
(
(
¸
(

¸

=
÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷
÷
÷
1 k 1,n k 1 k 2,n 1 k 1,n 1 k 0,n
1,1 k 2,1 1,1 0,1
1,0 k 2,0 1,0 0,0
p . . . p p p . .
. . . . . . . . . . .
. . . . . . . . . . . .
p . . . p p p . .
p . . . p p p . .
1 0 0
.
0 1 0
0 0 1
H
Dr Jacqueline John George
(
(
(
¸
(

¸

=
1 1 1 0 1 0 0
0 1 1 1 0 1 0
1 1 0 1 0 0 1
H
Step 1: For an (n,k) code, generate a blank n-k x k matrix Step 3: Transpose rows of G into columns in H Step 2: Insert a n-k identity matrix in the left hand side of the matrix
How do I generate the Parity Check Matrix?
(
(
(
(
¸
(

¸

=
(
(
(
(
¸
(

¸

=
1 0 0 0 1 0 1
0 1 0 0 1 1 1
0 0 1 0 1 1 0
0 0 0 1 0 1 1
3
2
1
0
g
g
g
g
G
(
(
(
¸
(

¸

=
1 0 0
0 1 0
0 0 1
H
(
(
(
¸
(

¸

= H
Dr Jacqueline John George
Decoding linear
block codes
Dr Jacqueline John George
r = v + e
Parity check matrix – practical implications
G
H
T
Encoder
Decoder
u
v = u.G
û
r is a valid codeword if r.H
T
= 0
v = u.[PI
k
]
e
r
Dr Jacqueline John George
Syndrome

Define a new vector s, such that s = r . H
T

From the definition of the parity check matrix H:

if s = 0 then r is a codeword
if s ≠ 0 then r is not a codeword

s is called the syndrome of the code r
Dr Jacqueline John George
Properties of the syndrome
The syndrome can not just tell us whether or not r is a
codeword. If there is an error, it can also give an
indication of where the error might occur:

s = r . H
T
= (v + e) . H
T
= v . H
T
+ e . H
T

But v.H
T
= 0

Therefore s = e.H
T

The syndrome of r is dependent only on the error
pattern e and not on the codeword v
Dr Jacqueline John George
Using the syndrome to perform error correction
Expanding s = e.H
T
gives n-k linear equations:

s
0
= e
0
+ e
n-k
p
0,0
+ e
n-k+1
p
1,0
+ . . + e
n-1
p
k-1,0
s
1
= e
1
+ e
n-k
p
0,1
+ e
n-k+1
p
1,1
+ . . + e
n-1
p
k-1,1
.
.
s
n-k-1
= e
n-k-1
+ e
n-k
p
0,n-k-1
+ e
n-k+1
p
1,n-k-1
+ . . + e
n-1
p
k-1,n-k-1

The syndrome digits are simply linear combinations of
the error digits and can therefore provide information
that can be used for error correction
Dr Jacqueline John George
Using the syndrome to perform error correction
A solution to these equations shown on the previous slide
provides a value for e, the error vector.

Since e and r are known and v = r + e

It is therefore possible to calculate v

But…
There are 2
k
possible solutions to this set of linear
equations
This means that there are 2
k
error patterns with the same
syndrome
Dr Jacqueline John George
Using the syndrome to perform error correction

The error correction circuit must choose which vector is
the true error vector.

To minimise the probability of a decoding error the most
probable error is used

For a BSC the most probable error is the one with the
least number of non-zero digits - “Maximum Likelihood”

If r contains errors but r.H
T
= 0. The presence of errors is
not detected and a decoding error occurs. Such errors
are called undetectable errors

Dr Jacqueline John George
Dr Jacqueline John George
Example - Error correction using a
syndrome calculation
Dr Jacqueline John George
Syndrome Error Correction Example

If the transmitted code word v= [1 0 0 1 0 1 1]
the received vector r= [1 0 0 1 0 0 1] and
the generator matrix G=

Find the error vector e?

(
(
(
(
¸
(

¸

=
(
(
(
(
¸
(

¸

=
1 0 0 0 1 0 1
0 1 0 0 1 1 1
0 0 1 0 1 1 0
0 0 0 1 0 1 1
3
2
1
0
g
g
g
g
G
Solution
To calculate the syndromes of r
S = r . H
T

Calculating the syndromes of r gives

S = r . H
T
= [1 1 1]
S = e . H
T

The error digits are related to the syndrome digits by:
1 = e
0
+ e
3
+ e
5
+ e
6
1 = e
1
+ e
3
+ e
4
+ e
5
1 = e
2
+ e
4
+ e
5
+ e
6

Dr Jacqueline John George
Dr Jacqueline John George
Syndrome error correction example

There are 2
4
= 16 patterns which satisfy the simultaneous
equations:
[0 0 0 0 0 1 0], [1 0 1 0 0 1 1],
[1 1 0 1 0 1 0], [0 1 1 1 0 1 1],
[0 1 1 0 1 1 0], [1 1 0 0 1 1 1],
[1 0 1 1 1 1 0], [0 0 0 1 1 1 1],
[1 1 1 0 0 0 0], [0 1 0 0 0 0 1],
[0 0 1 1 0 0 0], [1 0 0 1 0 0 1],
[1 0 0 0 1 0 0], [0 0 1 0 1 0 1],
[0 1 0 1 1 0 0], [1 1 1 1 1 0 1],
Applying Maximum Likelihood to a BSC means that we
assume the error is that with the least number of non-
zero elements.
Dr Jacqueline John George
Syndrome error correction example

The error vector e = [0 0 0 0 0 1 0] has the smallest number
of nonzero components, therefore

v* = r + e
= [1 0 0 1 0 0 1]
+ [0 0 0 0 0 1 0]
= [1 0 0 1 0 1 1]

c.f. v = [1 0 0 1 0 1 1]