NEAR SHANNON LIMIT ERROR - CORRECTING ‘CODING AND DECODING : TURBO-CODES (1) Claude Berrou, Alain Glavieux and Punya Thitimajshima Claude Berrou, Integrated Circuits for Telecommunication Laboratory ‘Alain Glavieux and Punya Thitimajshima, Digital Communication Laboratory Ecole Nationale Supérieure des Télécommunications de Bretagne, France (1) Patents N® 9105279 (France), N° 92460011.7 (Europe), N° 07/870,483 (USA) ‘Abstract - This paper deals witha new class of convolutional codes called Turbo-codes, whose performances in terms of Bit Error Rate (BER) are close tothe SHANNON limit, The Turbo-Code encoder is built using a parallel concatenation of two Recursive Systematic Convolutional codes and the associated decoder, using a feedback decoding rule, is implemented as P pipelined identical elementary decoders, T-intRopuction Consider a binary fate ReL2 convoluonal encoder with contin length K and memory M=K-1. The input to the encode atime is abit andthe comesponding codeword Ck isthe binary coupe Ye) with X= Beh, mad? gy Sends, mod.2 g,=01 (1b) where Gi: (gu), G2: (g2¢} are the two encoder generators, generally expressed in octal form. Tis well known, that the BER of a classical Non ‘Systematic Convolutional (NSC) code is lower than that of classical Systematic code with the sume memory M at large SNR, At low SNR, itis in general the other way round. The new class of Recursive Systematic Convolutional (RSC) codes, proposed in this paper, can be better dhan the best NSC code al any SNR for high code rates. ‘A binary rate R=1/2 RSC code is obtained from a NSC code by using a feedback loop and seting one of the {wo outputs Xj or Yq equal to the input bit ds. For an RSC code, the sift resister (memory) inpot is no longer the bit de but is a new binary variable ap. If Xe=dy (respectively Ye=dy), the output ¥ (resp. Xp) is equal to equation (1) (esp. 1a) by substituting ay for dk and the variable axis recursively calculated as aad + St, 2 ‘where 7 is respectively equal to gy; if Xi=dy and to gp; if Yd Equation (2) can be rewrite as = "Erm, mod2. (3) One RSC encoder with memory Mf=4 obtained from aa NSC encoder defined by generators G1=37, Gy=21 is depicted in Fig (ia) % mod.2 Generally, we assume that the input bit de takes values 0 oF I with the same probability. From equation 2), we can show that variable ay exhibits the same statistical Property 0-7803-0850-295/53.0001991BEE Bfan =0/0, = 6.044 with es equal e='Fre, mod? 2-04. (5) ‘Ths the welisstucture is Hentiel for the RSC cove and the NSC: code and these two codes have the sme ite distanced, However, he two oupot sequences (4) and (%4) donot erespond othe same iapet sequence (dy) for RSC and NSC cos. Tass the main ference between the two ces, ‘When punctured code is considered, some output bits Xe or Fee deleted according to a chosen punctrag pattem defined by a marx P. Fo insane, staring from fae Rei? coe, the mix P of rate 23 punctured code is 11 10, eva} = P-{dy =e} =1/2 (4) Pp aad Fig. 1a Classical Non Systematic code & Fig. 1b Recursive Systematic code. 1068PARALLEL CONCATENATION OF RSC CODES With RSC codes, a new concatenation scheme, called parallel concatenation can be used. In Fig. 2, an example of two identical RSC codes. with parallel concatenation is shown, Both elementary encoder (Cy and C2) inputs use the same bit de but according to a different sequence due to the presence of an interleaver. For an input Dit sequence (dg), encoder outputs Xx and Ye atime k are respectively equal to dy (systematic encoder) and to encoder Cy ouput ¥ i, oF to encoder Cz output Ya. If the coded outputs (Yk, ¥ox) of encoders Cy and Cz are used respectively m times and a times and soon, the encoder Cy, rate Ry and encoder Cy rate Ra are equal to siven encoder (Cj or C2) is not emitted, the coresponding ‘decoder input is set to zero. This is performed by the DEMUX/INSERTION block. It is well known that soft decoding is bewer than hard decoding, therefore the first decoder DEC must deliver to the second decoder DEC a weighted (soft) decision. The Logarithm of Likelinood Ratio (LLR), Ay(d, ) associated with each decoded bit dy by the frst decoder DEC) is a relevant pice of information for the second decoder DEC} where Pr{dy =i (observation), i = 0, 1 is the a posteriori ‘probability (APP) ofthe data bit de. Repth ap Bth 6) x = — Aix) Ax(@n) dna % > | I \ % pesrerey Gwe | i [eeseere) f G X DEC, an DEC, ered Lateney Ly ateney Lg uo 2) “le Ye % igh DEMUX IRRERTION mar Yu Si T Y, Seeley a 1S osha oe 731) Fig.2 Recursive Systematic codes with paral concatenation, The decoder DEC depicted in Fig. 3a, is made up of wo elementary decoders (DEC; and DEC) in serial concatenation scheme. The fst elementary decodet DEC is associated with the lower rate Ry encoder Cy and yields a soft (weighted) decision. The error bursts at the decoder DEC; output are scattered by the interieaver and the encoder delay Ly is inserted to take the decoder DEC delay into account. Parallel concatenation is a very attractive scheme because both elementary encoder and decoder use a single frequency clock. For a discrete memoryless gaussian channel and a binary modulation, the decoder DEC input is made up of a couple Ry of two random variables xg and yg, at time k = (2dy-V 4h (Ta) M4 = 2% -V+q, (70) where ik and gy are two independent noises with the same variance 0, The redundant information yx is demultiplexed ‘and sent to decoder DEC; when Ye =¥;, and toward decoder DEC; when Y_=¥ax. When the redundant information of a ns Fig. da Principle ofthe decoder aocoraing to 8 serial concatenation achome II. OPTIMAL DECODING OF RSC CODES WITH WEIGHTED DECISION ‘The VITERBI algorithm is an optimal decoding method which minimizes the probability of sequence error for convolutional codes. Unforwnately ths algorithm is not able to yield the APP for each decoded bit. A relevant algorithm for this purpose has been proposed by BAHL ea. (1). This algoritam minimizes the bit error probability in 0 &=0 Ff Ald)<0. U4) Im order to compute the probability 25(m), let ws introduce the probity funtion a(n), B(m) and (Rm) Ady ayy Ply =iSy =m : aim) = phn fd, =i.S, =m/et} 15) Be] { } PRN /S, =m} im 16) Pulm 8 Ta ie Ream) = Poly =i Ry Sy = M/S. =m). 1) “Te joint probability A4(m) canbe rewriten using BAYES rule Prfde = iS =m, RE Ren Balm — RE Rh, (18) ‘Thos we obtain p {dy =i Se Aim) = fd = iS, = mkt} {RE dy =i.S =m PRR, ale} a» ‘Taking into account that events after time k afe not influenced by observation RF and bit de if state Seis known, the probability 2(m) is equal Ay(m) = aj(m)p(m). (20) ‘The probabilities aj(m) and P,(m) can be recursively calculated from probability 7,(R,,m’,m). From annex I, we obiain EER’, miay rm’) im) =i ay EEE Lyi(Rm’smal.s(m’) ad EER Beale’) TELE Ram’ mialim’) ‘The probability 7)(R,,m’m) can be determined from Uwansition probabilities of the discrete gaussian memoryless Balm) 22) 1066 channel and transition probabilities of the encoder trellis, From relation (17), 7,(R,,m’,m) is given by ) Wily.’ m) = plRy Fd =.Sy = mS. dy = 1/5, = m,S,., = m')A(S, = M/S.) =m") (23) where p(/.) is the transition probability of the discrete gaussian memoryless channel. Conditionally to cn Skea =m), X4 and yg are (wo uncorrelated ‘gaussian variables and thus we obta PAR Sd, 1S, = mSy4 =m’) Play Hd =i,5, =m S.4 =m’) ia ldy =i.S, = mS... =m) (24) Since the convolutional encoder i a deterministic machine, ld, =i/5y = mS =m") is equal 10 0 oF 1. The transition state probabilities (5, = m/S,., =m’) of the trellis are defined by the encoder input statistic. Generally. P,{d, =1} = P.{d, =0}=1/2 and since there are two possible transitions from each sate, x(S, =m/S,., =m)=1/2 for each of these Diterent stepsof modified BAHT. eta algorithm “Siep 0: Probabilities am) and ym) are inaliaed according elton (12) a5(0)=1 ajim) =0 Ym 20 (25a) Bu(0)=1 By(m)=0 vm 20. (250) “Step 1: For each observation Re the probabiliies aai(m) and 7,(Rem.m) se computed using rations 21) and (23) respectively “Step 2: When the soquence RY has been completely received, probabil Bm) ae computed sing relation (22), and probabilities a(m) and. By(m) are multiplied in order to obtain Aj(m). Finally the LLR associated with each decoded bit dis computed from relation (13). Mt IV. THE EXTRINSIC INFORMATION OF THE RSC DECODER In this chapter, we will show that the LR A(dy) associated with each decoded bit dk isthe sum of the LL. ‘of dat the decoder input and of another information called ‘extrinsic information, generated by the decode. Using the LLR (a) definition (13) and relations (20) and 21, we obtain EE Ey (Resm’smay_(m’)Bylm) top Bel EEL yo(Rem’mial_(m’/By(m) Since the encoder is systematic (Xp = dp, the wansition probability pry /dy = i,5y = m.S,.1 =m’) in expression (Ry, m'sm) is independent of state values Sy and Sp. ‘Therefore we can factorize this transition probability in the ‘numerator and inthe denominator of relation 26) Ad (26)* Ave) Arlen) i 1eSTATE inter igerare, leaving Decooer ; X DEC: leaving DEC2 emery Ne (Ax) e oavi Yak Yox ae 3! v ew INSERTION decoded output Fig, 3b Feedback decoder (under 0 internal deley assumption) 4 EE En Oum’smal sim iBalm) Lyte EE Ero(neomsmad lm’ )Bxlm) en Conditionally to dk=1 (sesp. de=0), variables x4 are gaussian with mean 1 (resp. -1) and variance 62, thus the LLR A (dg) is still equal to. Ad)= Fem + (8) where We = Ald bye EE En(owm’smal (mgt) EE Erle’ smlad lm Balm) We is function ofthe redundant information introduced by the encoder. In general W, has the same sign as dj; therefore We may improve the LLR associated with each decoded data bit dy. This quantity represents the extrinsic information supplied by the decoder and does not depend on decoder input ze. This property will be used for decoding the two parallel concatenated encoders. Log (29) Y - DECODING SCHEME OF PARALLEL ‘CONCATENATION CODES In the decoding scheme represented in Fig. 3a, decoder DEC} computes LLR A1(@p for each transmitted bit dy fom sequences {x4} and (y4), then the decoder DEC) performs the decoding of sequence(dg} from sequences {(A1(d)} and (yx). Decoder DEC) uses the modified BAHL tal, algorithm and decoder DEC may use the VITERBI algorithm. Tae global decoding rue isnot optimal because the first decoder uses only a fraction of the available redundant information. Therefore itis possible to improve the performance of ths serial decoder by using a feedback loop. 1067 "Vel Decoding witha feedback loop We consider now that both decoders DEC) and DEC} use the modified BAHL et al. algorithm. We have seen in section IV thatthe LLR atthe decoder ourput can be ‘expressed asa sum of two tems if the decoder inputs were independent, Hence ifthe decoder DEC3 inputs Ay(dy) and ‘Yak are independent, the LLR Ag(dy) atthe decoder DEC, ‘ouput can be written as Agld) = HA dd) + Woy 0) with Ad) = 2p +H ou (dy) = + Me From relation (29), we ean see that the decoder DEC2 extrinsic information Wag is a function of the sequence (AtGn! nate Since A. (dq) depends on observation’, extrinsic information Wg is correlated with observations x. ‘and yi. Nevertheless from relation (29), the greater |n-t lis, the less correlated are A (dy) and observations x ye Thus, due to he presence of interleaving between decoders DEC and DEC, extrinsic information We and observations x5 yux are weakly correlated, Therefore extrinsic information Wag and observations x, yx canbe joy used for caying ‘ut a new decoding of bit dy. the extrinsic information 2e= Woe acting asa diversity effect in an iterative process. In Fig. 3, we have depicted a new decoding scheme using the extinsic information Wax generated by decoder DEC) in a feedback loop. This decoder does not take into account the different delays introduced by decoder DEC and DEC and a more realistic decoding suvetre will be resented later ‘The first decoder DECy now has thre data inputs, Gm, yu. 20 and probabilities ajy(m) and) ase computed in substituting Rk =(2u,yi4) by RE=Cu Yb 3) in relations (21) and (22). Taking into account that zis weakly comelated with x: and yi and supposing that ze can be approximated by a gaussian variable with vasance @2 #0, ‘the transition probability of te discrete gaussian memoryless ‘channel canbe now factored in three termsPUR, /dy = iS = Spo PAA IY! )PLza/.) (32) The encoder Cy with nial rat Ry, trough the feedback Toop is ow equivalent toa rae encoder with R =f 5 aap ow ‘The first decoder obtains an onal redundant infomation with that may significantly improves performances: the term Turbo-codes is given fr this erative decoder scheme tit reference ote trbo engine pane With the feedback decor, the LLR Ay a) generated by decoder DEC; now equal (0 Ad)= Free My OH whore Wie depends on sequence {zalpuy AS indicated above, information x hasbeen bul by decoder DEC at the previous docoting scp, Therefore ze mist not be osed a8 input information for decoder DECg, Thus decodce DEC inpot sequences at step p (p22) will be sequences (Ald, )fand (924) wit A\(d,) = Ay (dy), 2 35) Finally from relation (30), decoder DEC, extrinsic information 24 = Wax, afer deinerleavng, can be writen as Whe = AA aye 06) and the decision atthe decoder DEC outpuis a= sienfA(d)). BT) The decoding delays introduced by decoder DEC (DEC=DEC;+DECY) the inerleaver and the dinerleaver ‘imply that he feedback information 2 mus be used trough an trative process as represented i Fig. 4, 4b. In fat, he flobal decoder cireuitis composed of P pipelined Wenticl elementary decoders (Fig. 4a). The pth decoder DEC (Fig. 4) inp, is made up of demodulator ouput sequences (p and (ptrough a delay ie and of existe information (@p generated by the @-1)h decoder DEC. Note that he variance o? of te extinsc information and the variance of ‘A(d) tus be estimated at each decoding step p 'V-2 Interleaving ‘The interleaver uses a square matrix and bits (4) are written row by row and read pseudo-randomly. This non uniform reading rule is able t© spread the residual error blocks of rectangular form, that may set up in te interleaver located behind the first decoder DEC}, and 10 give the sreater fre distance as possible to the concatenated (parallel) ode, VI- RESULTS For arate R=1/2 encoder with constraint length K=5, generators G1=37, Gz=21 and parallel concatenation (R=R2=2/3), we have computed the Bit Error Rate (BER) after each decoding step using the Monte Carlo method, as a function of signal to noise ratio Ep/No where Ep is the energy received per information bit de and No isthe noise smonolateral power spectral density. The interleaver consists fof a 256x256 matrix and the modified BAHL. eta. algorithm thas heen used with length data block of N=65536 bits In 1068 2 — 0h) hr (iy uc | 6 one “Gp rou From Me | * [Oh]? [Oe Obed)? demodulator y decoded output MP Fig. 4a Modular pipelined decoder, coresponding o an erative processus ofthe feedback decoding % pene eee tb =o eLS ow, a) wine —,, v 4, = Fo. Dosing med fave). tat coder to evaluate @ BER equal to 10%, we have considered 128 data blocks ie. approximatively 8 x10® bits dy The BER versus E,/No, fr diferent valves of pis ploted in Fig. 5. For any given signal to noise ratio greater than 0 dB, the BER decreases a8 a function of the decoding sep p. The coding ‘gain is fairly high forthe fist values of p (p.2,3) and Carieson increasing forthe subsequent values ofp. Fo pl8 for instance, the BER is lower than 10° at Ey/No= 0,7 dB. Remeber thatthe Shannon limit fora binary modulation With Re1/2, is Pe= O (several auhors take Pe=10°5 as a reference) for Ey/No= 0 dB. With parle} concatenation of RSC convolutional codes and feedback decoding, the performances are at 0,7 dB from Shannoo's limit “The influence of the constraint length on the BER has also been examined. For K greater than 5, at E,/No= 0.7 OB, the BER is slightly worst atthe fist (p=1) decoding step and the feedback decoding is inefficient 19 lmprove the final BER. For K smaller than 5, at F,/No= 0.7 OB, the BER is slightly beter at the first decoding step an for K equal 10 5, but the eoretion capacity of encoders Cy and C3 i too weak to improve the BER with feedback decoding. For Ked (ie. 8-stte clementary decoder) an after iteration 18, BER of 105i achieved at £p/Np = 0,9 dB. Por K equa o 5, we have tested several generators (Gi, Gp ) and the best results were achieved With Gy=37, Gy=2.— Bins a ae 107 1 Exe | | 1‘ | | = t wo" cater cc I 1 ae a err Geotcal oe Fry oot te hanya adr pt 8) eric ar arg ees For low signal 10 noise Fitios, we have sometimes nouced that BER could increase during the iterative decoding process. In onder to overcome this effect, we have divided the enttinsic information 2 by [ 1+ fa (dy with 8= 015, In Fig. 6, the histogram of extrinsic information (2p thas been drawn for several values of iteration p, with all data bits equal to 1 and for a low signal to noise ratio (Ep/No= 0.8 6B). For p=1 (first iteration), extrinsic information (pis very poor about bit da, furthermore the ‘gaussian hypothesis made above for extrinsic information ‘G)p, isnot Satisfied! Nevertheless when iteration p increases, the histogram merges towards a gaussian law with a mean equal to I. For instance, for p=13, extrinsic information (2)p ‘becomes relevant information concerming data bis 1069 VIICONCLUSION In this paper, we have presented a new class of ‘convolutional codes called Turbo-codes whose performances in tems of BER are very close to SHANNON'S limit. The decoder is made up of P pipelined identical elementary ‘modules and rank p elementary module uses the data information coming from the demodulator and the extrinsic Information generated by the rank (p-1) module, Each elementary module uses a modified BAHL et al. algorithm which is rather complex. A much simpler algorithm yielding Weighted (Soft) decisions has also been investigated for Turbo-codes decoding (2), whose complexity is only twice the complexity of the VITERBI algorithm, and with Performances which are very close to those of the BAHL et aL. algorithm, This new algorithm will enable encoders and‘decoders to be integrated in silicon with error correcting performances unmatched at the present ime. | | . Bal I wine) ma] om | es Fig. 6 Histograms of extinsc information 2 ater heratone # 1.4.13 at EbNo = 0.8 dB; allieformatin bits dt ‘ANNEX I; EVALUATION OF PROBABILITIES 0m) AND B,(m) From relation (15) probability f/m) is equal to Pr{d, =i 5, =m. RR, sim ai Pr, Pr{dy =i.S, =m, Ry/RE PRR ‘The numerator of jm) can be expressed from sate Sk. and bit Pr{dy =i,Sp=m,Ry/RE*} EER tebe = 15. =m Ss By using BAYES mule, we can write Prf{dy =i.S, =. Ry/ RE} ay PR /RE} (42) Bd, #15, =m Rd LR}. (49) BY taking into account that evens ater te (-1) are not influenced by observation Rt“ and bit dj.1 if state Se.1 is kun ad from lation (17) we obtain Prd, 1S, = mR / RE EE Rem masa (a4) “Te denominator can be aso expreste fom bit de and sate be nfs} Prfdy iS =mR IR} (AS) and from relation (Ad), we can write: 1070 PrfR IR} EEE Ey Rem mah im) (66) Finally probability (m) can be expressed from probability de (m) by the following relation EL 7(Rym’ mam) ah(m)=—2 ize cary EEE LR, m' mal _,(m') In the same way, probability Bx (m) can be recursively calulted trom probability ey (From relation (1), we have 5 Pen (Ss =m) P= TREY EER dean =6Sie =I’ Ra Reng /Sy =m} a 7 (a8) PR /R T By using BAYES rule, tbe numerator is equal 19 PRS EP, {Ra/Sn=m'} Fefdees =irSeas = Rass /Se= mj. (A9) By taking into account expressions of ¥(Ree mm and Brot (im), we can write EZ Rev mm Brom’) In substituting k by (k+/) in relation (A6), the denominator of (A10) sequal © Prf/Rt} EE £ Finally probability Bx (mean be expressed from probability Bos (m, by the following relation EEN RMA Beal) (a10) Symmes an Bytm) (any EEE Ly (Raomimalim) REFERENCES, [1] LR. Bahl, J. Cocke, F. Jeinek and J, Raviv, “Optimal decoding of linear codes for minimizing symbol ror rte", IEEE Trans. Inform. Theory, vol. 17-20, pp. 248- 287, March 1974, 21, Berrou, P, Adde, E. Angui and S, Faudeil, “A Tow complexity sofi-output Viterbi decoder architecture” 10