TRAINING CODE-SWITCHING LANGUAGE MODEL WITH MONOLINGUAL DATA
Shun-Po Chuang, Tzu-Wei Sung, Hung-yi Lee
National Taiwan University
ABSTRACT
Lack of code-switching data is an issue of training code-
switching language model. In this paper, we propose an ap-
proach to train code-switching language models with mono-
lingual data only. By constraining and normalizing output
projection matrix in RNN based language model, we make
the embeddings of different languages close to each other.
With the numerical and visualized results, we show that the
proposed approaches remarkably improve the code-switching
language modeling trained from monolingual data. The pro-
posed approaches are comparable or even better than training
code-switching language model with artificially generated
code-switching data. Furthermore, we use unsupervised
bilingual word translation to analyze if semantically equiva-
lent words in different languages are mapped together.
Index Terms— Code-Switching, Language Model
1. INTRODUCTION
Code-switching is the practice that two or more languages are
used within a document or a sentence and widely observed in
multicultural areas. It related research suffers from data insuf-
ficient issue, and applying prior knowledge [1, 2] or designing
constraints [3, 4] would alleviate such issues. Because it is
easier to collect monolingual data than code-switching data,
to efficiently utilize a large amount of monolingual data
would be a solution to a lack of code-switch data [5]. Recent
work [6] tried to train code-switching language model with
fine-tuning technique. Similar work [7] integrated two mono-
lingual language models by introducing special “switch”
token in both languages for training language model, and
further incorporated within automatic speech recognition
(ASR). Other works have synthesized more code-switching
text by the modeling distribution from the data [8, 9]. Gener-
ative adversarial neural network [10, 11] is also proposed to
learn code-switching point distribution from code-switching
text [12].
In this paper, we propose to utilize some constraints to
make word embeddings of different languages close in the
same latent space, and to normalize each word vector to gen-
erally improve code-switching language model. Similar con-
straints were used in end-to-end ASR [13] before, but has
not yet reported on code-switching language modeling. And
related prior works [14, 15] attempted to initialize the word
978-1-5090-6631-5/20/$31.00 ©2020 IEEE
embedding with unit-normalized vectors in ASR, but without
keeping the possession of unit norm during training. Initial
experiments on code-switching data showed that constrain-
ing and normalizing output projection matrix helped language
model trained with monolingual data better handling code-
switching data.
2. CODE-SWITCHING LANGUAGE MODELING
In our approach, we only use monolingual data for training,
code-switching data is only for validation and testing.
2.1. RNN based Language Model
In this work, we adopt the recurrent neural network (RNN)
based language model [16]. Given a sequence of words
[w1 , w2 , . . . , wT ] we can obtain predictions yi by applying
transformation W on RNN hidden states hi with softmax
computation :
yi = sof tmax(W hi ) (1)
where i = 1, 2, . . . , T and h0 is a zero vector. Specifically, the
output projection matrix is denoted by W ∈ RV ×z , where V
is the vocabulary size and z is the hidden layer size of RNN.
Then using gradient descent to update the parameters with
cross entropy as loss function.
Consider two languages Lang 1 and Lang 2 in code-
switching language modeling, the output projection matrix W
could be partitioned into two parts: W1 and W2 with each row
indicating the latent representations of each word in Lang 1
and Lang 2 respectively. The output projection matrix or em-
beddings can be written as : W = [W1 ; W2 ] , where square
bracket indicates concatenation operation.
2.2. Constraints on Output Projection Matrix
Optimizing language model with Lang 1 and Lang 2 mono-
lingual data, it is possible to improve the perplexity on both
sides. The word embedding distributions would be arbitrary
shaped based on their language characteristics. But without
seeing bilingual word pairs, two distributions may converge
into their own shape without correlating to each other. It is
hard for language model to learn how to switch between lan-
guages. To train language model with only monolingual data,
we assume that embedding overlapping would benefit code-
switching language modeling. To this end, we try to make
7949 ICASSP 2020

Fig. 1: Overview of proposed approach. Square brackets in-
dicate concatenation operation between W1 and W2
word embeddings of Lang 1 and Lang 2, that is W1 and W2 ,
close to each other. We constrain W1 and W2 in the follow-
ing two ways and Fig. 1 shows the overview of our proposed
approach.
2.2.1. Symmetric Kullback–Leibler Divergence
Kullback–Leibler Divergence (KLD) is a well-known mea-
surement on computing distance between distributions. With
minimizing KLD between language distributions, the embed-
ding space would be semantically overlapped. We assume
both W1 and W2 follow a z-dimensional multivariate Gaus-
sian distribution, that is,
W1 ∼ N (µ1 , Σ1 ), W2 ∼ N (µ2 , Σ2 )
where µ1 , µ2 ∈ Rz and Σ1 , Σ2 ∈ Rz×z are the mean vector
and co-variance matrix for W1 and W2 respectively. Based on
the assumption of Gaussian distribution, we can easily com-
pute KLD between W1 and W2 . Due to the asymmetric char-
acteristic of KLD, here we adopt symmetric form of KLD
(SKLD), that is, we use the sum of KLD between W1 and W2
and the one between W2 and W1 , yielding
1
LSKLD = tr(Σ−1 −1
1 Σ 2 + Σ 2 Σ1 )
2 needed monolingual data to train language model) from train
+(µ1 − µ2 )T (Σ−1 −1

1 + Σ2 )(µ1 − µ2 ) − 2z .
2.2.2. Cosine Distance
Cosine distance (CD) is a common measurement for seman-
tic evaluation. With minimizing CD, we assume the semantic
latent space of languages would be closer. Similar to SKLD,
we can compute the mean vector µ1 and µ2 of W1 and W2 re-
spectively, and CD between two mean vectors can be obtained
as follows:
µ1 · µ2
LCD = 1 − , data, we also introduced pseudo code-switching data train-
kµ1 kkµ2 k
where k·k denotes the `2 norm. We hypothesize the latent rep-
resentation of each word in Lang 1 and Lang 2 will distribute
in the same semantic space and will overlap by minimizing
SKLD or CD.
7950
2.3. Output Projection Matrix Normalization
Apart from constraints stated in Subsection 2.2, we propose
to normalize the output projection matrix, that is, each word
representation is divided by its `2 norm to possess unit norm.
Note that normalization is independent of constraints, and can
be applied together.
The rationale behind normalization is explained as fol-
lows: consider two words wj and wk are semantically equiv-
alent, the cosine similarity between their latent representa-
tion vj and vk should be 1, implying the angle of them is
0 or they have the same orientation. By Eq. (1), we ob-
exp(vj ·hi )
serve that the probabilities yi,j = PV exp(v ·h )
and yi,k =
exp(vk ·hi )
m=1 m i
PV are not necessarily equal because the mag-
m=1 exp(vm ·hi )
nitude of vj and vk might not be the same. However, being a
unit vector, normalization guarantees that given the same his-
tory, the probabilities of two semantically equivalent words
generated by language model will be equal. That means,
normalization would be helpful for clustering semantically
equivalent words in the embedding space, which in general
improves language modeling.
3. EXPERIMENTAL SETUP
3.1. Corpus
South East Asia Mandarin-English (SEAME) corpus [17] is
used for the following experiments. It can be simply sepa-
rated into two parts with its literal language. The first part is
monolingual, and it contains pure Mandarin and pure English
transcriptions, which are two main languages in this corpus.
The second part is code-switching sentences, with the syntax
structure specific language, that is, Mandarin or English, and
the transcriptions composed of mixing two of these languages
words.
The original data consists of train, dev man and dev sgn1 .
Each split contains monolingual and code-switching sen-
tences, but dev man and dev sgn are dominated by Mandarin
and English respectively. We held out 1000 Mandarin, 1000
English and all code-switching sentences (because we only
as the validation set. Remaining monolingual sentences were
for the training set. Similar to the prior work [13], we used
dev man and dev sgn for testing, but to balance the ratio of
Mandarin and English, we combined them together as the
testing set.
3.2. Pseudo Code-switching Training Data
In order to compare the performance of constraints and nor-
malization with language model trained by code-switching
ing, in which we use monolingual data to generate artificial
code-switching sentences. Two approaches were used to gen-
erate pseudo code-switching data:
1 https://github.com/zengzp0912/SEAME-dev-set
 • Word substitution: Give only monolingual data, we
randomly replaced the word from monolingual sen-
tences with its corresponding word in another language
to produce code-switching data based on the substitu-
tion probability. However, this requires a mapping of
vocabulary between two languages. To this end, we
used the bilingual translated pair mapping provided by
MUSE [18]2 . Note that not all translated words are in
our vocabulary set.
• Sentence concatenation: By randomly sample differ-
ent lingual sentences from original corpus, we concate-
nated them as a pseudo code-switching sentence then
added it into the original monolingual corpus.
3.3. Evaluation Metrics
Perplexity (PPL) is a common measurement of language
modeling. Lower perplexity indicates higher confidence of
predicted target. To better observe the effects of applying
different techniques mentioned in previous sections, we com-
puted five kinds of perplexity on the corpus: (i) ZH: PPL
of monolingual Mandarin sentences; (ii) EN: PPL of mono-
lingual English sentences; (iii) CS: PPL of code-switching
sentences; (iv) CSP: PPL of code-switching points, which oc-
cur when the language of next word is different from present
one; (v) Overall: PPL of the whole corpus including mono-
lingual and code-switching sentences. Due to the different
meanings of CS and CSP, the perplexity are separately mea-
sured. Obviously, the improvement of CS does not ensure the
the improvement of CSP, because a code-switching sentence
often contains much more non code-switching points, CS
would benefit from improving monolingual perplexity rather
than from CSP.
3.4. Implementation
Because of a small amount of training data, we adopted
only a single recurrent layer with Long Short-Term Memory
(LSTM) cell for language modeling [19]. Hidden size for
both the input projection and LSTM cell was set to 300. We
used a dropout of 0.3 for better generalization, and trained the
models using Adam of an initial learning rate 0.001. In order
to obtain better results, the training procedure was stopped
when the overall perplexity on the validation set did not de-
crease for 10 epochs. All reported results are the average of 3
runs.
4. RESULTS
4.1. Language Modeling
The results are in Table 1 with three sub-tables indexed by
capital alphabet. Each of them represents (A) language model
trained with monolingual data only, (B) word substitution
2 https://github.com/facebookresearch/MUSE
7951
with substitution probability 3 , and (C) sentence concatena-
tion as mentioned in Subsection 3.2; (D) (E) (F) represents the
results after applying normalization mentioned in Section 2.3
on (A) (B) (C) respectively. Baselines in rows (a)(d)(g) repre-
sent the language model was trained without constraints and
normalization. 4 Observing rows (a)(d)(g), we find out that
learning with pseudo code-switching sentences indeed helps
a lot in code-switching perplexity. It is reasonable because
language model had seen some code-switching cases during
training even though the training data is synthetic. However,
compared rows (b)(c) with (d) and (g), we see that with ap-
plying constraints additionally, language model trained with
only monolingual data could be comparable or even better on
both monolingual (column ZH and EN) and code-switching
(column CS and CSP) perplexity than the one trained with
pseudo code-switching data. No matter using monolingual
or pseudo code-switching data for training, normalizing out-
put projection matrix generally improves language modeling.
Even trained with monolingual data only, normalization can
also improve CSP as shown in rows (a) and (j). We concluded
that the monolingual data in our corpus has similar sentence
structure, and normalization resulted in an similar latent
space, aiding in switching between languages. Then with
applying SKLD and normalization together, it improved the
CSP to achieve the best results in monolingual data training
case. Perplexity of code-switching points reduced signifi-
cantly when applying constraints on the output projection
matrix by minimizing SKLD or CD without degrading the
performance on monolingual data. Rows (k)(n)(q) also show
that combining SKLD constraint with normalization tech-
nique got the best performance on each kind of perplexity
over only monolingual and pseudo code-switching data.
(a) Baseline - (b) Baseline – word (c) SKLD -
monolingual substitution monolingual
CSP : 1102.31 CSP : 877.27 CSP : 750.64
Fig. 2: PCA visualization with different training strategies.
Note that the figures are plotted from 1 out of 3 runs.
4.2. Visualization
Besides numerical analysis, we want to know if the overlap-
ping level of embedding space is aligned with the perplexity
3 We performed grid search on substitution probability and 0.2 achieved
the lowest perplexity.
4 5-gram model with smoothing techniques were also experimented, but
got worse performance than our baseline. Due to the limited space, we did
not show it in our experiment.
 Without normalization Metric (A) Mandarin → English (B) English → Mandarin
Approach MRR P@10 MRR P@10
(A) Monolingual only (i) Baseline 0.0274 5.4% 0.0718 20.0%
CS CSP ZH EN Overall (ii) + normalization 0.0554 14.5% 0.0885 23.6%
(iii) SKLD + normalization 0.1024 21.8% 0.1496 30.9%
(a) Baseline 424.80 1118.88 160.40 125.41 289.20
(b) SKLD 319.71 752.03 152.66 115.50 228.79 Table 2: Results for unsupervised bilingual word translation
(c )CD 328.04 778.55 150.78 112.11 231.83 using different approaches, all with monolingual training: the
(B) Pseudo training data - Word substitution translation (A) from Mandarin to English and (B) from En-
(d) Baseline 348.88 884.74 156.90 119.98 246.41 glish to Mandarin.
(e) SKLD 298.24 671.38 157.53 120.36 219.62
(A) Input (B) Baseline (C) SKLD + normalization
(f) CD 296.84 680.19 156.09 117.10 217.56 (i) 你 知道 maybe 你 知道 maybe i think 你 知道 maybe 你 要 去 那边 的 时候 就 会
(you know maybe) (you know maybe i think) (you know maybe when you go there you will)
(C) Pseudo training data - Sentence concatenation (ii) they think 这里 they think 这里 的 时候 我 就 会 去 了 they think 这里 is like a lot of people
(they think here) (when they think here i will go) (they think here is like a lot of people)
(g) Baseline 340.34 831.19 160.21 138.89 248.83
(h) SKLD 289.64 628.09 152.27 126.06 216.39 Table 3: Example generated sentences for different ap-
(i) CD 293.98 652.35 150.76 124.05 217.83 proaches, all with monolingual training: the code-switching
With normalization point is (i) from English to Mandarin, and (ii) from Mandarin
(D) Monolingual only to English. English translation is in the parentheses.
CS CSP ZH EN Overall
(j) Baseline 311.77 754.21 123.28 90.71 212.44 (MRR) is used as an evaluation metric, which is the average
(k) SKLD 277.94 601.58 130.11 96.27 197.15 of the reciprocal ranks, so MRR should be less than 1.0, and
(l) CD 282.24 602.35 132.94 97.86 200.33 the closer to 1.0 the better. The proportion that the correct
(E) Pseudo training data - Word substitution translations are in the top 10 candidate list (r ≤ 10) is also
(m) Baseline 264.93 583.65 131.31 97.50 190.79 reported as “P@10” [20]. In order to mitigate the degrada-
(n) SKLD 248.87 512.27 136.85 101.12 184.14 tion in performance caused by low-frequency words, we se-
(o) CD 251.60 517.85 138.48 101.27 185.84 lected words only with frequency lager than 80, ending up
(F) Pseudo training data - Sentence concatenation
with about 200 vocabularies in Mandarin and English respec-
(p) Baseline 266.11 586.83 123.31 95.82 189.88
(q) SKLD 241.73 490.00 128.75 102.44 179.83
tively, and 55 bilingual pairs used for unsupervised bilingual
(r) CD 247.60 499.41 128.91 103.90 183.49 word translation.
The results of bilingual word translation are in Table 2.
Table 1: ZH, EN, CS, CSP and overall perplexity on the test- We can see the the translation from Mandarin to English (col-
ing set. umn (A)) performed worse on both MRR and P@10 than the
one in the reverse direction (column (B)).
results. We applied Principal Component Analysis (PCA) on Row (i) demonstrates that the baseline without any con-
the output projection matrix, and then visualized the results straint performed not so well, while applying constraints and
on 2-D plane. Fig. 2 shows the visualized results of different normalization in addition, rows (ii) and (iii) show signifi-
approaches. Fig. 2a shows that embeddings of two languages cantly improved MRR score and P@10 compared with row
are linear separable with monolingual data only and without (i). It concludes that applying constraints and normaliza-
applying any proposed approach. With synthesizing pseudo tion for code-switching language modeling can truly enhance
code-switching data for training as shown in Fig. 2b embed- semantic mapping.
dings of two languages are closer than Fig. 2a but do not over-
lap each other too much. As for Fig. 2c, it totally overlap. It 4.4. Sentence Generation
actually meets up with our numerical results shown in Table 1:
We further test the capacity of sentence generation of the lan-
the closer the embeddings are, the lower the perplexity is. 5
guage models trained only with monolingual data. Given par-
tial of a sentence, we used the language models to complete
4.3. Unsupervised Bilingual Word Translation
the sentence. Two generated sentences and their given inputs
To analyze whether the words with equivalent semantics in are shown in Table 3. We can see that our best approach with
different languages are mapped together with the proposed SKLD constraint and normalization, which is listed in column
approaches, we experimented on unsupervised bilingual word (C), could switch language either from English to Mandarin
translation. (row (i)) or from Mandarin to English (row (ii)). However,
Given a word w existing in the same bilingual pair map- the baseline model in column (B) failed to switch code from
ping mentioned in Subsection 3.2, each word in another lan- either side.
guage is ranked according to the cosine similarity of their em- 5. CONCLUSIONS
beddings. If the translated word of w is ranked as r-th candi-
In this work, we trained code-switching language model with
date, then the reciprocal rank is 1r . The mean reciprocal rank
monolingual data by constraining and normalizing the output
5 Due to limited space, we did not show the visualization results of sen- projection matrix. Improved performance was obtained and
tence concatenation/ CD which is quite similar to Fig. 2b/ 2c some analysis of the results are reported in this paper.

7952
 6. REFERENCES
[1] Zhiping Zeng, Haihua Xu, Tze Yuang Chong, Eng-
Siong Chng, and Haizhou Li, “Improving n-gram lan-
guage modeling for code-switching speech recognition,”
in 2017 Asia-Pacific Signal and Information Processing
Association Annual Summit and Conference (APSIPA
ASC). IEEE, 2017, pp. 1596–1601.
[2] H. Adel, N. T. Vu, F. Kraus, T. Schlippe, H. Li, and
T. Schultz, “Recurrent neural network language model-
ing for code switching conversational speech,” in 2013
IEEE International Conference on Acoustics, Speech
and Signal Processing, May 2013, pp. 8411–8415.
[3] Ying Li and Pascale Fung, “Improved mixed language
speech recognition using asymmetric acoustic model
and language model with code-switch inversion con-
straints,” in IEEE International Conference on Acous-
tics, Speech and Signal Processing (ICASSP), 2013.
[4] Ying Li and Pascale Fung, “Language modeling with
functional head constraint for code switching speech
recognition,” in Proceedings of the Conference on
Empirical Methods in Natural Language Processing
(EMNLP), 2014.
[5] Injy Hamed, Mohamed Elmahdy, and Slim Abdennad-
her, “Building a first language model for code-switch
Arabic-English,” Procedia Computer Science, vol. 117,
pp. 208–216, 2017.
[6] Hila Gonen and Yoav Goldberg, “Language modeling
for code-switching: Evaluation, integration of monolin-
gual data, and discriminative training,” arXiv preprint
arXiv:1810.11895, 2018.
[7] Saurabh Garg, Tanmay Parekh, and Preethi Jyothi,
“Dual language models for code switched speech recog-
nition,” arXiv preprint arXiv:1711.01048, 2017.
[8] Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu,
and Pascale Fung, “Learn to code-switch: Data augmen-
tation using copy mechanism on language modeling,”
arXiv preprint arXiv:1810.10254, 2018.
[9] Emre Yilmaz, Henk van den Heuvel, and David A. van
Leeuwen, “Acoustic and textual data augmentation for
improved ASR of code-switching speech,” in the Con-
ference of the International Speech Communication As-
sociation (INTERSPEECH), 2018.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,
Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville, and Yoshua Bengio, “Generative adversar-
ial nets,” in Advances in Neural Information Processing
Systems (NIPS), 2014.
7953
[11] Martin Arjovsky, Soumith Chintala, and Léon Bottou,
“Wasserstein generative adversarial networks,” in In-
ternational Conference on Machine Learning (ICML),
2017.
[12] Ching-Ting Chang, Shun-Po Chuang, and Hung-Yi Lee,
“Code-switching sentence generation by generative ad-
versarial networks and its application to data augmenta-
tion,” arXiv preprint arXiv:1811.02356, 2018.
[13] Yerbolat Khassanov, Haihua Xu, Van Tung Pham, Zhip-
ing Zeng, Eng Siong Chng, Chongjia Ni, and Bin Ma,
“Constrained output embeddings for end-to-end code-
switching speech recognition with only monolingual
data,” arXiv preprint arXiv:1904.03802, 2019.
[14] Kartik Audhkhasi, Bhuvana Ramabhadran, George
Saon, Michael Picheny, and David Nahamoo, “Direct
acoustics-to-word models for English conversational
speech recognition,” arXiv preprint arXiv:1703.07754,
2017.
[15] Shane Settle, Kartik Audhkhasi, Karen Livescu, and
Michael Picheny, “Acoustically grounded word embed-
dings for improved acoustics-to-word speech recogni-
tion,” in IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2019.
[16] Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cer-
nocký, and Sanjeev Khudanpur, “Recurrent neural net-
work based language model,” in the Conference of the
International Speech Communication Association (IN-
TERSPEECH), 2010.
[17] Dau-Cheng Lyu, Tien Ping Tan, Chng Eng Siong, and
Haizhou Li, “SEAME: a Mandarin-English code-
switching speech corpus in south-east Asia,” in the Con-
ference of the International Speech Communication As-
sociation (INTERSPEECH), 2010.
[18] Guillaume Lample, Alexis Conneau, Marc’Aurelio
Ranzato, Ludovic Denoyer, and Hervé Jégou, “Word
translation without parallel data,” in International Con-
ference on Learning Representations (ICLR), 2018.
[19] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney,
“LSTM neural networks for language modeling,” in the
Conference of the International Speech Communication
Association (INTERSPEECH), 2012.
[20] Chao Xing, Dong Wang, Chao Liu, and Yiye Lin, “Nor-
malized word embedding and orthogonal transform for
bilingual word translation,” in the Conference of the
North American Chapter of the Association for Com-
putational Linguistics: Human Language Technologies
(NAACL-HLT), 2015.