See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/359274746

BERT-fused Model for Finnish-Swedish Translation

Conference Paper · December 2021

DOI: 10.1109/SNAMS53716.2021.9731849

CITATIONS READS
0 77

2 authors, including:

Jouko Vankka
The Finnish Defence Forces
115 PUBLICATIONS 1,428 CITATIONS

SEE PROFILE

All content following this page was uploaded by Jouko Vankka on 20 March 2022.

The user has requested enhancement of the downloaded file.

BERT-fused Model for Finnish-Swedish Translation
Iiro Kumpulainen∗† , Jouko Vankka∗
∗ Department of Military Technology
National Defence University
Helsinki, Finland
jouko.vankka@mil.fi
† Department of Computer Science

University of Helsinki
Helsinki, Finland
iiro.kumpulainen@helsinki.fi

Abstract—Translation between Finnish and Swedish is a com-
mon yet time-consuming and expensive task. In this paper, we
train new neural machine translation models and compare them
with publicly available tools for automatic translation of Finnish
to Swedish. Furthermore, we analyze if fusing BERT models with
traditional Transformer models produces better translations. We
train a base Transformer and a large Transformer model using
Fairseq and compare the results with BERT-fused versions of
the models. Our large transformer model matches the state-of-
the-art performance in Finnish-Swedish translation and slightly
improves the BLEU score from 29.4 to 29.8. In our experiments,
fusing the smaller Transformer model with a pre-trained BERT
improves the quality of the translations. Surprisingly, the larger
Transformer model in contrast does not benefit from being fused
with a BERT model.
Index Terms—machine translation, natural language process-
ing, NMT, NLP, BERT, Finnish, Swedish
I. I NTRODUCTION
Finnish and Swedish are the two official languages of
Finland, and translating between the two languages is nec-
essary in many different contexts. These translation efforts
are expensive and time-consuming, and better tools for au-
tomating the translation process are required. In this paper,
we train new neural machine translation (NMT) models for
Finnish-Swedish translation, and compare them with existing
automatic translation systems. In addition, we analyze if fusing
BERT models with traditional Transformer models produces
better translations in this context.
The BERT model is a state-of-the-art method for a wide
range of natural language processing (NLP) tasks such as
language inference and question answering [1]. On the other
hand, current state-of-the-art neural machine translation mod-
els rely on the Transformer architecture, which use self-
attention mechanisms to compute representations for the input
and output sequences, introduced by Vaswani et al. [2].
Incorporating a language understanding model such as
BERT into neural machine translation seems like a promising
idea, but achieving better results in practice has proven to
be difficult. We follow the approach by Zhu et al. [3] by
training our NMT model for Finnish-Swedish translation,
and fusing it with a pre-trained Finnish language BERT
model by Virtanen et al. [4]. We compare the results with
previous research work, as well as with translation engines
Google Translate, Microsoft Translator, and EU Council Pres-
idency translator1 , which are available online. Our models
and translation results are made publicly available online at
https://github.com/MusserO/BERT-fused fi-sv.
II. R ELATED W ORK
NMT models have previously been trained for Finnish-
Swedish by Tiedemann et al. [5] as a part of a project called
FISKMÖ. The FISKMÖ project aims to provide a large corpus
of parallel data in Finnish and Swedish, and to develop openly
accessible translation services between the two languages. We
follow their work by using the same datasets and replicating
their NMT model to use it as a baseline. We then expand on
their work by training two new NMT models and experiment
with fusing them with a pre-trained Finnish language BERT
model.
Another project with NMT for Finnish-Swedish is OPUS-
MT by Tiedemann and Thottingal [6]. In OPUS-MT, the focus
was not on Finnish-Swedish, but the authors trained NMT
models for over a thousand language pairs. For their Finnish-
Swedish translation model, they used the same parallel dataset
and training framework as in the FISKMÖ project.
While fine-tuning BERT models has been successful in a
wide variety of NLP tasks, there is little work on successfully
incorprating BERT into machine translation. Zhu et al. [3]
managed to achieve state-of-the-art results for a number of
English language translation tasks by introducing BERT-fused
models, where a pre-trained NMT model and a pre-trained
BERT-model are fused together. In this work, we apply this
method for Finnish-Swedish translation by training new NMT
models and fusing them with a Finnish language BERT model
trained by Virtanen et al. [4]. While the experiments by Zhu
et al. [3] suggest that BERT-fused models outperform the
original Transformer models, we show that this is not always
the case for large Transformer models.
1 https://presidencymt.eu/
 III. M ETHODS
Following the example by Tiedemann et al. [5], in order
to train our models, we used data from OPUS, the open
parallel corpus [7]. We used all the available OPUS datasets
for training, except for the Tatoeba dataset, which we used
for validation and testing. Our training dataset consists of
30,877,167 sentences that are both in Finnish and Swedish.
From the Tatoeba dataset, we used 5,000 randomly selected
sentences for validation and the remaining 4,778 sentences for
testing.
In order to keep our training setups comparable with pre-
vious work, we tried to match our training setup with the
training environment in the FISKMÖ project [5]. This way
possible improvements in performance would be caused by
changes in the methodology rather than by a large increase in
computing power. We used four Tesla V100 GPUs for training,
and we first trained a baseline 6-layer Transformer model
with 8 attention heads using Marian NMT framework [8]
with similar hyperparameters as Tiedemann et al. [5]. All the
details about model architecture and hyperparameters that are
necessary for reproducing our results can be found from our
publicly available online repository2 .
Next, in order to get an NMT model that is compatible
with the Finnish language BERT model by Virtanen et al.
[4], we trained a 6-layer Transformer with 8 attention heads
using Fairseq, a sequence modeling toolkit for Python [9]. We
tried to set the hyperparameters to match the model trained
with Marian, but several distinctions remained due to the
differences in the Fairseq and Marian frameworks. After pre-
training the model until convergence, we fused the Fairseq
model with the BERT model using a recently updated version
of the code provided by Zhu et al. [3].
The fused model is then further trained on the same training
data until the new model converges. Finally, we repeat the
experiment with a larger 6-layer Transformer with 16 attention
heads using Fairseq. For this model, we use hyperparameters
following Fairseq recommendations for scaling models [10].
The architecture of a fused Transformer NMT model and
BERT model is shown in Figure 1. In the fused model, inputs
are first fed to the BERT model, which produces an encoded
representation of the input. This representation given by the
last layer of the BERT model is then used in the attention
components in each encoder and decoder layer in the fused
NMT model. Regular Transformer models use self-attention
components, as well as encoder-decoder attention components
in the decoding layers. The BERT-fused model augments these
attention components by adding BERT attention to the self-
attention in the encoding layers and to the encoder-decoder
attention in the decoding layers.
We omit detailed background explanations of the attention Fig. 1. The architecture of a 6-layer Transformer model that has been fused
with a BERT model.
mechanism and the Transformer model architecture to increase
readability. For more details about the Transformer model
architecture, we refer readers to the publication by Vaswani

2 https://github.com/MusserO/BERT-fused fi-sv
et al. [2] and to guides such as ”The Annotated Transformer” 3 .
For a more detailed description of BERT attention and the
method of fusing BERT with an NMT model, we refer to the
publication by Zhu et al. [3].
IV. R ESULTS AND D ISCUSSION
For comparing our models with previous methods and
existing translation engines online, we used the same testing
dataset as FISKMÖ, consisting of 523 parallel sentences
translated by hand [5]. While we reserved a part of the
Tatoeba dataset for testing, these were not used for these
test results and comparisons between previous models because
at least FISKMÖ and OPUS-MT used the Tatoeba data for
training. To automatically evaluate the translation quality, we
use SacreBLEU [11] and chrF [12] scores.
We compare our translation results with online translation
engines Google Translate, Microsoft Translator, and EU Coun-
cil Presidency, as well as the previous works on Finnish-
Swedish translation FISKMÖ and OPUS-MT. The Swedish
output translations for each of the models are available in our
repository for a qualitative analysis of the translations. The
BLEU and chrF scores are shown in Table I.
Translation Engine BLEU chrF
Google Translate 20.8 0.548
Microsoft Translator 20.5 0.547
EU Presidency Translator 29.4 0.626
FISKMÖ 26.6 0.593
OPUS-MT 27.8 0.603
Marian Base (Ours) 27.5 0.602
Fairseq Base (Ours) 20.3 0.535
BERT-fused Base (Ours) 23.4 0.560
Fairseq Large (Ours) 29.8 0.611
BERT-fused Large (Ours) 28.7 0.607
TABLE I
C OMPARISON OF AVAILABLE TRANSLATION SYSTEMS AND MODELS
TRAINED BY US FOR F INNISH -S WEDISH TRANSLATION . B EST SCORES
ARE MARKED IN BOLD .
Google Translate and Microsoft Translator achieve very
similar scores, whereas EU Council Presidency Translator
provides significantly better translations. As expected, our
base Marian Transformer model achieves results similar to
FISKMÖ and OPUS-MT, which also use OPUS dataset and
Marian framework [5, 6]. These methods outperform Google
Translate and Microsoft Translator but fall slightly behind EU
Council Presidency Translator.
Surprisingly, our base Fairseq Transformer model falls
significantly behind the Marian model, despite using same
architecture and our attempts to use similar hyperparameters.
We therefore believe it would be better to use different
hyperparameters for Fairseq models due to the differences in
the frameworks. When training the large Fairseq model, the
hyperparameters were more suitable for Fairseq and resulted
in faster training and state-of-the-art results matching the EU
Council Presidency Translator. Since the hyperparameters for
training the smaller Fairseq model was clearly not optimal,
3 http://nlp.seas.harvard.edu/2018/04/03/attention.html
the training times of our models are not compared here but
approximate training durations be found from our publicly
available repository.
When fusing the base Fairseq model with the Finnish BERT
model, the results clearly improved. However, when fusing
the BERT model with our large Fairseq model, the scores
did not improve, but slightly decreased. These results suggest
that fusing BERT with NMT may be useful in low-resource
scenarios when larger models cannot be trained, but in high-
resource scenarios BERT-fused models are not necessarily
better than the original Transformer models. However, more
experiments are needed to determine how and when to use
BERT-fused NMT models most effectively.
V. C ONCLUSION AND F UTURE W ORK
In this work, we trained neural machine translation models
for translating Finnish into Swedish and compared the results
with previous research and publicly available translation sys-
tems. Our large transformer model matched the state-of-the-
art performance in Finnish-Swedish translation and slightly
improved the BLEU score from 29.4 to 29.8. The model is
made publicly available online.
In addition, we investigated the benefits of fusing a BERT
model with Transformer models in the context of Finnish-
Swedish translation. In our experiments fusing a pre-trained
BERT model was beneficial for a smaller Transformer model,
whereas our large Transformer model did not improve from
being fused with a BERT model. These results agree with
previous work that BERT-fused models lead to better transla-
tions for smaller models. In contrast with the previous work,
our results also show that large Transformer models do not
always benefit from being fused with a BERT model. This
suggests that fusing machine translation models with BERT
models may be useful primarily for language pairs that do not
have enough parallel training data for training large machine
translation models. However, more experiments are needed
before this can be confidently concluded.
Future work may expand our work to different datasets and
language pairs to gain further insight as to when it is beneficial
to use BERT-fused models for NMT. In addition, future work
may analyze if using a larger BERT model that matches the
size of the large Transformer model could improve the results.
Besides using a BERT model, monolingual data from the
target language may be included in the training by using back-
translation, which has been shown to be an effective data
augmentation technique for machine translation tasks [13].
While the BERT model in our experiments is trained us-
ing Finnish sentences, back-translation would use Swedish
sentences by translating them to Finnish using a Swedish-
Finnish translation model that has been trained on a parallel
corpus. These artificially translated sentences could then be
added to the training data of a new model for translating
Finnish to Swedish by training the new model to retrieve
3 Our chrF score seems to slightly differ from that used by Tiedemann
et al. [5]. We used the Python NLTK module version 3.5 implementation of
crhf score with default parameters.
the original Swedish sentences from the artificial translations. [11] M. Post, “A call for clarity in reporting bleu scores,”
Furthermore, the translation results may also be improved in Proceedings of the Third Conference on Machine
by using ensembles of multiple models, which often slightly Translation: Research Papers, 2018, pp. 186–191.
outperform the individual models. [12] M. Popović, “chrf: character n-gram f-score for automatic
mt evaluation,” in Proceedings of the Tenth Workshop on
Statistical Machine Translation, 2015, pp. 392–395.
R EFERENCES [13] S. Edunov, M. Ott, M. Auli, and D. Grangier, “Un-
derstanding back-translation at scale,” in Proceedings of
[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, the 2018 Conference on Empirical Methods in Natural
“Bert: Pre-training of deep bidirectional transformers for Language Processing, 2018, pp. 489–500.
language understanding,” in Proceedings of the 2019
Conference of the North American Chapter of the Asso-
ciation for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), 2019,
pp. 4171–4186.
[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,
“Attention is all you need,” in Advances in neural infor-
mation processing systems, 2017, pp. 5998–6008.
[3] J. Zhu, Y. Xia, L. Wu, D. He, T. Qin, W. Zhou,
H. Li, and T. Liu, “Incorporating bert into neural
machine translation,” in International Conference on
Learning Representations, 2020. [Online]. Available:
https://openreview.net/forum?id=Hyl7ygStwB
[4] A. Virtanen, J. Kanerva, R. Ilo, J. Luoma, J. Luoto-
lahti, T. Salakoski, F. Ginter, and S. Pyysalo, “Multi-
lingual is not enough: Bert for finnish,” arXiv preprint
arXiv:1912.07076, 2019.
[5] J. Tiedemann, T. Nieminen, M. Aulamo, J. Kanerva,
A. Leino, F. Ginter, and N. Papula, “The fiskmö project:
Resources and tools for finnish-swedish machine transla-
tion and cross-linguistic research,” in Proceedings of The
12th Language Resources and Evaluation Conference,
2020, pp. 3808–3815.
[6] J. Tiedemann and S. Thottingal, “Opus-mt–building open
translation services for the world,” in Proceedings of the
22nd Annual Conference of the European Association for
Machine Translation, 2020, pp. 479–480.
[7] J. Tiedemann, “Parallel data, tools and interfaces in
opus,” in Proceedings of the Eighth International Confer-
ence on Language Resources and Evaluation (LREC’12),
2012, pp. 2214–2218.
[8] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak,
H. Hoang, K. Heafield, T. Neckermann, F. Seide, U. Ger-
mann, A. F. Aji, N. Bogoychev et al., “Marian: Fast
neural machine translation in c++,” in Proceedings of
ACL 2018, System Demonstrations, 2018, pp. 116–121.
[9] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng,
D. Grangier, and M. Auli, “fairseq: A fast, extensible
toolkit for sequence modeling,” in Proceedings of the
2019 Conference of the North American Chapter of the
Association for Computational Linguistics (Demonstra-
tions), 2019, pp. 48–53.
[10] M. Ott, S. Edunov, D. Grangier, and M. Auli, “Scaling
neural machine translation,” in Proceedings of the Third
Conference on Machine Translation (WMT), 2018.

View publication stats