See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/367359399

Is ChatGPT A Good Translator? A Preliminary Study

Preprint · January 2023

CITATIONS READS

0 1,348

5 authors, including:

Wenxiang Jiao Wenxuan Wang

Tencent Nanjing Agricultural University
39 PUBLICATIONS 222 CITATIONS 17 PUBLICATIONS 56 CITATIONS

SEE PROFILE SEE PROFILE

Xing Wang Zhaopeng Tu

Beijing children's hospital of Capital Medical University Huawei Technologies
57 PUBLICATIONS 864 CITATIONS 142 PUBLICATIONS 4,251 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Wenxiang Jiao on 19 March 2023.

The user has requested enhancement of the downloaded file.

 Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine

Wenxiang Jiao∗ Wenxuan Wang Jen-tse Huang Xing Wang Zhaopeng Tu

Tencent AI Lab

Abstract
This report provides a preliminary evaluation
of ChatGPT for machine translation, includ-
arXiv:submit/4797616 [cs.CL] 19 Mar 2023

ing translation prompt, multilingual transla-

tion, and translation robustness. We adopt
the prompts advised by ChatGPT to trigger
its translation ability and find that the candi-
date prompts generally work well and show
minor performance differences. By evalu-
ating on a number of benchmark test sets1 ,
we find that ChatGPT performs competitively Figure 1: Prompts advised by ChatGPT for machine
with commercial translation products (e.g., translation (Date: 2022.12.16).
Google Translate) on high-resource European
languages but lags behind significantly on low-
resource or distant languages. For distant It integrates various abilities of natural language
languages, we explore an interesting strategy processing, including question answering, story-
named pivot prompting that asks ChatGPT telling, logic reasoning, code debugging, machine
to translate the source sentence into a high- translation, and so on. We are particularly inter-
resource pivot language before into the target ested in how ChatGPT performs for machine trans-
language, which improves the translation per- lation tasks, especially the gap between ChatGPT
formance significantly. As for the translation
and commercial translation products (e.g., Google
robustness, ChatGPT does not perform as well
as the commercial systems on biomedical ab- Translate, DeepL Translate).
stracts or Reddit comments but exhibits good In this report, we provide a preliminary study of
results on spoken language. With the launch ChatGPT on machine translation to gain a better
of the GPT-4 engine, the translation perfor- understanding of it. Specifically, we focus on three
mance of ChatGPT is significantly boosted, be- aspects:
coming comparable to commercial translation
products, even for distant languages. In other • Translation Prompt: ChatGPT is essentially a
words, ChatGPT has already become a good
large language model, which needs prompts as
translator.
guidance to trigger its translation ability. The
1 Introduction style of prompts may affect the quality of trans-
ChatGPT2 is an intelligent chatting machine devel- lation outputs. For example, how to mention
oped by OpenAI upon the InstructGPT (Ouyang the source or target language information mat-
et al., 2022), which is trained to follow an instruc- ters in multilingual machine translation models,
tion in a prompt and provide a detailed response. which is usually solved by attaching language
According to the official statement, ChatGPT is tokens (Johnson et al., 2017; Fan et al., 2021).
able to answer followup questions, admit its mis-
• Multilingual Translation: ChatGPT is a single
takes, challenge incorrect premises, and reject in-
model handling various NLP tasks and covering
appropriate requests due to the dialogue format.
∗
different languages, which can be considered a
Correspondence: joelwxjiao@tencent.com
1 unified multilingual machine translation model.
Scripts and data: https://github.com/wxjiao/
Is-ChatGPT-A-Good-Translator Thus, we are curious about how ChatGPT per-
2
https://chat.openai.com forms on different language pairs considering
 Table 1: Information of adopted test sets. Table 3: Comparison of different prompts for ChatGPT
to perform Chinese-to-English (Zh⇒En) translation.
Test Set Direction Domain Size
System BLEU↑ ChrF++↑ TER↓
Flores-101 Any General 1012
WMT19 Bio De⇒En Biomedical 373 Google 31.66 57.09 56.21
En⇒Ja 1376 DeepL 31.22 56.74 57.84
WMT20 Rob2 Reddit Tencent 29.69 56.24 57.16
Ja⇒En 997
WMT20 Rob3 De⇒En Common Voice 5609 ChatGPT w/ T P 1 23.25 53.07 66.03
ChatGPT w/ T P 2 24.54 53.05 63.79
Table 2: Candidate translation prompts. ChatGPT w/ T P 3 24.73 53.71 62.84

Translation Prompt
T P 1 Translate these sentences from
[SRC] to [TGT]:
T P 2 Answer with no quotes. What do
these sentences mean in [TGT]?
T P 3 Please provide the [TGT]
translation for these sentences:
both the resource difference (e.g., high vs. low)
and language family (e.g., European vs. Asian).
• Translation Robustness: ChatGPT is devel-
oped upon GPT3, which was trained on large-
scale datasets that cover various domains. There-
fore, we wonder if it can perform robustly well
on domain-specific or even noisy sentences.
To trigger the translation ability of ChatGPT,
we ask ChatGPT itself for advice and obtain three
candidate translation prompts. By evaluating on
the Chinese⇒English translation task, we find that
the candidate prompts generally work well and
show minor performance differences. Neverthe-
less, we adopt the best-performing prompt for the
rest parts of the study. By evaluating the transla-
tion among four selected languages on the Flores-
101 test sets, we find that ChatGPT performs com-
petitively with commercial translation products
(e.g., Google Translate) on high-resource Euro-
pean languages but lags behind significantly on
low-resource or distant languages. For distant lan-
guages, we explore an interesting strategy named
pivot prompting that asks ChatGPT to translate the
source sentence into a high-resource pivot language
before into the target language, which improves the
translation performance significantly. As for the
translation robustness, results on three robustness
sets suggest that ChatGPT does not perform as well
as the commercial systems on biomedical abstracts
or Reddit comments but exhibits good results on
spoken language.
With the launch of GPT-4 (OpenAI, 2023)3 on
March 15, 2023, we re-evaluate the translation abil-
ity of ChatGPT and observe a significant boost of
performance, becoming comparable to commercial
translation products, even for distant languages. To
this end, we can conclude that ChatGPT has al-
ready become a good translator, with GPT-4 as
the engine!
2 ChatGPT for Machine Translation
2.1 Evaluation Setting
We provide a brief introduction of the evaluation
setting, which mainly includes the compared base-
lines and test data.
Baselines. We compare ChatGPT with three
commercial translation products, namely, Google
Translate4 , DeepL Translate5 , and Tencent TranS-
mart6 . So far, the three commercial systems sup-
port translation in 133, 29, and 16 languages, re-
spectively. By default, the results in this report
come from the ChatGPT version on 2022.12.16.
For new results, we will mark the updated version
information correspondingly.
Data. For multilingual translation, we evalu-
ate the above translation systems on the Flores-
101 (Goyal et al., 2021)7 test sets, which consists
of 1012 sentences translated into 101 languages.
To test the translation robustness, we adopt the test
set of WMT19 Biomedical Translation Task (Baw-
den et al., 2019, i.e., Bio) and the set2 and set3 of
WMT20 Robustness Task (Specia et al., 2020, i.e.,
Rob2 and Rob3). We obtain the first two test sets
through SacreBLEU and the third pre-processed by
3
https://openai.com/research/gpt-4
4
5
https://translate.google.com
https://www.deepl.com/translator
6
https://transmart.qq.com/zh-CN/index
7
https://github.com/facebookresearch/
flores
 Table 4: Performance of ChatGPT for multilingual translation.

De-En Ro-En Zh-En

System
⇒ ⇐ ⇒ ⇐ ⇒ ⇐
Google 45.04 41.16 50.12 46.03 31.66 43.58
DeepL 49.23(+9.3%) 41.46(+0.7%) 50.61(+0.9%) 48.39(+5.1%) 31.22(-1.3%) 44.31(+1.6%)
Tencent n/a n/a n/a n/a 29.69(-6.2%) 46.06(+5.6%)
ChatGPT 43.71(-2.9%) 38.87(-5.5%) 44.95(-10.3%) 24.85(-46.0%) 24.73(-21.8%) 38.27(-12.1%)

De-Zh Ro-Zh De-Ro

System
⇒ ⇐ ⇒ ⇐ ⇒ ⇐
Google 38.71 21.68 39.05 25.59 33.31 32.27
DeepL 40.46(+4.5%) 22.82(+5.2%) 38.95(-0.2%) 25.39(-0.7%) 35.19(+5.6%) 34.27(+6.1%)
Tencent 40.66(+5.0%) 19.44(-10.3%) n/a n/a n/a n/a
ChatGPT 34.46(-10.9%) 19.80(-8.6%) 30.84(-21.0%) 19.17(-25.0%) 33.38(+0.2%) 29.89(-7.3%)

Wang et al. (2021)8 . Table 1 lists the information of
these test sets. However, obtaining the translation
results from ChatGPT is time-consuming since it
can only be interacted with manually and can not
respond to large batches. Thus, we randomly sam-
ple 50 sentences from each set for evaluation.
Metric. We adopt the mostly used BLEU
score (Papineni et al., 2002) as our primary
metric and also report ChrF++ (Popović, 2017)
and TER (Snover et al., 2006) in some cases.
These three metrics are all supported by Sacre-
BLEU (Post, 2018)9 .
2.2 Translation Prompts
To design the prompts for triggering the machine
translation ability of ChatGPT, we seek inspiration
from ChatGPT by asking it for advice. Specifically,
we ask ChatGPT with the following prompt:
Provide ten concise prompts
or templates that can make you
translate.
and obtain the results as shown in Figure 1. The
generated prompts look reasonable but share simi-
lar formats. Thus, we summarize them into three
candidate prompts as shown in Table 2, where
[SRC] and [TGT] represent the source and target
languages of translation. Note that we add an extra
command into T P 2 to ask ChatGPT not to gen-
erate double quotes around the translation, which
8 Translate. Table 4 presents the results.
https://github.com/hsing-wang/
WMT2020_BioMedical/tree/master/
Bio-18-19-testset
9
https://github.com/mjpost/sacrebleu
often occurs with the original format. Nevertheless,
it is still unstable such that sentences in a batch
(in multiple lines) are translated into a single line
occasionally.
We compare the three different candidate
prompts on the Chinese-to-English (Zh⇒En) trans-
lation task with the test set from Flores-101. Ta-
ble 3 shows the results of ChatGPT and three com-
mercial systems. While ChatGPT provides rea-
sonably good translations, it still lags behind the
baselines by at least 5.0 BLEU points. Concerning
the three candidate prompts, T P 3 performs the best
in terms of all the three metrics. Thus, we use T P 3
throughout this report by default.
2.3 Multilingual Translation
We select four languages to evaluate the capabil-
ity of ChatGPT in multilingual translation, includ-
ing German (De), English (En), Romanian (Ro),
and Chinese (Zh), which are commonly adopted in
both research (Wang et al., 2022a; Jiao et al., 2021,
2022b) and competitions (Bojar et al., 2016; Farhad
et al., 2021). The first three languages come from
the same family with Latin scripts while the last
is from another family with Chinese scripts (Fan
et al., 2021). We test the translation performance
between any two languages, which involves 12 di-
rections in total. For clarity and comparison, we
report the BLEU scores and the improvement or
drop of performance (i.e., +/-) relative to Google
Resource Difference. We consider the resource
difference of languages in the same family. In ma-

Figure 2: Translation results by ChatGPT with pivot
prompting (Date: 2023.01.31).
chine translation, German⇔English translation is
usually regarded as a high-resource task supported
by over ten million sentence pairs (Farhad et al.,
2021) while Romanian⇔English translation is sup-
ported by much less data (Bojar et al., 2016). This
resource difference can also be indicated by the
data statistics10 of GPT-3 (Brown et al., 2020), al-
though we do not know the data information of
ChatGPT. As shown in Table 4, ChatGPT per-
forms competitively with Google Translate and
DeepL Translate for both German⇒English and
English⇒German translations. However, it lags
behind them significantly on Romanian⇒English
and English⇒Romanian. Specifically, ChatGPT
obtains a BLEU score on English⇒Romanian that
is 46.4% lower than Google Translate and the
value is 10.3% on Romanian⇒English. We spec-
ulate that the huge resource difference of mono-
lingual data between English and Romanian lim-
its the language modeling capability of Roma-
nian, which partially explains the poor perfor-
mance on English⇒Romanian. On the contrary,
Romanian⇒English can benefit from the strong
language modeling capability of English such that
the resource gap of parallel data can be somewhat
compensated.
Language Family. We also take the impact of
language families into account. In machine trans-
lation, translating between different language fam-
ilies is often considered harder than that within
the same language family, due to the differ-
ent cultures and writing scripts. By compar-
ing German⇔English with Chinese⇔English or
German⇔Chinese translation, we find that the gap
10
https://github.com/openai/gpt-3/tree/
master/dataset_statistics
Table 5: Performance of ChatGPT with pivot prompt-
ing. New results are obtained from the updated Chat-
GPT version on 2023.01.31. LR: length ratio.
De⇒Zh Ro⇒Zh
System
BLEU LR BLEU LR
Google 38.71 0.94 39.05 0.95
DeepL 40.46 0.98 38.95 0.99
ChatGPT (Direct) 34.46 0.97 30.84 0.91
ChatGPT (Directnew ) 30.76 0.92 27.51 0.93
ChatGPT (Pivotnew ) 34.68 0.95 34.19 0.98
between ChatGPT and the commercial systems be-
comes larger. We attribute to the better knowledge
transfer within the same family (i.e., from English
to German) than between different families (e.g.,
from English to Chinese). For language pairs that
are both low-resource and from different families
(e.g., Romanian⇔Chinese), the performance gap
can be further enlarged (Wang et al., 2022b). Since
ChatGPT handles different tasks in one model, low-
resource translation tasks not only compete with
high-resource translation tasks (Jiao et al., 2022a),
but also with other NLP tasks for the model capac-
ity, which explains their poor performance.
Pivot Prompting. We explore an interesting
strategy named Pivot Prompting to improve the
translation quality between distant languages.
Specifically, we ask ChatGPT to translate the
source sentence into a high-resource pivot language
(i.e., English by default) first and then into the tar-
get language. We adjust the T P 3 prompt as below:
Please provide the [PIV]
translation first and then the
[TGT] translation for these
sentences one by one:
where [PIV] denotes the pivot language. As a
large language model, ChatGPT will naturally con-
dition on both the prompt and the translation result
in the pivot language to generate the translation
into the target language. Figure 2 shows an exam-
ple when using pivot prompting. There are several
advantages of pivot prompting:
• Knowledge Transfer: While parallel data be-
tween two distant languages is often scarce (Fan
et al., 2021; Wang et al., 2022b), the parallel data
between them and the pivot language can be rel-
atively considerable, which is expected to learn
better translation ability for source-pivot and
 Table 6: Performance of GPT-4 (Date: 2023.03.15) for multilingual translation.
System Zh⇒En
Google 31.66
DeepL 31.22
Tencent 29.69
ChatGPT (Direct) 24.73
ChatGPT (Directnew ) n/a
ChatGPT (Pivotnew ) n/a
GPT-4 28.50
pivot-target directions than that for the source-
target direction. Thus, pivot prompting will
potentially transfer the knowledge of the high-
resource pivot language to the low-resource tar-
get languages (Zoph et al., 2016; Aji et al., 2020;
Li et al., 2022; He et al., 2022).
• Convenience: Essentially, pivot prompting is
similar to the pivot translation technique in previ-
ous studies (Cheng et al., 2016) but is more con-
venient for ChatGPT. For the commonly adopted
multilingual sequence-to-sequence translation
models (Fan et al., 2021), pivot translation re-
quires two steps: (1) Input the source sentence
and translate it into the pivot language; (2) Input
the translation results in pivot language and trans-
late it into the target language. In contrast, Chat-
GPT can identify both the [PIV] and [TGT]
languages and translate the source sentence into
the two languages sequentially (see Figure 2),
which requires only one step operation.
Table 5 presents our results in BLUE score
and length ratio of translation results over refer-
ences. We obtain the translation results by us-
ing T P 3 (i.e., Direct) and pivot prompting (i.e.,
Pivot) through English (i.e., source-to-English-to-
target), respectively. As seen, the latest update
for ChatGPT seems to harm the translation qual-
ity for German⇒Chinese and Romanian⇒Chinese
translations, compared with the previous version
we used (i.e., Directnew vs. Direct). Nev-
ertheless, pivot prompting can significantly im-
prove the translation performance by nearly 3.9
and 6.6 BLEU points for German⇒Chinese
and Romanian⇒Chinese translations, respectively,
which demonstrates its effectiveness. By inspecting
the translation results, we find that direct transla-
tion with T P 3 will under-translate some tokens in
source sentences, which can be noticeably fixed by
En⇒Zh De⇒Zh Ro⇒Zh
43.58 38.71 39.05
44.31 40.46 38.95
46.06 40.66 n/a
38.27 34.46 30.84
n/a 30.76 27.51
n/a 34.68 34.19
42.50 38.16 37.84
pivot prompting. This can be reflected by the length
ratio results. Note that, while pivot prompting is
convenient for ChatGPT, how to further accelerate
the inference process is still an important research
question as we need to generate longer sentences.
GPT-4 further Bridges the Gap. With the
launch of GPT-4 (OpenAI, 2023) on March 15,
2023, we re-evaluate the performance of four trans-
lation directions. As shown in Table 6, GPT-4
boosts the performance over ChatGPT significantly
on all the four directions, bringing the BLEU scores
to the level of top commercial translation systems.
Note that these results only come from zero-shot
settings. With modern techniques like in-context
learning with demonstrations (Brown et al., 2020;
Agrawal et al., 2022), the translation performance
could be further improved. In other words, GPT-4
has already become a good translator!
2.4 Translation Robustness
We further evaluate the translation robustness of
ChatGPT on the WMT19 Bio and WMT20 Rob2
and Rob3 test sets, which introduce the impact
of domain bias and potentially noisy data. For
example, WMT19 Bio test set is composed of
Medline abstracts, which require domain-specific
knowledge to handle the terminologies. WMT20
Rob2 are comments from the social media website
reddit.com that could contain various errors,
including spelling/typographical errors, word omis-
sion/insertion/repetition, grammatical errors, spo-
ken languages, Internet slang, and so on (Michel
and Neubig, 2018).
Table 7 lists the BLEU scores. Obviously, Chat-
GPT does not perform as well as Google Translate
or DeepL Translate on the WMT19 Bio and WMT2
Rob2 test sets. The reason may be that commer-
cial translation systems like Google Translate often
need to continuously improve their ability to trans-
 Table 7: Performance of ChatGPT for translation robustness.

WMT19 Bio WMT20 Rob2 WMT20 Rob3

System
De⇒En En⇒Ja Ja⇒En De⇒En
Google 37.83 29.72 19.21 42.91
DeepL 37.13 26.25 19.83 41.29
ChatGPT 33.22 22.36 18.34 44.59

Table 8: Examples from WMT20 Robust Set3.
Example
S RC Haben wir noch Nudeln?
R EF Do we still have noodles?
Google Do we still have pasta?
DeepL Do we have any noodles left?
ChatGPT Do we still have noodles?
S RC Tatsächlich ist der zu häufige Gebrauch von
Seife schlecht für die Haut.
R EF Actually, very frequent usage of soap is bad
for the skin.
Google In fact, using soap too often is bad for your
skin.
DeepL In fact, using soap too often is bad for the skin.
ChatGPT In fact, the frequent use of soap is bad for the
skin.
late domain-specific (e.g. biomedical) or noisy
sentences, since they are real-world applications
that require better generalization performance over
out-of-distribution data. However, these may not
be done in ChatGPT.
An interesting finding is that ChatGPT outper-
forms Google Translate and DeepL Translate sig-
nificantly on WMT20 Rob3 test set that contains
a crowdsourced speech recognition corpus. It sug-
gests that ChatGPT, which is essentially an artifi-
cial intelligent chatting machine, is capable of gen-
erating more natural spoken languages than these
commercial translation systems. We provide some
examples in Table 8.
3 Conclusion
We present a preliminary study of ChatGPT for
machine translation, including translation prompt,
multilingual translation, and translation robustness.
By evaluating on a number of benchmark test
sets, we find that ChatGPT performs competitively
with commercial translation products (e.g., Google
Translate) on high-resource European languages
but lags behind significantly on low-resource or
distant languages. For distant languages, we ex-
plore an interesting strategy named pivot prompting
that asks ChatGPT to translate the source sentence
into a high-resource pivot language before into the
target language, which improves the translation
performance significantly. As for the translation
robustness, ChatGPT does not perform as well as
the commercial systems on biomedical abstracts
or Reddit comments but exhibits good results on
spoken language. With the launch of the GPT-4
engine, the translation performance of ChatGPT
is significantly boosted, becoming comparable to
commercial translation products, even for distant
languages. In other words, ChatGPT has already
become a good translator.
Limitations
We admit that this report is far from complete with
various aspects to make it more reliable:
• Coverage of Test Data: Currently, we randomly
select 50 samples from each test set for evalua-
tion due to the response delay of ChatGPT. While
there are some projects in GitHub trying to auto-
mate the access process, they are vulnerable to
browser refreshes or network issues. While Ope-
nAI has released the gpt-3.5-turbo API,
it shows to be less powerful than the Webpage
ChatGPT version11 . Therefore, we still report
the current results.
• Reproducibility Issue: By querying ChatGPT
multiple times, we found that the results of
the same query may vary across multiple trials,
which brings randomness to the evaluation re-
sults. For more reliable results, it is best to re-
peat the translation multiple times for each test
set and report the average result.
• Evaluation Metrics: The results here are cal-
culated by automatic metrics with single refer-
ences, which may not reflect some characteristics
11
https://github.com/
inspired-cognition/critique-apps/tree/
main/prompt-gym
 of translation properly, e.g., nativeness. Human
evaluation can provide more insights for compar-
ing ChatGPT with commercial translators.
• Translation Abilities: We only focus on multi-
lingual translation and translation robustness in
this report. However, there are some other trans-
lation abilities that can be further evaluated, e.g.,
constrained machine translation and document-
level machine translation.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke
Zettlemoyer, and Marjan Ghazvininejad. 2022. In-
context examples selection for machine translation.
arXiv.
Alham Fikri Aji, Nikolay Bogoychev, Kenneth
Heafield, and Rico Sennrich. 2020. In neural ma-
chine translation, what does transfer learning trans-
fer? In ACL.
Rachel Bawden, Kevin Bretonnel Cohen, Cristian
Grozea, Antonio Jimeno Yepes, Madeleine Kittner,
Martin Krallinger, Nancy Mah, Aurelie Neveol, Mar-
iana Neves, Felipe Soares, et al. 2019. Findings of
the WMT 2019 biomedical translation shared task:
Evaluation for medline abstracts and biomedical ter-
minologies. In WMT.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann,
Yvette Graham, Barry Haddow, Matthias Huck, An-
tonio Jimeno Yepes, Philipp Koehn, Varvara Lo-
gacheva, Christof Monz, et al. 2016. Findings of the
2016 conference on machine translation. In WMT.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, et al. 2020. Language models are few-shot
learners. NeurIPS.
Yong Cheng, Yang Liu, Qian Yang, Maosong Sun, and
Wei Xu. 2016. Neural machine translation with
pivot languages. arXiv.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi
Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep
Baines, Onur Celebi, Guillaume Wenzek, Vishrav
Chaudhary, et al. 2021. Beyond english-centric mul-
tilingual machine translation. JMLR, 22(107):1–48.
Akhbardeh Farhad, Arkhangorodsky Arkady, Biesial-
ska Magdalena, Bojar Ondřej, Chatterjee Rajen,
Chaudhary Vishrav, Marta R Costa-jussa, España-
Bonet Cristina, Fan Angela, Federmann Christian,
et al. 2021. Findings of the 2021 conference on ma-
chine translation (WMT21). In WMT.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-
Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr-
ishnan, Marc’Aurelio Ranzato, Francisco Guzman,
and Angela Fan. 2021. The flores-101 evaluation
benchmark for low-resource and multilingual ma-
chine translation. arXiv.
Zhiwei He, Xing Wang, Zhaopeng Tu, Shuming Shi,
and Rui Wang. 2022. Tencent AI Lab-Shanghai Jiao
Tong University low-resource translation system for
the WMT22 translation task. In WMT.
Wenxiang Jiao, Zhaopeng Tu, Jiarui Li, Wenxuan
Wang, Jen-tse Huang, and Shuming Shi. 2022a. Ten-
cent’s multilingual machine translation system for
WMT22 large-scale african languages. In WMT.
Wenxiang Jiao, Xing Wang, Shilin He, Zhaopeng Tu,
Irwin King, and Michael R Lyu. 2022b. Exploit-
ing inactive examples for natural language genera-
tion with data rejuvenation. IEEE/ACM TASLP.
Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Shuming
Shi, Michael Lyu, and Irwin King. 2021. Self-
training sampling with monolingual data uncertainty
for neural machine translation. In ACL.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Tho-
rat, Fernanda B. Viégas, Martin Wattenberg, Greg
Corrado, Macduff Hughes, and Jeffrey Dean. 2017.
Google’s multilingual neural machine translation
system: Enabling zero-shot translation. TACL.
Zhaocong Li, Xuebo Liu, Derek F Wong, Lidia S Chao,
and Min Zhang. 2022. ConsistTL: Modeling con-
sistency in transfer learning for low-resource neural
machine translation. In EMNLP.
Paul Michel and Graham Neubig. 2018. MTNT: A
testbed for machine translation of noisy text. In
EMNLP.
OpenAI. 2023. GPT-4 technical report. arXiv.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car-
roll L Wainwright, Pamela Mishkin, Chong Zhang,
Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
2022. Training language models to follow instruc-
tions with human feedback. arXiv.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-
Jing Zhu. 2002. BLEU: A method for automatic
evaluation of machine translation. In ACL.
Maja Popović. 2017. ChrF++: Words helping charac-
ter n-grams. In WMT.
Matt Post. 2018. A call for clarity in reporting bleu
scores. In WMT.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin-
nea Micciulla, and John Makhoul. 2006. A study of
translation edit rate with targeted human annotation.
In AMTA.
 Lucia Specia, Zhenhao Li, Juan Pino, Vishrav Chaud-
hary, Francisco Guzmán, Graham Neubig, Nadir
Durrani, Yonatan Belinkov, Philipp Koehn, Hassan
Sajjad, et al. 2020. Findings of the WMT 2020
shared task on machine translation robustness. In
WMT.
Wenxuan Wang, Wenxiang Jiao, Yongchang Hao, Xing
Wang, Shuming Shi, Zhaopeng Tu, and Michael Lyu.
2022a. Understanding and improving sequence-to-
sequence pretraining for neural machine translation.
In ACL.
Wenxuan Wang, Wenxiang Jiao, Shuo Wang,
Zhaopeng Tu, and Michael R Lyu. 2022b. Un-
derstanding and mitigating the uncertainty in
zero-shot translation. arXiv.
Xing Wang, Zhaopeng Tu, and Shuming Shi. 2021.
Tencent ai lab machine translation systems for the
WMT21 biomedical translation task. In WMT.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin
Knight. 2016. Transfer learning for low-resource
neural machine translation. In EMNLP.

View publication stats