Comparative Analysis of Accuracy and Reliability in Machine Translation versus

Human Translation: An Empirical Investigation in Real-life Conditions

Abstract

The purpose of this research is to investigate and compare the quality of human translation and
machine translation in the target language. For this purpose, using samples of both types of
translation, the grammatical structure of sentences, the diversity of vocabulary usage, and the
desired translation speed have been evaluated and compared in both types of translation. In this
study, the participants consisted of 32 students of the Mofid University in Qom, aged between
19 and 24, who were enrolled in the fields of Law, English Language Teaching, and English
Translation. The finding revealed that in many cases, human translation has performed better
than machine translation. Considering that human translation is costly and time-consuming,
machine translation has become one of the alternative solutions for translation. However, there
are still many cases where machine translation has failed to translate accurately. Therefore, the
comparison between human translation and machine translation has become an important topic
in the field of translation. Your research can help to better understand the strengths and
weaknesses of both methods and overall improve the quality of future translations.

Keywords: Machine translation, Human translation, translation technology, Translation studies

Introduction

The field of Machine Translation (MT) is constantly evolving and plays a crucial role in
breaking down communication barriers in our fast-paced globalized world. Unfortunately, the
traditional way of teaching this subject through lectures does not adequately prepare students
for the real-world challenges they may face. However, by incorporating practical examples into
the teaching of MT, and continuing to research and develop new approaches to pedagogy, we
can better equip students to succeed in this exciting and important field.

Machine Translation (MT) is a field that involves the development of automatic

systems that use advanced computational linguistic analysis to process source documents and
generate target texts without human intervention. Research has shown that modern
technologies used in developing MT continue to provide new possibilities for its development,
with implications for professional and social life. For example, the internet has connected
people of different languages and cultures, making MT essential for translating webpages and
social network sites. MT systems now serve both professional and non-professional translators
in various fields of life, but teaching and learning MT as an academic discipline remains a
challenging task due to its complex characteristics, requiring knowledge in linguistics,
translation studies, mathematics, and statistics. To address this challenge, new approaches to
teaching MT are needed. Specifically, this study investigates the effectiveness of using project-
based teaching strategies, which are based on cognitive tactics founded on brain research.
These strategies can be integrated into problem and inquiry-based techniques and can be applied
in the MT classroom since the three approaches are closely connected with information-
processing learning theories. By using project-based teaching strategies, students can acquire
practical skills that allow them to understand and tackle the challenges of MT in a real-world
context. This study aims to explore the potential of project-based teaching strategies in teaching
MT, to better prepare students for the complex challenges they may encounter in this field.

Machine translation is an automatic system that uses advanced computational linguistic

analysis to process source documents and generate target texts without human intervention. The
continuous development of modern technologies has opened up new possibilities for machine
translation to impact daily professional and social life. The increasing need for machine
translation has been fueled by the internet, which has connected people from different language
backgrounds and cultures. Despite the significant research devoted to developing machine
translation systems, there is still a need for innovative strategies to teach and learn machine
translation as an academic discipline. This is challenging because machine translation is a
complex field that requires pedagogical expertise in various domains. Therefore, this study
aims to investigate the effectiveness of project-based teaching strategies that are based on brain
research and closely linked with information-processing learning theories in teaching machine
translation. By adopting these strategies, students can acquire practical skills that enable them
to understand and tackle the complex challenges of machine translation in a real-world context.

1. Research question:

This research aims to investigate and compare the quality of human translation and machine
translation in the target language. For this purpose, using samples of both types of translation,
the grammatical structure of sentences, the diversity of vocabulary usage, and the desired
translation speed have been evaluated and compared in both types of translation. After
conducting the necessary analyses and comparisons, the obtained results will be examined, and
the necessary conclusions will be presented to others in the field of human and machine
translation quality. The ultimate goal of this research is to improve the quality of machine
translation and provide suggestions for improvement in this area.

1. Under what circumstances does human translation demonstrate superior performance to

machine translation?

2. Could machine translation be utilized as a substitute for human translation in certain

situations?

2. Assumptions and Hypotheses

Based on previous studies, it has been determined that in some cases, human translation
outperforms machine translation in terms of quality and accuracy. This hypothesis is based on
the fact that human translation, due to its expertise and experience in the field of translation, can
better identify and translate difficult and complex idioms and phrases. Additionally, human
translation can be the best option for languages with difficult rules and complex structures. To
investigate this hypothesis, this study has focused on examining and comparing the quality of
human translation and machine translation in the target language. Samples of both types of
translation have been evaluated and compared in terms of grammatical structure of sentences,
diversity of vocabulary usage, and translation speed.

Review of literature

The literature review discusses various topics related to machine translation, including its
development, different systems, hybridization, and quality of translation, project-based teaching
methodology, and educational technology. It also evaluates the practical use of machine
translation. Wilks suggests that the 1400s to 1600s were a discovery era when communication
between different language speakers became necessary. Leibniz and Descartes researched codes
to relate words between languages in the 17th century. Later in the 1930s, Georges Artsrouni
developed an automatic bilingual dictionary. In 1949, Warren Weaver proposed the first
computer-based machine translation in his Translation Memorandum. He was influenced by
McCulloch and Pitts' theory on mathematical modeling of the neural structure of the human
brain and the applicability of cryptographic methods. Claude Shannon's information theory is
related to the concept of cryptography, and his theory is concerned with the statistical properties
of communication. The Weaver's Translation Memorandum led to the appointment of Bar-Hillel
at the Massachusetts Institute of Technology in 1951 to research mathematical formulae's use in
machine translation.

The development of machine translation is traced back to the 17th century when
Leibniz and Descartes proposed codes to link words between languages, and a bilingual
automatic dictionary was developed in the 1930s. The first proposal for computer-assisted
machine translation was made in 1949 by Warren Weaver, who was influenced by the theory of
modeling the neural structure of the human brain. In the 1950s, researchers at Georgetown
University experimented with fully automated translation, and real progress was made with the
report of the Automatic Language Processing Advisory Committee in 1966, which highlighted
the feasibility of high-quality machine translation. The initial research in machine translation
was based on linguistic theories of translation, and empirical trial-and-error methods were used
to analyze grammatical regularities and vocabulary between languages. The first generation of
machine translation applied the word-to-word translation method and had limited capabilities,
but large-scale projects funded by the US government eventually led to the development of
more advanced machine translation systems.

Hybridization refers to strategies of combining two or more machine translation

systems to improve the translation process. For example, Costa-jussà and Fonollosa (2015)
argued that a hybrid system could contain corpus-based rules or combine different corpus-based
approaches. This combination can be done in two main ways: by using rules in pre- and post-
processing or by integrating dictionaries/rules in the basic model. Machine translation
applications with hybrid components include multilingual information retrieval, computer-
assisted translation, and language translation. Hasibuan (2020) investigated a distinction
between human and machine translation. They found that human translation was more effective
and acceptable because it had a greater understanding of context and meaning, while machine
translation was more literal. The researcher also identified that machine translation was unable
to understand the purpose or intention behind the words in the target language, limiting its
capabilities. This shows that human translation is even more valuable since it can provide
accurate translation with better conception. Doherty (2017) argued that the widespread use of
translation technologies, particularly Machine Translation (MT) has led to a range of practical
definitions related to MT quality and metrics. They further state that MT quality has been
considered as a means to achieve product or improve MT systems, with a focus on a pragmatic
approach that combines both human and automatic evaluations. The author suggested that there
is a need for a balance of both practical and theoretical approaches to achieve high-quality MT.

A recent research by Lihua (2022) indicates that the capabilities of Machine Translation
(MT) have improved significantly under the influence of Artificial Intelligence (AI). This has
led to the conclusion that MT could replace human translation in some areas. However, there is
still a consensus that MT cannot fully replace the nuances and expressiveness of human
translation, even with the impact of AI. This study discussed the implications of the
development of Artificial Intelligence (AI) based Machine Translation on the relationship
between human translation and Machine Translation. While AI-based machine translation has
achieved similar accuracy and overall status as human translation. Bizzoni et al. (2020)
examined translationese patterns that may appear in human and machine translation outputs.
This paper explored two non-invasive methods of detecting translationese, as well as compared
them across different types of translation outputs in both tow text and speech, and also seeks to
address the hypothesis that patterns are caused by the model of human vs. machine rather than
by the type of data. The study found that while machine translation presents traces of
translationese, it does not reproduce all of the patterns found in human transcription.

Maučec and Donaj (2019) give an overview of different machine translation

approaches, such as statistical machine translation (SMT) and neural machine translation
(NMT). This research discusses the limitations of SMT and its capacity for certain languages,
particularly highly agglutinative languages. Moreover, explains neural machine translation, as
well as evaluation methods and comments on the integration of humans and machine translation
as a future workflow. Xiu and Xeauyin (2018) explained how human and machine translation
can work together to produce the best possible translation. Human translation (HT) is the oldest
and most sophisticated form of translation. It relies on human intelligence to convert one
language into another. HT is a crucial component of communication and knowledge sharing. It
is highly necessary for effective and empathetic communication between cultures and is
essential for social harmony and peace. Human translation is also superior to Machine
Translation (MT), which cannot render such nuances accurately. MT is incapable of handling
such nuances due to its limited capabilities.

Research method

In this study, the participants consisted of 32 students of the Mofid University in Qom, aged
between 19 and 24, who were enrolled in the fields of Law, English Language Teaching, and
English Translation. Both male and female students from these fields participated in the study.
They were divided into two groups of 16, and each member was presented with twenty similar
sentences to translate.

Group A traditionally translated the sentences based on the Prescriptive theory. This
approach specifies how a language should be used and what rules should be followed based on
the specific language instructions. Group A also translated the sentences according to word by
word. The only tool that Group A used was an English-to-Persian dictionary. Group B, on the
other hand, translated the sentences using machine translation and artificial intelligence. They
mostly used Google Translate.

In this research, participants translated their sentences through social media. Translation
accuracy, speed, variety in the use of vocabulary and grammatical structures, as well as the
ability to convey all aspects of meaning from the source language to the target language, were
used as evaluation criteria. The sentences used in the study were neither too difficult nor too
easy. Overall, this research was conducted to investigate the effectiveness of a project-based
teaching method in machine translation

Data analysis

This study investigated the effectiveness of a project-based teaching method in machine

translation. The project-based teaching method was designed to integrate theories of learning
related to brain research and bridge the gap between theories of machine translation and
operations. Using the project-based teaching approach, the researchers intended to increase the
motivation, ability, and performance of students in machine translation. In this text, this study
examines the effectiveness of project-based teaching methods in machine translation through an
experimental study. By dividing students into two groups and using various criteria to evaluate
their translations, the study aimed to find the impact of this teaching approach on improving
accuracy, speed, variety in the use of vocabulary and grammatical structures, and the ability to
convey all aspects of meaning from the source language to the target language.

1. Grammatical Structures

In some translations, both traditional and machine, there was not much difference apart from the
choice of words. However, some participants decided to strictly adhere to the structure of the
target sentence and even translated the sentence order just like the source language (English).
They did not follow the translation standards of the target language (Persian), as in the
following sentences:

1. She woke up early and went for a jog in the park.

‫او بیدار شد زود و رفت برای دویدن در پارک‬

‫او بیدارشدن زود و برای یک دوی آهسته در پارک رفت‬

 2. The restaurant was packed with people enjoying their meals.

‫رستوران پر از افرادی(مردمی) که داشتن از غذاشون لذت میبردن بود‬

‫رستوران از آدمایی که از غذاشون لذت می‌بردن پر بود‬

This translation, which was done traditionally conveys the general meaning of the sentence.
However, it has a grammatical mistake, such as the phrase "‫( "از غذاشون‬meaning "their food")
being used as the object of the verb "‫( "لذت بردن‬meaning "to enjoy") which is incorrect in
Persian grammar. Due to this grammatical error, the object and subject of the sentence have
been reversed, making it incorrect. Therefore, these translations have a writing mistake.
Although some participants were instructed to translate word by word, some of them adhered
well to the translation standards of the target language. It seems that their sensitivity and
attention to detail in translation led them to provide a creative translation, such as the following:

‫او صبح زود بیدار شد و برای پیاده روی به پارک رفت‬.

1. The museum exhibits ancient artifacts from different civilizations.

‫ تمدن های مختلف را به نمایش می گذارد‬،‫موزه آثار باستانی‬

2. His team won the championship after an intense game.

‫ قهرمانی را از آن خود نمود‬،‫تیمش بعد از یک بازی جدی‬

In the second group, the participants used machine translation and artificial intelligence to
translate. Some relied solely on machine translation and translated without making any changes,
with the same translation errors as those made by the AI translator, such as:

.‫او زود از خواب بیدار شد و برای دویدن در پارک رفت‬

This translation, "‫( "در پارک رفت‬meaning "went in the park"), is grammatically incorrect.

However, some translations showed the creativity of some participants:

3. The politician's speech was met with both applause and criticism.

.‫سخنان این سیاستمدار با تشویق و انتقاد مواجه شد‬

Unlike other translations that relied solely on machine translation, these translations not only
conveyed the meaning and fluency of the sentence but also showed a creative approach to
transferring all aspects of the original meaning to the target language in the shortest possible
.form

2. Conveying meaning:

Some participants in group (A) were unable to provide an accurate and fluent translation due to
their traditional and word-by-word translation approach. For example:
 ‫او بیدارشد زود و رفت برای قدم زدن به پارک‬

Some individuals attempted to provide an accurate translation by adhering to precise rules of

grammar and literal interpretation of words, but it seems they did not pay enough attention to
the clarity and fluency of the translation. A translation that solely relies on strict adherence to
grammar rules and literal interpretation of words may lead to more ambiguity for the reader or
listener and may fail to convey the meaning clearly and fluently Therefore, for a better
translation, in addition to adhering to accurate grammar rules and interpretation of words,
attention must also be paid to the clarity and fluency of the translation, and efforts made to
ensure that the translated text is understandable, flows well, and conveys the intended meaning
of the original text to the reader or listener.

.‫او(اشاره به مونث)زود بیدار شد و به دویدن(آهسته دویدن)در پارک رفت‬

.‫او(صبح) زود از خواب بیدار شد و برای(آهسته) دویدن به پارک رفت‬

In the second group, who relied on machine translation, they could not provide a clear
translation of the sentence. Their translation was ambiguous. In these translations, it is as if the
concert has been sold, not the concert tickets.

.‫کنسرت چند ساعت پس از فروش بلیط ها فروخته شد‬

.‫کنسرت در ساعاتی پس از عرضه بلیت به فروش رفته بود‬

Some are translated freely to convey more meaning.

.‫همه ی بلیط های کنسرت که به حراج گذاشته شده بودند در عرض چند ساعت به فروش رفتند‬

3. Diversity in the use of vocabulary

In this sentence, the word "jog" caught the attention of some of the participants. Some
participants have deciphered this word completely in various ways into the target language.

for example, "‫ " دوان دوان‬or "‫"ورزش صبحگاهی‬. The first group, who translated more traditionally,
paid more attention to this word. However, the second group, who relied on machine
translation, did not give much attention to this word.

1. The artist's paintings are known for their vivid colors and bold brushstrokes.

In this sentence, the word "bold brushstrokes" was challenging for many participants. And they
provided different translations of this word. The first group of human translation and the
second group of machine translation chose different words.

Human translation:

.‫نقاشی‌های این هنرمند به خاطر رنگ‌های زنده و "قلم‌موهای جسورانه‌شان" شناخته می‌شوند‬

.‫ به دلیل "رنگ های شفاف" و ”اثر واضح قلمو” مشهور اند‬،‫نقاشی های این هنرمند‬

Machine translation:
 .‫به رنگ های شفاف و ”قلم های ضخیمش” مشهور است‬،‫اثار این هنرمند‬

.‫ با "رنگ‌های روشن" و ”خطوط قلم پهن” شناخته می‌شوند‬،‫نقاشی‌های آن هنرمند‬

Results and findings

In this table, different quality criteria for machine translation and human translation are
specified and displayed as a percentage. According to the mentioned result, it can be said that
human translation works best in most of the quality criteria. It performs better in terms of
translation accuracy, translation speed, ability to convey all semantic aspects, use of appropriate
vocabulary, grammatical structure, and ability to interpret complex concepts of human
translation. However, in criteria such as diversity in grammatical structures and translation
speed, machine translation performs better. Also, in some special cases, machine translation
can work better than human translation, but this depends on the creativity of the translator and
cannot be shown in general. In any case, according to this table, we can conclude that human
translation works better than machine translation in most cases. Table 1 presents the results for
each metric based on different percentages.

Table 1: comparative percentage results

№ Quality criteria Machine translation Human translation

1 Translation accuracy 30% 80%

2 Ability to convey all semantic aspects 35% 85%
3 Use of appropriate vocabulary 25% 80%
4 Correct grammatical structure 60% 40%
5 Ability to interpret complex concepts 25% 85%
6 Translation speed 70% 30%

From this research, to answer the first question, we can conclude that in most cases,
human translation works better than machine translation. According to the table comparing the
quality of machine translation and human translation in different criteria, it can be seen that
human translation performs better than machine translation in terms of accuracy of translation,
conveying the meaning, and using appropriate vocabulary. However, according to reports,
machine translation had a higher performance than human translation in criteria such as
translation speed and variety in correct grammatical structure. Therefore, it can be concluded
from this research that human translation works better than machine translation in situations
where translation accuracy, the ability to convey all semantic aspects, the use of diverse
vocabulary, and the ability to interpret complex concepts are of great importance. Also, in
situations where there is a need to pay attention to details and consecutive concepts. On the
contrary, in situations where translation speed and diversity in grammatical structures are more
important, machine translation works better.

Although the ability of machine translation using neural networks and learning has
improved in recent years, human translation still outperforms machine translation in many
aspects. For example, human translation can be better than machine translation in interpreting
complex concepts and terms, precision in translation, use of appropriate vocabulary, ability to
convey all semantic aspects, and ability to interpret linguistic innovations. Therefore, in
response to the second question, it should be said that machine translation can be used in certain
situations, such as the translation of simple and technical texts with a specific grammatical
structure, and in cases where speed and efficiency in translation are very important. But in
cases where precision in translation, transfer of meaning, and interpretation of complex
concepts are of great importance, it is better to use human translation. Also, in cases where
translation is done for different and diverse cultures of users, human translation works better
than machine translation due to more cultural and linguistic knowledge. In general, machine
translation is recognized as a useful tool in the translation industry, but there is still a need for
human translation to translate complex texts with complex grammatical structures, as well as
better translation for cultural and linguistic diversity. As a result, depending on the type of text
and different circumstances, machine translation or human translation may be used.

Conclusion

In this study aimed to investigate under what circumstances human translation outperforms
machine translation and also how Machine translation can be utilized as a substitute for human
translation in certain situations. The finding of this study shows that In most cases, human
translation is better than machine translation when it comes to accuracy, conveying meaning,
and using appropriate vocabulary. However, machine translation can perform better than
humans in terms of speed and grammatical structure. Human translation is more suitable for
situations where precision in translation, conveying all semantic aspects, and interpretation of
complex concepts are crucial. Machine translation can be used for simple and technical texts
with specific grammatical structures where speed and efficiency are important. Cultural and
linguistic knowledge is essential for human translation, making it more suitable for translating
different and diverse cultures. Machine translation is recognized as a useful tool in the
translation industry, but human translation is still needed for complex texts and better
translation for cultural and linguistic diversity. Depending on the text type and situation, either
machine or human translation may be used.

This study has been conducted in the field of translation and its quality and has
compared the quality of human and machine translations from two different sources. This study
can be useful in evaluating and improving the quality of machine translations. Additionally, it
can help users and translators to better understand the differences between human and machine
translations. This research can also assist researchers and students in the field of translation and
language technologies. Furthermore, the results of this study can be utilized in decision-making
regarding the use of human or machine translations in translation projects. Overall, this study
can be beneficial in the development and improvement of translation technologies, as well as in
enhancing the quality of machine translations. This article had some weaknesses, such as the
lack of enough sample sentences which were also too short. Perhaps they could have been
included in a single paragraph.
 Future research in this field could explore several areas to advance our understanding of
the subject. Firstly, researchers could investigate the use of different machine translation models
and algorithms to improve the quality of machine translations. This could involve comparing
the performance of different models in terms of accuracy, efficiency, and adaptability to
different languages and translation tasks. Secondly, researchers could explore the impact of
machine translations on the translation industry and the job market for human translators. This
could involve analyzing the market trends and demand for machine translations, as well as the
skills and competencies required for human translators to adapt to the changing landscape of the
industry.

Thirdly, future research could investigate the potential of using machine translations in
specialized fields such as medicine, law, and engineering. This could involve evaluating the
accuracy and effectiveness of machine translations in these fields, as well as the potential
benefits and challenges of using machine translations in professional settings. Lastly, future
research could focus on developing new evaluation metrics and standards to assess the quality
of machine translations. This could involve designing new benchmarks and test sets to measure
the performance of machine translations, as well as exploring new approaches to evaluating the
fluency, coherence, and overall quality of machine translations. Overall, future research in this
field could contribute to the development of new technologies and approaches to improve the
quality of machine translations, as well as our understanding of the impact of machine
translations on the translation industry and the society at large.

References

Bizzoni, Yuri & Juzek, Tom & España-Bonet, Cristina & Chowdhury, Koel & Genabith, Josef
& Teich, Elke. (2020). How Human is Machine Translation? Comparing Human and
Machine Translations of Text and Speech. DOI:10.18653/v1/2020.iwslt-1.34

Costa-jussà, M. R., & Fonollosa, J. A. R. (2015). Latest trends in hybrid machine translation
and its applications. Computer Speech & Language, 32(1), 3-10.
http://dx.doi.org/10.1016/j.csl.2014.11.006.

Doherty, Stephen. 2017. Issues in human and automatic translation quality assessment. In:
Dorothy Kenny (ed.). Human issues in translation technology, 154–178. London:
Routledge

Hasibuan, Zainuddin. (2020). A Comparative Study Between Human Translation and Machine
Translation as an Interdisciplinary Research. Journal of English Teaching and Learning
Issues, 3 (2) 115 – 130, 2020. DOI: 10.21043/jetli.v3i2.8545

Maučec, M. S., & Donaj, G. (2019). Machine translation and the evaluation of its quality. In A.
Sadollah (Ed.), Recent trends in computational intelligence (pp. 1–20). IntechOpen

Lihua, Z. (2022). The Relationship between Machine Translation and Human Translation under
the Influence of Artificial Intelligence Machine Translation. Mobile Information
Systems, 2022(5). http://dx.doi.org/10.1155/2022/9121636
Xiu, P., & Xeauyin, L. (2018). Human translation vs machine translation: The practitioner
phenomenology. Linguistics and Culture Review, 2(1), 13-23.
https://doi.org/10.37028/lingcure.v2n1.8