This action might not be possible to undo. Are you sure you want to continue?
London: Aslib, the association for information management.
Chapter 8 Quality and standards - the
evaluation of translations
JUAN C. SAGER
There are no absolute standards of translation quality but only more or less appropriate translations for the purpose for which they are intended. Preliminary observations In order to discuss meaningfully the quality of a translator's work we have to consider the process of translation as an industrial process, subject to considerations of time and effort by which any work is measured. Similarly, the result of this process, the translation itself, is a commercial product of the information market, to which a certain price can be attached. It follows that the price and hence the quality of translations should be commensurate with the value attached to an original which is used for the same purpose and with the importance attributed to it in the process of communication. Translations are dependent texts in two senses: they are dependent upon an original which has a comparable form and serves the same function; in special cases the translation serves a different function possibly with a different form in which case we speak of translation as derived texts, e.g. in the gist translation of a letter or the summary translation of a contract. They are also dependent upon someone who commissions the translation either for their own language. The initiators of the translation implicitly or explicitly specify the function the translation is to serve within a communication process. They determine the time available for the work of the translator, and, through the price they are willing to pay, the type and quality of the translation required. The initiators thus play a crucial role in establishing valid criteria for the assessment of translations, their cost-effectiveness, their appropriateness and their quality. effectiveness, their appropriateness and their quality. Different types of texts require different methods of translation and lead to different end products. Most translations are modelled on, if not identical with, text forms of the target language but there are also translations which
do not match known forms of target language texts and which can therefore
THE TRANSLATOR'S HANDBOOK
While there are established text types which we can identify and whose general characteristics can be described, there is not an ideal model of letter, scientific report or instruction; each organisation develops its own variants according to the function any such document has in the communication processes which represent or accompany its activities. Consequently there is no ideal type of translation for any of these forms. Instead, any organisation which regularly requires translations decides the function of translations in the overall system of communication and develops patterns for types of translated texts. In reaching such decisions, organisations rely on the advice of translators who then have the role of information advisers. While translation is essentially concerned with the problems of interlingual transfer of messages, the practice of translation requires modifications of texts which are independent of the languages involved. It is therefore convenient to distinguish between the obligatory modifications of linguistic transfer with concomitant pragmatic changes conditioned by cultural differences and the deliberate modifications which are necessitated by a change in function of the translated text. This latter type of modification (selection, reduction, expansion of text) could theoretically be performed prior or subsequent to translation, but in practice these two processes are performed simultaneously. Conference translators spend the greater part of their time on this dual process. Other examples of such modifications are the translation of a text with a directive purpose into an information document and the selective or summarising oral translation. Any evaluation involves both comparison and measurement on a relative or absolute scale. Translations can be assessed in terms of completeness or accuracy but in practice such measurements are insufficient or even irrelevant (in the case of derived translations). In many instances a translation has to be assessed by the same criteria as an original, that is in terms of the adequacy of a text for its intended purpose and the cost-effectiveness of the method of production. With the increasing availability of various automated means of assistance and partial or total substitution of the human translator there is a greater diversification of products of translation and considerable variation in production costs which have to be considered. As translating is a diversified activity and only one phase in a complex communication process, full translations have to be compared with selective or summary translations, human translation with machine translation, wherever such alternatives may be appropriate. Factors such as the reader's time, the cost of that time, the maintenance cost of machine aids, time delays, typing costs and even abstracting or summarising cost and efficiency must therefore be taken into account and measured. The time and effort of translation varies with the nature of the original, the purpose of the dependent or derived text to be created and the skill of the translator. While in general certain types of text and their translation can be said to be more difficult than others and hence require more time, there is considerable variation in this assessment according to individual trans
QUALITY AND STANDARDS
reading comprehension and writing skills appropriate to their work. They are thus more or less familiar with the structures and styles of texts according to frequency of use. Translators are usually required to have a wider range of these skills than other professionals and a greater facility in using them; they may also need a wider scope of subject knowledge. In practice individuals develop propensities towards particular text types which together with solid subject
knowledge make certain translation tasks easier to perform than others, regardless of the theoretically perceived difficulty.
In order to determine the criteria which are of significance in the evaluation of translations we have to describe the considerable number of variables which affect the translation process. The form and type of the original The basis of the translation can be a written text or an oral instruction, e.g. to write a letter containing a certain amount of information. The original may be a full text, a draft for a translation which loses its validity as a message after it has been written in the foreign language or a note containing the content of a message required in the foreign language. The original text may be a complete text or part of a text, an excerpt or a title only. The form and type of the translation Independent of the original which provides the stimulus for the translation, the end-product may be a complete text or a part, e.g. a book, an article, an abstract, an excerpt or a title and it may be reduced or expanded in relation to the original, e.g. the minutes of a meeting in one language written from notes, the gist translation of a letter or a summary translation of an article. Translations can therefore be full substitutes of an original with the same range of functions and readership or derived texts which co-exist with the original involving two processes, e.g. a translation and a reduction. The status of the translation The status of a translation is determined by its communicative function in relation to the original. W e can distinguish three basic types of translations: A. The translation is an independent document, i.e. a full substitute for the monolingual reader. B. The translation is an alternative to the original and co-exists with it, e.g. a multilingual brochure. Either type may serve the same communicative functions as the original or be derived texts serving a different function. C. The translation is a full equal with the original in all respects and may therefore serve as a basis for other translation. This type is frequent in international organisations and some legal documents but is not as
THE TRANSLATOR'S HANDBOOK
Parameters of translation which affect evaluation
out being exhaustive, gives a number of uses which may affect the nature of translation: - scanning and discard - reading for information - detailed information and storage for future reference - draft for other texts - publication - for prestige - for public record - legal validity In each case the readership may vary from one or a few to a very large number. The textual factor As originals are characterised by subject matter, general or specialised mode of expression and intention (directive, informative, persuasive, evaluative, social, etc. ) which are reflected in appropriate text types and conventions of presentation, translations must either conform to equivalent text types and conventions (type A and C), deliberately choose others (type B), or indeed evolve their own types on the basis of user requirements (generally applied to type B, but also conceivable for type C, or with even greater force to machine translation output).
Research factor The amount and type of research required for a translation plays an important part in the quantitative aspects of evaluation, but as this varies considerably with the individual's knowledge and the degree of specialisation permitted by the job, it is practically impossible to quantify.
Revision factor The amount and cost of revision can significantly affect the cost and quality of translation. In this context revision is not understood as the checking for accuracy and completeness, but rather for consistency of style. The latter refers to both the internal consistency of a long document translated by several individuals and the harmonisation, with a house style, of an entire translated text: Summary We can represent some of the factors which contribute to the diversification of translation in a table as below, from which we exclude literary translation since it is subject different criteria and must therefore be evaluated
Criteria for evaluation Evaluation of translation can be carried out with many different objectives and accordingly different criteria and factors acquire varying importance. It is convenient to distinguish between macro-evaluation which aims at assessing the value of the product and micro-evaluation aimed at improving the product. Our main concern is macro-evaluation.
Objectives of evaluation Evaluation can be carried out as partial assessment of translators; it can be concerned with assessing the faithfulness of the translation with regard to content and intention - which is what most people understand when they speak of quality. Alternatively or simultaneously evaluation can be aimed at assessing the cost of a translation in comparison with other translations produced by the same or different means, e.g. human versus machine translation. Finally and perhaps most importantly, translation can be assessed in terms of the appropriateness for its intended purpose, which assumes that there are various alternatives possible for achieving the desired communicative effect. This global approach includes such criteria as intelligibility, which must be measured against the intelligibility of
the original, time of translation and time of reading by the end user. Marginal criteria which come into play - especially with the alternative of machine translation in mind - are disponibility of translation (i.e. the speed with which a translation can be completed and the availability of translators for less common languages and highly specialised subjects), the requirements for single or multiple translation and even the possibility of -storing translation versus repeating a translation when the need arises,
THE TRANSLATOR'S HANDBOOK
Preparation of translation Translators spend a considerable part of their time preparing for the task of writing or dictating a translation. This involves interpreting the specification and formulating strategies for implementing the modifications required in each case. When routines have been developed this preparation time is substantially reduced. The type and degree of deliberate modification is decided by an analysis of the text in relation to its new readership. It involves an analysis of the presuppositions on which the transfer of knowledge is based, the assumptions made about the correct interpretation of the intention of the message as well as the expectation the reader has about the message. The following table shows the scope of the analysis required.
Situational factors can only be taken into account when the translation is virtually concurrent with the transfer of the message; otherwise a new situation is created by the intervention of the translation process. (Incidentally, this table also shows the limitations of machine translation, which can only act on the analysis of linguistic signals.) Deliberate modification can be negligible if there is a large measure of coincidence between writer intention and reader expectation. The British Highway Code translated for French tourists would be such a case. A greater degree of modification may be appropriate for the translation of a Trade Union Rule Book as it is unlikely to be used for the same purpose, but only as a comparison of procedures. Presuppositions about differences in the knowledge between writer and reader are only rarely taken into account.
The role of the initiator The type of translation produced depends on the role of the initiator who can
either be no act for. the writer of the original or the reader of the trans QUALITY AND STANDARDS
translation is usually of type A, as in the case of a publisher who commissions the translation of a book in order to open up another language market, or a manufacturer who seeks a new export market. In both cases the translation has to be evaluated in terms of the writer's intention and the extent to which it achieves the desired effect, e.g. selling a product. The reader is then not interested in the original, except when he or she finds fault with the new text. If the initiator is the reader or his or her representative the type of translation is usually of type B. The reader knows that there is an original which he or she wants to read but is unable to do without a translation, and will specify what is sought from the translation regardless of the writer's intention. The quality of the translation is then judged on the extent to which the reader expectation is satisfied without distorting the content of the original. When reader expectation differs from writer intention the translator produces a derived text. Translations of type C are neutral with respect to the initiator as translation occurs simultaneously with writing, as for instance, in the work of the jurist/linguist panels of the Council of Ministers of the EC. In production terms these two roles can be interpreted as incoming (reader determined) and outgoing (writer determined) texts. Standards applied to each type are then questions of cost effectiveness and speed of information.
The awareness factor Translations are often judged by the extent to which the reader is aware of dealing with a translation. Such an awareness may be caused by linguistic interference, which is generally deprecated (on the level of pragmatic or cultural transfer and adjustment a translation may, however, deliberately retain characteristics of the original for stylistic reasons, e.g. appeal to certain prestige values). In reader initiated/oriented translation the knowledge of dealing with a translation is always present and it is then a question of deliberate decisions to what extent a translation should obliterate its origins. If the writer is aware of translation, he or she can offer great assistance to the translator to produce a translation of type A. The original may then be a draft to be replaced by the translation. If readers are aware of the translation they can instruct the translator as to their expectations, they can adjust their reading habits, decide whether to have a translation revised, or choose raw or post-edited output from a machine translation system. Reader awareness can lead to type B translation producing full dependent, alternative, partial or selective translations.
The user factor
The nature of a translation can also be influenced by the use that is to be made of it. While this is true of all texts, a translation can take into account
THE TRANSLATORS HANDBOOK w QUALITY AND STANDARDS
Modes of evaluation Evaluation has two aspects which usually complement each other in varying proportion according to the type of translation involved. (i) A new text can be compared to the original, via a metalinguistic representation, relatively to other translations, as in examinations or absolutely to an ideal type for the particular type. In each case there are three categories to be compared as in the diagram. ideal translation metaliguistic _
I original } ---> representation
(ii) A translation can be described, analysed and evaluated as if it were the result of a writing or re-writing process, the translator performing the role of a specialist writer. The greater the distance between contents form and intention between the original and the translation (draft, advertising copy, gist, directives for information) the more important is the second aspect. These two aspects also play an important role in the methods of evaluation. As translation is a mediation process evaluation should involve the end user and possibly the initiator, if a different person. The end user can, however, assess only under the second aspect as he or she would not require the translation if they were able to read the original. The first aspect requires assessors who have an equal command of the languages involved, which in practice means other translators. Dimensions of evaluation Evaluation can be qualitative or quantitative. Purely qualitative assessment ignores time, cost and other factors relevant to industrial processes and products. It can only be applied relatively to the original or absolutely to the end product. It has its value both for examinations of translators and for literary translation but in practice it is usually combined with aspects of a quantitative assessment. The latter take into account the significance of quality in relation to the importance attributed to the translation by the initiator in the wider industrial, commercial, political or social context of which it is a part. Quality is then subsumed under the heading of appropriateness which consists of accuracy, intelligibility, functional adequacy and formal presentation. A second major aspect in quantitative assessment is composed of economic factors: time spent on translation both as a production cost factor and as the time lag between initiation and delivery, ancillary costs of clerical and information services, reading time by end user, etc.
Quantitative evaluation also has to consider alternatives to human translation, e.g. the partial or total use of machines or even, where appropriate, an alternative linguistic regime; this could involve, e.g. adopting a single working language in a multilingual organisation, introducing a controlled language or languages for certain text types and purposes, or even reducing the overall volume of translation if the outlay does not appear to justify the benefit.
Methods of evaluation
A distinction must be made between the evaluation -------------------------------------------- of a translator's work - of a single translation - of a translation service - of a translation system We exclude from consideration the evaluation of student exercises as belonging to the area of training which is marginal to this handbook and concentrate on the evaluation of single translations.
Evaluation of a translator's work Unlike any other profession it is customary to select translators by competitive examination. This is particularly common among large employers of translators and felt to be necessary because there is no agreed standard of training, no body which controls admission to the profession and not even a protection for the title of translator. _ Employers must therefore feel that one or two test pieces produced in a two or three hour examination provide a more reliable indication of the qualities they seek in a translator than previous examination results, references or work carried out under unconstrained conditions. The ability to produce a certain amount of work in a given period of time is therefore an important criterion for selection and confirms the importance of economic considerations. There is considerable variation in the criteria used by translators' organisations for admission to membership. Some admit only by examination which may contain translation itself, general and special subject knowledge and even knowledge of the spoken language; some admit members on the basis of appropriate degree qualifications, professional experience or submitted work; others require a combination of these skills and achievements. In all these cases we are dealing with peer assessment and this is appropriate since the selection is made for membership of a professional group whatever the collective quality of this group may be. Evaluation of a single translation This kind of assessment consists of examining a translation both as an independent piece of writing and a linguistic and pragmatic transfer. Which of these two dimensions has greater importance is decided by the type of translation, the degree of deliberate modification and derivation specified and the declared purpose of the new text.
THE TRANSLATOR'S HANDBOOK QUALITY AND STANDARDS
In order to be objective the evaluation procedure must be repeatable; a set of clearly defined parameters is necessary and a scale of values with appropriate gradations. Since every text is different and any one text may be translated to different specifications, the parameters must be adjustable and susceptible to weighting so that an appropriate assessment scheme can be designed for any one task based on the original and its full specification. The evaluation procedure must be economically feasible. A simple characterisation of gravity of mistakes on a scale, e.g. distortion of meaning, omission and grammatical incorrectness and minor stylistic infelicities and orthographic errors each on a point scale is clearly insufficient for the evaluation of an industrial product; it may be, however, adequate or indeed justified in large-scale examinations. A basic distinction must be made between errors caused by inadequate knowledge of the vocabulary, orthography, morphology or syntax of the target language and those resulting from misinterpretation of the text or inadequate expression. Since the former are relatively rare - the target language of a translation is generally the language of habitual use of the translator - the latter are the main concern of evaluation. Any detailed evaluation must assume the translation process to consist of selection procedures with many predetermined and some free choices; many of these are mutually exclusive. The following elements must be combined in a grid for assessment and interdependences must be accounted for both at the lexical and the syntagmatic level. Types of error: - inversion of meaning - omission - addition unless justified by the specification. - deviation - modification Effect of error on the whole text; e.g. a typographic error can change the meaning of a word and thereby distort the whole text: - linguistic effect: does the error affect a main or secondary part of the sentence, e.g. the subject or a modifier? - semantic effect: does the error affect a major or minor element, e.g. the main argument, or an example? - pragmatic effect: does the error affect the intention in a significant or a negligible way, e.g. the general persuasive purpose or the tone of voice? It is obvious that with such a diverse matrix of criteria there will be a considerable variation
in the results of assessment, which can only be balanced by averaging scores obtained by different assessors. A different method involves the end user as principal assessor after the accuracy of the translation has been established (after a simple check for
intelligible (this may, of course, reflect on the original) and whether it is acceptable in terms of comparable texts in their own language given a certain cost for translation. Both these rather subjective assessments can be objectified to some extent by comparing the time required for the reading of a translation and an original. Other factors, such as presentation, layout, cost of production are important but are essentially the same for the production of any text, whether translated or not. "' Evaluation of a translation service The quality of translation is only a part of the complex set of criteria that enter into the assessment of a translation service which requires operation and management methods. Factors which are important from a translator's point of view are - contact with and feedback from initiators and end users - distribution of work among a team - support by and cost of revision - reference tools and library assistance - filing and access to previous translation - secretarial assistance Evaluation of a translation system As machine translation is still in a stage of rapid development there are no established methods of assessment, though considerable experimentation has taken place regarding the evaluation of quality of output and the comparative cost of human versus machine translation. Cost-effectiveness of machine translation cannot sensibly be calculated partly because there is no clear philosophy how development costs should be reflected, partly because there is insufficient data on maintenance including continuous dictionary-making costs and partly because it is as yet impossible to calculate the cost of translator involvement in pre- or postediting. The development of criteria for evaluation of machine translated texts is important not only for its own sake; it is also an integral part of the feedback necessary for the improvement of the machine translation system. Only in the very few instances where a machine translation system is fully operational and regularly processes large quantities of texts is a genuine comparison with human alternatives possible.
GAMBIER, Y (ed.) Trans, University of Turku School of Translation Studies, Turku, 1986. . GOUADEC, D. Paramètres de l'évaluation des traductions, in: META 2$, 2, 1981, 99-117. HOUSE, J. A model of translation quality assessment, Tübingen: Narr,
THE TRANSLATOR'S HANDBOOK
NEUBERT, A. Text and Translation, VEB Verlag Enzyklopaedie, Leipzig, 1985. REISS, K. Möglichkeiten und Grenzen der Übersetzungskritik, München: Huber, 1982 (4th ed.). van SLYPE, G. Conception d'une méthodologie générale d'évaluation de la traduction automatique, in: Multilingua 1, 4, 1982, 221-223. SNELL-HORNBY, M. (ed. ) Übersetzungswissenschaft - EineNeuorientierung, Zur Integrierung von Theorie und Praxis, Francke Verlag, Tübingen, 1987.
This action might not be possible to undo. Are you sure you want to continue?