1.

Introduction

The issue of how to measure quality has long been an area of keen discussion in

translation studies. And while academic studies have proposed a number of models to

measure the translation competence of students or the quality of a translation product,

in many cases these efforts have progressed in a virtual vacuum separate from the

professional field. One of the results of this disconnection between the academic and

the professional is that end-users of translation services still show “a general lack of

understanding of the translation process (Ramussen and Schjoldager 2011: 87),”

leading to situations where the translator must explain why a customer’s demands

cannot be accommodated or, in extreme cases, a misunderstanding escalates and the

customer lodges a claim about the translation product.

Due to an understandable lack of information on the subject, the issue of

claims lodged against a translation product has largely remained from the focus of

research. Data on the specifics of quality-based claims leveled against a translation

service provider (TSP) are largely absent, understandable as the majority of such

claims are often resolved on an ad-hoc basis in accordance with customer demands

made at the time, and the TSP in most cases lacks the time and motivation to fully

document the incident, let alone publicize the fact.

As a quasi-academic institution founded by alumni of the Graduate School of

Interpretation and Translation (GSIT) within the Hankuk University of Foreign

Studies, Seoul, South Korea, the Hankuk University of Foreign Studies Center for

Interpreting and Translation (HUFSCIT) is positioned at an opportune intersection

between academia and practice in the field of translation studies. Operating within the

university, the Center is both a school organ and a provider of professional translation
services, placing it in a position to provide customers with lessons gained from

academic studies while also sharing knowledge about professional situations within

the GSIT community.

Situations where a customer lodges a claim can be seen as extreme examples

of customer dissatisfaction, thus serving as critical data offering vivid insights into

what customers expect when they commission a professional translation. A study of

such claims will hopefully contribute to the ongoing discussion on expectations of

translation quality, particularly in regard to the customer as a stakeholder in the

translation process. The ultimate aim of this study is to provide practitioners with a

more balanced understanding about the parameters of quality as perceived by

customers, and through this to facilitate communication between the providing and

commissioning sides of translation services and between academic and professional

endeavors within our field.

2. Divide between practice and theory in translation studies

The need for translation studies to address the existing gap between theory and

practice in the field of translation has been voiced in a number of publications (cf.

Lauscher, 2000; Katan, 2009; Drugan, 2013). While translation studies as an

academic discipline has branched out into diverse areas and today enjoys a wealth of

connectivity with other academic disciplines, the links between academic studies and

real-world practice of translation remain relatively sparse, leading to descriptions such

as by Milton (2001) that “Translation Studies as an academic area exists as an almost

separate domain from that of professional translation” (as cited in Katan, 2009, p.

113).
 This situation is particularly marked in the area of translation quality

assessment (TQA). While there is a clear need for reliable and practical methods to

evaluate the quality of translation products, such methodology yet remains beyond

ready access. And should we understand Bassnett’s opinion that “translation studies

has now, in a sense, arrived” (2012, p. 18) as an observation about the field

establishing itself as a valid and important discipline within the realm of academic

study, it follows that such a status should be further supported by rigorous theoretical

constructs to assess the quality of translations, constructs which can be empirically

tested in professional translation situations. Such empirical testing becomes

particularly important when self-reflection in itself, on the part of the researcher,

becomes insufficient to ensure that the results of our research remain valid and useful

to the outside world. Setting aside for the moment discussions of whether academic

studies should aim to yield practical implications, it has been noted that the lack of

clear theoretical frameworks in “models of TQA […] inevitably reflect an overall

theoretical framework (or lack of it) and can be discussed in terms of such”, (Honig,

1998: 6). and while Hague, Melby and Zheng (2011) report that “convergence is now

within reach” (Hague et al., 2011) when it comes to academic views of TQA and its

theoretical basis, there is no doubt that the field as a whole could benefit from

continuous work to further refine these views and link them to the realities of our

profession.

As a somewhat humorous but astute saying goes, translation quality is like

love; everyone knows what it is and knows it when they see it, most have had their

own personal experiences with it, but no one can say what it exactly is (taken from

oral presentation by Martin Forstner, 2015, at Hankuk University of Foreign Studies).

The fact that anyone with some experience in translation eventually develops their
own invested view as to what quality in translation should be is closely linked to is

why it has persistently proven difficult to devise a comprehensive and coherent

statement of what translation quality actually is. Out of this state of affairs a wide

range of theoretical views have developed, which will be briefly discussed with a

focus on practical implications in the field of professional translation, within which

this study will limit itself.

3. TQA approaches proposed within academia

Marking the interest in the area of TQA, in 2001 the academic journal Meta dedicated

a special edition to efforts in translation evaluation. For the sake of effective brevity

in an area where a vast amount of research already exists, it could be useful to

summarize academic views on TQA into pre- and post-2001 views on this subject.

(1) Pre-2001 views on translation evaluation

Parting from non-empirical (hermeneutic or mentalist) approaches to TQA, there was

a pronounced movement toward norm-based assessment (DTS) of translations,

perhaps most visibly by Gideon Toury (1995), whose work continues today. This was

followed by linguistics-based or textological approaches to TQA (cf. House

1979/2001; Larose, 1987; Bensoussan and Rosenhouse, 1990; Hatim and Mason

1990, 1997; Jickey 1998), among which the Julian House (1979/2001) linguistic-

functional model is rooted in a firm view of translation as “a linguistic phenomenon”

(House, 2015, p. 21), thus allowing for an assessment of the semantic and textual

equivalence between a pair of source and target texts. The model has remained valid
and significant throughout sweeping changes within translation studies, has been

further modified to incorporate empirical findings conducted on text pairs belonging

to multiple genres, and in its final form stands today as the single most sustained and

consistent body of research in the area of translation evaluation.

At the same time, however, House’s model foregrounds the question of how

we are to bridge the gap between academic studies of translation and its actual

practice. This gap is even more evident when we consider an actual evaluator tasked

with employing the model in real-life settings. The sheer complexity of the concepts

and terminology used by the model inevitably lead to a lack of facile operation, and

while the model can be made to forward a compelling and exhaustive statement of

quality, a convincing use of the model would be would near impossible if the

evaluator lacked a background in translation theories. Also, the model cannot

compare other different TTs and/or produce a quantifiable judgment. In regard to this

House herself states that absolute objectivity in evaluation should not be the goal and

rather we should appreciate the inevitability of subjective elements when evaluating

translation, as “human beings are here important variables” (House, 2015, p. 34).

In the end, in its current form, House’s TQA model remains a crucial source of

necessary descriptions upon which statements of quality can draw on, but is limited in

its potential for use in time-constrained situations or wide use by all stakeholders in

the translation process.

Another important development in academic views on what quality

translations should endeavor toward is the functionalist view toward translation,

which grew out of the Skopos Theory proposed and refined by Vermeer and Reiss

(1984). Further complemented by Nord (1997) and her concept of loyalty on the part

of translators, this view provided the framework to view the purpose, or skopos, of a
translation as the “measuring stick by which translation quality should be measured”

(Angelelli & Jacobson 2009). While this approach eventually became subject to

criticism as to whether there can be only one objectively identifiable purpose for a

translation, it served as an important development in the evolving discussion on

translation quality, i.e. one more oriented toward the receiving side of translations.

(2) Post-2001 views on translation evaluation

As seen with Jamal Al-Qinai’s (2000) suggestion to employ a comprehensive textual

analysis for TQA, there was a shift away from the discourse of accuracy, as “a

paradigm which requires the translation to be accurate [on a lexico-semantic level]

faithful ‘to the source text’, objective, and impartial,” (Zhong, 2002, as cited in

Angelelli & Jacobson, 2009) and to recognize the varied contexts in which translation

is carried out. Malcolm Williams (2004) proposed an approach based on

argumentation theory, which has been taken up in certain sectors but is criticized by

House due to “the fact that it may not be the case that argument structure is important

for all types of texts” (House, 2015, pp. 19-20).

Grounded in the functionalist approach with its roots in skopostheorie, the

Colina (2008/2009) functional-componential model of TQA is important in that its

integrity and rigorous approach to designing, executing and reporting on the test

allows for continuous refinement and incorporation of real-life situations. It employs a

holistic assessment of the quality of a translation, compartmentalized into four

components: aspects of target language, functional and textual adequacy, non-

specialized content-meaning, and specialized content and terminology (refer to

Appendix 1). While the categories largely overlap with those of previous TQA
models, the assessment scheme is defined in non-technical terms and can be further

weighted to reflect the situation in which the translation proceeded, thus providing a

flexible template where even lay-persons can offer their assessment of quality in clear

terms and with an emphasis on what they consider to be important for a specific

project. Once the reliability of Colina's TQA model has been established sufficiently

along a wider range of multiple language pairs, genres and situations, it shows much

promise for feasible use for the reliable and facile evaluation of professional

translations.

It is interesting to note [shrinking interest in TQA studies from Western

academia, but surge in studies from Asia, in particular from China]

As a field well past the initial stage of establishing its legitimacy, Translation

Studies now needs concrete and feasible models to perform TQA, theoretical

constructs that can be well refined through interaction with real-world situations.

Academic efforts along these lines will strengthen the link between theory and

practice, thus adding real-world substance to theoretical pursuits, and “translating” the

fruits of academic study into a more accessible form for use by practitioners.

4. Methodology

(1) Description of data

The source texts (ST), translation texts (TT) and customer feedback for 32 cases

roughly spanning a two-year period starting from March 2013, where customers

lodged a complaint on the grounds of unacceptable quality at the HUFS Center of

Interpreting and Translation, were collected for analysis. The scope of data collected
for a single case can be described as follows: the ST and TT for the project, with

analysis focusing on certain “rich points” identified by the customer as particularly

problematic; all available business correspondence pertaining to the claim lodged by

the commissioning party, particularly that which provided insight into the views on

quality held by the customer, both express and implied; the results of a TQA analysis

performed by a group of three professional translators, based on the Colina (2009)

model; and in certain cases, the results of a quality assessment procedure performed

by the commissioning party itself. Thusly collected, it was hoped that the data would

paint a multi-dimensional picture of how the academia, professionals, and the

customer would assess the quality of the translation project at hand.

(2) Qualitative search for real-world implications

In many cases, and lacking a clear consensus on the exact parameters of translation

quality, it proved difficult to reconcile the views of all stakeholders to the translation

process. An exploration within the realm of professional translation must take into

consideration the extra-textual situations within which the translation and claim-

resolution process takes place, e.g. whether the translator was provided sufficient time

and motivation to do a proper job, and this adds an extra layer of uncertainty arising

from the complex behavior of the human agents influencing the process. Thus a

customer’s claim that the “overall quality” was “lower than hoped, particularly when

considering the price” at which the translation was commissioned, can be seen as a

provocation that must be refuted and the tools provided by the academia as weapons

to brandish against a perceived affront. Mindful of the potential that this problematic

dimension had to invalidate any meaningful discussions of product quality, the

researcher found it necessary to ground his views only on the data presented within

the study and strive for impassive evaluation to the best possible.

Another persistent concern was how the researcher could sufficiently distance

himself from the frequently charged atmosphere within which the claims had

progressed. While this report attempts to document all details pertinent to each

specific case, it will be necessary to confess the difficulties associated with

disregarding preconceptions that had formed during the claim-resolution process and

focus only on issues directly pertinent to the quality of the translation product. One

positive aspect was that quality assessments gained with the Colina (2009) model

yielded unexpected results in several cases, hinting at the usefulness of the model

itself as a valid tool to focus and operationalize the evaluator’s inevitably subjective

views, and at the validity of a TQA process that incorporates multiple perceptions,

thus painting a picture that is perhaps “fuzzy” in some aspects, but necessarily so

when describing a phenomenon predominantly linguistic in its nature and involved

with multiple human agents. As understood by other scholars, an attempt to reduce

such phenomena to “objective” data could yield a “positivist chimera, … [and is]

unlikely to provide the field of Translation Studies with much information of lasting

value or transferable worth and, therefore, would not be worth the effort” (Tymoczko,

1998, p. 5). This researcher also agrees with the view expressed by Temple and

Young (2004) that, within translation studies, “There is no neutral position from

which to translate and the power relationships within research need to be

acknowledged” (p. 164).

5. Case Studies
(1) Economics research institute, claim based on overall dissatisfaction with

translation product(s)

In this case, for which the customer necessarily remains anonymous, there were a

total of four (4) texts upon which the commissioning party had based its claim of low

quality. As an influential economics research institute, the customer was without

doubt in need of quality translations that would clearly convey the content to English-

speaking readers, but another demand that surfaced during the process of the claim-

resolution process was that the institute wished for the translation texts to read in a

polished manner, i.e. as if the report had been drafted in English from the beginning.

As most professionals working in the translation sector would probably agree, this is a

very high goal to aim at in the first place – one which would ideally be accompanied

by cooperation from the drafting stage, instead of a source text being prepared

according to the writing conventions of the source language (SL) and genre

conventions, then finalized before being translated into another language – and further

compounded by the evident disparity of the source and target languages. [insert quote

on relative difficulty working between English and Korean texts]. Another

translational challenge was presented by the density of the wording used in the ST.

The customer was translating a summarized version of the full report, and the wording

in Korean, already a language accommodating toward packaging strategies [insert

quote], had to be “unpackaged” using explicitation tactics to render the content clearly

in English.

Some rich points, as identified by the customer, are summarized below in

Table 1. For the purpose of convenient reference the four translations will be

numbered by the chronological order in which they were commissioned, as TT1, TT2,
 TT3 and TT4. Among these TTs 2 and 4 were performed by the same translator and

TT1 and TT3 each by a different translator, with all the translators holding a

professional status as graduates of the HUFS GSIT program and at least five years of

experience in the field.

Number Source Text (ST) Literal Back Translation Translation Text (TT)
(BT)

1 발전시설 부족 및 송배전 Chronic power shortage The electricity

전력 손실, 도전(盜電) arising from lack of power shortages caused by a
등으로 발생한 만성적인 generation facilities, loss of lack of power
전력부족은 매년 GDP 의 transmission/distribution generation facilities,
2%를 손실시키는 등 power, power theft, etc., power losses during the
파키스탄의 경제성장을 caused annual loss of 2% of transmission and
저해하는 주요 요인으로 the GDP [and acts as one of distribution process,
작용하고 있음. the major hindering factors to and power theft ate into
Pakistan’s economic growth1]. 2% of GDP each year.
Customer’s evaluation The expression “ate into” is not necessarily incorrect in
(researcher’s summarized itself, but it is awkward to see the expression “eat into”
translation from Korean) in an economic report. Should be substituted with [more
formal] expressions indicating increase, decrease, fall,
climb, etc.
2 인구 1 억 8,010 만 명인 If Pakistan, of a 180.1 million If Pakistan, a country
파키스탄이 정치적인 population, achieves political with a population of
안정을 이룩하는 데 stability and in addition 180.1 million carries
더하여 시장개방과 carries out pro-business out business-friendly
경쟁촉진 등 친기업적인 policies such as market policies (market
정책을 펼친다면, 우리 opening and competition opening and the
기업은 유망산업을 promotion, etc. there is a need promotion of
중심으로 파키스탄 진출을 for our companies to consider competition) in
보다 적극적으로 검토할 entering into Pakistan, addition to achieving
필요가 있음. centered on promising political stability,
industries, more actively. Korean companies
would need to seriously
consider entering the
Pakistani market,
especially its promising
industries.
Customer’s evaluation Too complicated sentence structure for one sentence.
(researcher’s summarized ([use of] parentheses, commas, if clauses)
translation from Korean)
3 파키스탄 정부가 [It] should be taken into It should be taken into
긴축재정을 펼치는 가운데 account that the Pakistani account that the new
투자지출 확대, 법인세 government is actively Pakistani government
인하 등 친기업적인 환경 establishing a pro-business is eager to create a pro-
조성에 적극적임을 environment such as by business environment
고려해야 함.

1 In the initial translation the translator chose to translate this portion into two sentences, and after the
customer’s comments were incorporated the portion was amended as “Chronic electricity shortages are
weighing down heavily on Pakistan’s economic growth. The country is losing 2% of its GDP each year
due to the electricity shortages caused by a lack of power generation facilities, power theft, and power
losses during the transmission and distribution process.”

Customer’s evaluation
(researcher’s summarized
translation from Korean)
4
5
Table 1. Summary of Rich Points
expanding investment (increasing investment
expenditures, lowering expenditures, cutting
corporate tax, etc., while corporate tax rates,
carrying out austere financial etc.), while running on
policies. a tight fiscal budget.
To deliver the meaning of the original, and in light of the
overall flow, the order of elements and the equivalents
for words can be changed according to the given
context. The sentence structure is difficult to understand.
Literal translation suspected.

The overall TQA score awarded by the evaluators was [insert results for TTs 1

through 4], while an example of the customer’s evaluation is provided below in Table

2, which has been translated from the original Korean (provided in Appendix 1). It

was observed that the scores awarded by the post-event evaluators largely coincided

with that by the customer’s internal evaluation, at least in a relative sense, with TT1

awarded the highest TQA score on the Colina (2009) model and also by the

customer’s evaluation, while TT3 received the lowest score by either measure.

Evaluation Score
Evaluation Category Comments
Lacking … Average … Good
1 Professionalism ① ② ③ √④ ⑤ Overall good, but should refrain
(Selection of high- from using the same expression
quality words/terms) (“amid” etc.) in the same
paragraph.
2 Faithfulness ① ② ③ √④ ⑤ Overall faithful
(Accurate delivery of
original content)
3 Readability ① √② ③ ④ ⑤ Many portions looks to be taken
(Accurate selection of directly from the Korean
terms and context flow) original, causing a fall in
readability. A concentration of
grammatically unsound
sentences in the conclusions
section, in particular, must be
revised.
4 Format/Style ① ② √③ ④ ⑤ Should adhere to original format.
(Overall composition Errors in team names, titles, etc.
and style, layout,
margins, etc.)
5 Misc. ① √② ③ ④ ⑤ Revision necessary
Table 2. Example of customer’s evaluation sheet

[Table 3. Comparison of TQA scores and customer’s evaluation]

(2) Financial institute-design company, claim based on readability issues

(3) Museum-design company, claim based on terminology issues

(4) Consulting agency, claim based on lack of subject knowledge

(5) Public research institute, claim based on accuracy and TL usage

(6) Other cases

6. Conclusion and Implications

- In light of research questions (1. Do the implicit and explicit evaluation by

customers coincide to a significant degree with those by professional evaluators, using

academic TQA models? 2. Does the Colina (2009) TQA model yield convincing

results in its application within the Korean professional translation market?), the study

indicates that 1. Yes, to a degree 2. Yes, quite so.

- The customer as evaluator: meaningful when aiming to address the gap between

academia-practice; problematic in that 1) “the customer” must be further defined

before offering operational results, 2) generally speaking the customer lacks the

necessary competences to perform reliable, valid evaluation, 3) the customer is

motivated by extra-textual and meta-linguistic considerations and evaluation made as

such should possibly be considered from a sociological standpoint.

- Limitations of current study + The current study could yield further implications

when accompanied by diachronic studies observing how/whether customer claims are

evolving in the Korean market for professional translation products. (후속, 공시적

연구로 보완)
7. References

– Al-Qinai, J. (2000). Translation quality assessment. Strategies, parametres and

procedures. Meta: Journal des traducteursMeta:/Translators' Journal, 45(3), 497-

519.

– Bowker, L. (2001). Towards a methodology for a corpus-based approach to

translation evaluation. Meta: Journal des traducteursMeta:/Translators' Journal,

46(2), 345-364.

– Brunette, L. (2000). Towards a Terminology for Translation Quality Assessment.

The Translator, 6(2), 169-182.

– Campbell, S. (2000). Critical Structures in the Evaluation of Translations from

Arabic into English as a Second Language. The Translator, 6(2), 211-229.

– Colina, Sonia (2008). "Translation quality evaluation: Empirical evidence for a

functionalist approach." The Translator 14.1, 97-134.

------ (2009). "Further evidence for a functionalist approach to translation quality

evaluation." Target 21.2, 235-264.

– Delizée, A. (2011). A global rating scale for the summative assessment of pragmatic

translation at Master’s level: an attempt to combine academic and professional

criteria. Perspectives on Translation Quality, Berlin/Boston, Walter de Gruyter, 9-

24.

– Drugan, J. (2013). Quality In Professional Translation: Assessment and

Improvement: Bloomsbury Academic.

– Fraser, J. (2000). The broader view: How freelance translators define translation

competence. BENJAMINS TRANSLATION LIBRARY, 38, 51-62.

– Gerzymisch-Arbogast, H. (2001). Equivalence parameters and evaluation. Meta:

Journal des traducteursMeta:/Translators' Journal, 46(2), 227-242.

– Hague, D., Melby, A., & Zheng, W. (2011). Surveying Translation Quality

Assessment. The Interpreter and Translator Trainer, 5(2), 243-267.

– House, J. (2001). Translation Quality Assessment: Linguistic Description versus

Social Evaluation. Meta: Journal des traducteurs, 46(2), 243.

------ (2014). Translation Quality Assessment: Past and Present: Taylor & Francis.

– Katan, D. (2009). Translation Theory and Professional Practice: a global survey of

the great divide. Hermes, 42(7), 111-153.

– Lauscher, S. (2000). Translation Quality Assessment. The Translator, 6(2), 149-

168.

– Maier, C. (2000). Introduction. The Translator, 6(2), 137-148.

– Martínez Melis, N., & Hurtado Albir, A. (2001). Assessment in translation studies:

Research needs. Meta: Journal des traducteursMeta:/Translators' Journal, 46(2),

272-287.

– Waddington, C. (2001). Different Methods of Evaluating Student Translations: The

Question of Validity. Meta: Journal des traducteurs, 46(2), 311

– Williams, M. (1989). The Assessment of Professional Translation Quality: Creating

Credibility out of Chaos. TTR : traduction, terminologie, rédaction, 2(2), 13.

------ (2001). The application of argumentation theory to translation quality

assessment. Meta, 46(2), 326-344.

------ (2009). Translation Quality Assessment. Mutatis Mutandis, 2(1), 3-23.

------ (2013). A holistic-componential model for assessing translation student

performance and competency. Mutatis Mutandis, 6(2), 419-443.

– 이향. (2010). 번역품질평가의 현황과 과제. 번역학연구, 11(2), 107-125.

– 장혜선. (2011). TES(Translation Evaluation System) 분석을 통해 살펴본

번역오류항목 가중치에 관한 연구 - 한일 번역물 분석을 중심으로.

통번역학연구, 15(1), 389-419.

– 조준형, 이영훈, 계명훈, 고여림, 김보현, 김지은, & 이상인. (2011).

병렬코퍼스를 활용한 번역평가 연구. 번역학연구, 12(3), 225-262.

[Appendix 1. Customer’s evaluation sheet]

insert cleaner, scanned version of Korean original