See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/323768888

Data collection and analysis in developmental L2 pragmatics research:

Discourse completion test, role play, and naturalistic recording

Chapter · March 2018

DOI: 10.1075/lllt.51.02tag

CITATIONS READS

8 2,680

1 author:

Naoko Taguchi
Northern Arizona University
125 PUBLICATIONS 3,214 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

MLJ special issue: Teaching and Learning Pragmatics in the Globalized World View project

Second Language Pragmatics: From Theory to Research View project

All content following this page was uploaded by Naoko Taguchi on 15 March 2018.

The user has requested enhancement of the downloaded file.

This is the accepted version of the manuscript in In A. Gudmestad & A. Edmonds (eds.),
Critical reflections on data in second language acquisition (2018). Philadelphia: John
Benjamins. This paper can be used for non-commercial purposes.

Data collection and analysis in developmental L2 pragmatics research: Discourse

completion test, role play, and naturalistic recording
Naoko Taguchi
Carnegie Mellon University

Abstract
This chapter compares common data collection and analysis methods used in longitudinal
studies in second language pragmatics. Specifically, three data collection methods used to
document speech act development are compared: discourse completion tasks (DCT) (written
and spoken), role plays, and naturalistic recordings. I critically discuss benefits and
shortcomings of these data collection methods, in response to three questions: (1) What is the
nature of speech act data collected in each method?; (2) What kind of analysis is performed
on the data collected?; and (3) What conclusions are drawn as evidence for speech act
development? The paper concludes with critical reflections and directions for future
investigation into pragmatic development.

Key words: research methods, L2 pragmatics, DCT, role play, naturalistic recording

Introduction
Pragmatics entails a complex interplay among a language, a language user, and a
language use context. For second-language (L2) learners, pragmatic competence means
developing abilities to perform a communicative function effectively and appropriately in a
social context. Because of the socially grounded nature of pragmatic competence, a challenge
for the researchers is to collect data that closely reflect L2 learners’ language use in social
contexts. Because social situations are the product of a real world outside of a laboratory,
experimental data elicited through a researcher-made instrument are often criticized for their
lack of authenticity and correspondence to real-life language use. At the same time, a
shortcoming of naturalistic data collected in a real-world situation is their generalizability due
to the small number of available data points and the restricted range of contexts where data
can be collected. The data points are often difficult to compare to each other because of the
immense variation involved in the contexts of data (e.g., speaker backgrounds, topics,
settings) (for a review of research methods, see Kasper & Rose, 2002; Taguchi & Roever,
2017)
The present chapter critically discusses the advantages and disadvantages of elicited
and naturalistic data when documenting pragmatic development. Specifically, I will compare
three data-collection methods commonly used in L2 pragmatics research: discourse
completion tasks (DCT) (written and spoken), role plays, and naturalistic recording of data.
On the basis of a systematic literature review, I have located longitudinal studies that used
these data-collection methods to examine L2 learners’ development in speech acts. With
speech acts as a common thread, I will compare different data-collection methods in terms of
(1) the nature of speech act data collected, (2) the types of analysis performed on the data, and
(3) the conclusions drawn for the development of speech act abilities. I will conclude the
paper with critical reflections about each method, focusing on the kinds of claims that each
method can or cannot generate given its strengths and weaknesses. In the following section I

will first present common definitions of pragmatic competence and then review data
collection methods used to examine this competence.

Pragmatic competence: Evolving definitions

Pragmatics studies a connection between a linguistic form and a social situation where
the form is used, and how such a connection is realized when people perform a
communicative function. The connection among form, function, and context of use has been
reiterated over time in a range of definitions of communicative competence. The origin of the
concept of communicative competence dates back to Dell Hymes’s framework, which
appeared five decades ago. Hymes (1972) proposed the two-dimensional definition of
language knowledge, with grammatical knowledge on one side and sociocultural knowledge
on the other. These two dimensions jointly determine how we use language appropriately in a
social context.
Hymes’ framework has advanced L2 research in two distinct directions. One is the
emergence of L2 models of communicative competence (Bachman & Palmer, 2010; Canale &
Swain, 1980). These models emphasize that a successful communicative act draws on four
distinct yet interrelated competences (grammatical, pragmatic, discourse, and strategic). The
pragmatic component involves functional knowledge (e.g., how to refuse someone’s
invitation) and sociolinguistic knowledge (e.g., which linguistic forms to use to refuse in
which situation). These two knowledge bases together enable people to use appropriate
utterances when performing a communicative function. The functional and sociolinguistic
knowledge distinction largely parallels the distinction between pragmalinguistics and
sociopragmatics (Thomas, 1983). The former refers to linguistic resources for performing
language functions in a given situation, while the latter enables us to understand features of
context and to select the most appropriate resource in context.
Another direction informed by Hymes’ framework is the concept of interactional
competence. Drawing on Hymes’ ethnography of speaking, Hall (1995) emphasized
interaction as a goal-oriented, situated activity that draws on participants’ linguistic and
interactional resources. Elaborating on Hall’s framework, Young (2002, 2008, 2011)

identified specific resources of interaction, such as knowledge of register-specific linguistic

forms, speech acts, topic management, turn-taking, and repair. Interactional competence
views these resources as shared among participants in their process of joint meaning making.
Hence, unlike models of communicative competence, a communicative act is not denoted in
the one-to-one correspondence among an utterance, force, and context of use. Rather, it is co-
constructed and negotiated among participants and emerges during the course of interaction.
These two divergent trends stemming from Hymes’ framework indicate a two-layered
definition of pragmatic competence. For one, pragmatic competence involves knowledge of a
form-function-context relationship (i.e., which forms to use to perform what communicative
functions in what social contexts). However, the form-function-context relationship is not
stable or pre-determined; rather, it is emergent and contingent upon unfolding interaction. We
adapt our ways of speaking corresponding to changing context: the change in the interlocutors’
familiarity, attitudes, affect, and directions of discourse. Hence, interactional competence adds
another layer to the definition of pragmatic competence, namely the ability to adapt to the
changing course of interaction and to achieve a communicative goal in collaboration with
interlocutors.
In addition to adaptability, the concept of learner agency has recently influenced our
understanding of pragmatic competence. LoCastro (2003) defines agency as a self-reliant
capacity that works with volition to bring about an effect on one’s behavior. Learners are
active agents who make their own linguistic choices and create social positions for themselves.
Learners do not blindly adhere to target pragmatic norms. When the norms contradict their
desired social identity, they can choose to express their agency by resisting language use
widely practiced under the community norm. For example, learners of Japanese in Ishihara
and Tarone (2009) and learners of Korean in Kim & Brown (2014) resisted using honorifics
in addressing seniors because they valued American egalitarianism and wanted to project the
image of a friendly and casual self. These findings clearly illustrate the role of agency in
pragmatic performance. Knowing the normative form-function-context relationships and
applying the knowledge to interaction is one thing, but deciding whether or not to implement
the knowledge in a community is a different matter.

As illustrated above, the concept of pragmatic competence has evolved over time. The
knowledge of relationship among forms, functions, and contexts of use (functional and
sociolinguistic knowledge) is the central aspect of pragmatic knowledge, but this knowledge
alone does not sufficiently describe pragmatic competence. Equally critical is the ability to
adapt one’s linguistic behaviour to changing contexts and construct a communicative act in
social interactions. A further extension of pragmatic knowledge is that of learner agency.
Even if learners possess pragmatic knowledge that underlies community norms, actual
implementation of the knowledge is a reflection of learners’ subjective decisions. Depending
on how they want to present themselves in relation to others, learners accommodate or reject
the normative language use, potentially opting not to use the pragmatic knowledge they
possess.
Knowledge of linguistic forms and contextual elements is the primary layer of
pragmatic knowledge, but the implementation of the knowledge involves another layer –
learner agency. On the other hand, context is dynamic and constantly changing, which calls
for another layer of pragmatic competence – adaptability. Learners need to be able to detect
subtle contextual changes corresponding to the course of interaction, and to align with the
changes by using their linguistic and interactional resources. Figure 1 offers a visual
representation of these dimensions of pragmatic competence.

Adaptability Agency

Form

Function Context

Figure 1. Illustration of dimensions of pragmatic competence

These multiple dimensions of pragmatic competence compel us to critically reflect
on what counts as pragmatic development. When pragmatic competence is viewed as
knowledge of form-function-context relationships, development can be considered as the
learners’ expansion of their linguistic repertoire for performing a communicative act. In the
course of development, learners’ initial tendency to adhere to simple, one-to-one
correspondence between form and function (e.g., asking a favor with ‘could you’ + verb) is
gradually replaced by the expansion of the linguistic repertoire to encode a communicative
function (e.g., differentiating multiple expressions for asking a favor, such as ‘could you’ +
verb and ‘I wonder if’ + verb).
Under the layer of adaptability, pragmatic development can be defined as the learners’
understanding that the form-function relationship is dynamic and contingent upon the
unfolding course of the discourse. Development can be defined as learners’ increasing
abilities to monitor subtle changes in discourse (e.g., speaker relationships, affect, and
attitudes) and adapt to the changes on the fly as they interact with their interlocutors.
 Finally, the layer of agency presents a very different account of what development
actually means. When agency is considered, development may not be signaled in terms of
how closely learners approximate native-speaker norms. Quite differently, a learner’s decision
not to conform to the normative practices can be viewed as development, because pre-
requisite to such a decision is their understanding of what normative practices mean. Based on
that understanding, learners decide to conform or not to conform to local norms. Hence,
learners’ self-expression (or agency) reflects their understanding of community norms and
how they want to situate themselves within local practices.

Definitions of pragmatic competence and methods of examination

As I described in the previous section, different dimensions of pragmatic competence
generate varying views regarding what counts as pragmatic development. These divergent
views are also reflected in the methods used to examine development of various dimensions.
Knowledge of form-function-context relationships
Knowledge of form-function-context mappings has been examined using structured
elicitation tasks, such as a discourse completion task (DCT), a multiple-choice questionnaire
(MCQ), and an appropriateness judgment task (AJT). A written DCT has been used
extensively to examine L2 speech acts. The original DCT (Blum-Kulka, 1982) presents a
situational scenario followed by a dialogue that has at least one open-slot turn to be completed

by participants with a target speech act utterance. Participants’ pragmatic knowledge is

inferred by the extent to which speech act utterances fit in the scenario in terms of the level of
directness and formality. More recently, researchers have used spoken DCTs to analyze
features of speech (e.g., pause length, speech rate) in addition to speech act utterances. Like
the DCT, the MCQ assesses speech act knowledge by presenting participants a situational
scenario with a prompt for a response. However, the MCQ does not have an open slot for
participants to fill in an utterance. Instead, it provides a list of alternative forms from which
participants select. Unlike the MCQ, in an AJT, participants read the target speech act
utterance and judge its appropriateness by responding to a yes/no question or using a Likert
scale. All these instruments are designed to assess participants’ knowledge of a linguistic
form and contextual factors associated with the form. In order to produce, select, or evaluate a
form’s appropriateness, participants need to know the form and its function, as well as its
associated contextual values, and work out whether the form is appropriate in a given
situation.
Adaptability in interaction
Adaptability has been examined via a variety of methods used to collect interactional
data. Those methods include role plays, naturalistic conversations, elicited conversations, and
computer-mediated communication. Role plays involve an interaction played out by two or
more people based on a situational scenario. By specifying roles and settings in a situational
scenario, researchers can examine how contextual factors (e.g., interlocutor relationship, topic,
and setting) affect participants’ co-construction of a communicative act. Researchers can also
analyze how contextual factors change corresponding to the shifting course of interaction and
affect participants’ pragmatic acts as a result.
Naturalistic conversation data are usually collected via audio-/video-recording in a
real-world setting with or without the presence of the researcher. Participants are usually
aware of being recorded while engaging in real-life tasks. Using discourse analysis or
conversation analysis techniques, researchers transcribe and analyze a conversation to reveal
how certain linguistic forms occur over conversational sequences and serve as resources for
participants as they jointly construct a pragmatic act. Although audio-/video-recordings of
1

face-to-face conversations have been the common sources of data, recently, researchers have
started using digital technology as a tool for collecting interactional data (e.g., social
networking, online gaming, and video conferencing; see Taguchi, 2015, for a review).
Unlike a naturalistic conversation, an elicited conversation means “any conversation
staged for the purpose of data collection” (Kasper & Rose, 2002, p. 84). Unlike role plays,

1
Conversation analysis evolved from the work by Sacks, Schegloff, and Jefferson and allows researchers to

examine how a social action is constructed in a sequential organization of talk. It uses strict transcription systems

for analysis (Sacks, Shegloff, & Jefferson, 1974). In contrast, discourse analysis (Brown & Yule, 1983) is not

bound to standard transcription systems. In addition, it is used to analyze a wider range of text than conversation

data, including written narratives and public discourse.

participants do not assume specific roles assigned by the researcher. Instead, they are
instructed to engage in a conversation based on a topic or task assigned by the researcher.
They might be asked to have a casual conversation on routine topics, or they might be asked
to have a discussion on controversial topics prepared by the researcher. Unlike a naturalistic
conversation, an elicited conversation is a conversation for research purposes. As such, the
researcher has a certain degree of control over the type of data being collected (e.g., topic,
setting). Similar to naturalistic conversations, discourse analysis or conversation analysis is
typically performed on elicited conversation data to reveal participants’ abilities to jointly
construct a pragmatic act.
Learner agency
A classic study by Siegal (1996) used ethnography with data triangulations
(naturalistic conversation data, field observations, and interviews) to illustrate how four
European women in Japan rejected the community norms of gendered language practice to
express their desired identities as independent professionals. Two decades after Siegal’s work,
the current trend in studies dealing with learner agency is the continued use of the mixed-
methods approach. Current studies typically analyze performance data to document learners’
pragmatic language use, combined with introspective data to gain insights into learners’
perspectives on their linguistic choices. For example, researchers have used a DCT to elicit
learners’ knowledge of speech acts and then have conducted follow-up interviews to explore

learners’ insider perspectives to determine why they responded the way that they did (Ishihara
& Tarone, 2010; Kim, 2014). By comparing learners’ pragmatic performance and their
perceptions about the performance, researchers explore whether the knowledge elicited in the
data reflects learners’ intention to accommodate or diverge from the target-language norms.
When deviation from normative language use is found, researchers can examine whether it is
due to learners’ lack of knowledge of L2 norms or due to their conscious resistance to these
norms.
In summary, pragmatic competence is best described as a multi-layered construct.
The dimensions characterizing the competence include (1) linguistic and sociocultural
knowledge of what forms to use in what context; (2) interactional abilities to use the
knowledge in a flexible, adaptive manner corresponding to changing context; and (3) agency
to make one’s own choices of whether or not to implement the knowledge in relation to others
in the community. These three dimensions inevitably bring about different understandings as
to what counts as evidence for pragmatic development and how to assess such development.
In the rest of the paper, I take a closer look at this connection among the construct of
pragmatic competence, methods used to examine the construct, and evidence of pragmatic
development by reviewing existing studies. Based on a literature search using multiple
databases, I have located longitudinal studies in L2 pragmatics published to date. From this
pool of studies I review 25 studies that examined the same pragmatic construct using different
methods. I compare studies in terms of the nature of the data collected, types of analysis
performed on the data, and evidence drawn for the development of pragmatic competence.
Several papers have provided an overview of research methods in pragmatics over
time (e.g., Félix-Brasdefer, 2010; Kasper, 2000; Kasper & Dahl, 1991). The most recent
review is found in Taguchi and Roever (2017). In addition, a book dedicated to research
methods in L2 pragmatics is underway (Culpeper, Taguchi, & Mackey, in preparation). This
chapter is different from those reviews in that it adapts a critical synthesis of all the
longitudinal studies located using a systematic literature search in order to explore methods
used in those studies (see the next section). In addition, the chapter presents an explicit
connection between definitions or operationalizations of pragmatic development on the one

hand and methods in data collection and analysis used to illuminate L2 development on the
other hand. By doing so, this chapter intends to enhance the understanding of what pragmatic
development means and how this knowledge is reflected in the types of data and analysis.

Literature review: Methods

In order to locate existing longitudinal studies in L2 pragmatics, I first went back to
my own synthesis paper (Taguchi, 2010). That paper used electronic bibliographic searches to
locate all longitudinal L2 pragmatics studies published before 2009. Using databases (e.g.,
Linguistics and Language Behavior Abstracts [LLBA], Education Resources Information
 Center [ERIC]) with various search terms (e.g., pragmatics, second language, development), I
found 21 studies for this paper that met the eligibility criteria in (1):
1. a. The study observed the development of participant(s) over a period of time.
b. The study examined the development of specific pragmatic features.
c. The study chronologically documented development and change by analyzing
learner data collected systematically over time (e.g., pretest and posttest results,
linguistic analysis).
d. The study did not involve instructional intervention or other types of training.
e. The study observed participants of secondary or post-secondary school age.
The same search process was conducted again in 2017 when I was writing this paper
in order to locate studies published after my review. I conducted two rounds of searching: one
using LLBA, which yielded 234 studies, and another using ERIC, which yielded 143 studies.
After cross-checking investigations and applying the criteria above and removing any
duplicate studies, I found 16 unique studies from the two databases. In addition to the
database search, I manually searched review articles, books, and handbooks and found nine
additional studies, which yielded a total of 25 studies.
When combined with the 21 studies in my synthesis, a total of 46 longitudinal studies
remained as primary studies. In order to create a corpus of studies that can be compared, I
looked for studies that examined the same pragmatic features using different methods (i.e.,

DCT, role play and naturalistic data). Because speech acts were most represented across
different data collection methods, I decided to focus on the studies that uncovered changes in
L2 learners’ speech-act behaviors. I found a total of 22 speech-act studies, which are
categorized into four groups: investigations that used (1) a written DCT, (2) a spoken DCT,
(3) a role play task, and (4) naturally or semi-naturally collected data (naturalistic interaction
and semi-naturalistic interaction based on the setting and topic pre-arranged by the researcher).
The studies were coded for target language, sample size, target speech act, data-collection
methods, participants, context of study, study length, and frequency of data collection (i.e.,
data points). Table 1 displays the studies.

View publication stats