Professional Documents
Culture Documents
Perspectives Do Matter Joint Screen A Promising Methodology For Multimodal Interaction Analysis
Perspectives Do Matter Joint Screen A Promising Methodology For Multimodal Interaction Analysis
To cite this article: Béatrice Arend, Patrick Sunnen, Pierre Fixmer & Monika Sujbert (2014)
Perspectives do matter: ‘Joint Screen’, a promising methodology for multimodal interaction
analysis, Classroom Discourse, 5:1, 38-50, DOI: 10.1080/19463014.2013.859843
1. Introduction
This article focuses on methodological issues arising from the implementation of
multiple cameras and the resulting tool, ‘Joint Screen’, when studying interactions
between young peers and artefacts in their environment. We shall discuss how this
particular video-based analysis can enhance our attempts to grasp learning processes
in joint activities in a real-time re-production of multimodally co-constructed
interactions. We will focus on video recordings as they are produced and edited
within an ethnomethodologically and CA-inspired framework by tackling an
empirical case: two young children coping with an open-ended baking task in a
non-school context.
First, we shall give an overview of the theoretical framework we rely on to
define our unit of analysis. After presenting the methodological design we used to
construct and analyse our data, we shall foreground the qualities of a dynamic
reading of the four combined camera perspectives by conducting a multimodal
interaction analysis of an excerpt.1 Finally, we shall draw our conclusions and point
out how our design can generate new perspectives to grasp learning processes in
joint activities in different contexts.
2. Theoretical framework
We draw upon a sociocultural view on learning inspired by Vygotsky (1978, 1987)
to investigate learning processes among children2. This means that we focus on
how children interweave their meanings and their mediational means in order to
co-create artefacts. A particular regard is given to ‘the management of material
resources’ that support children’s joint activity (Crook 1995, 544).
In line with this approach, we consider object-oriented artefact-mediated joint
activities as unit(s) of analysis (Engeström, Miettinen, and Punamäki 1999). These
units are considered in their temporal and phenomenal depth and in relation to their
emergent context of occurrence. Children’s productive joint engagement in activity is
articulated in interrelated verbal and non-verbal utterances. Hence, we focus our anal-
ysis on social interactions between subjects, as well as on subject–object interaction.
The interactions take place in an area of both agreement and disagreement by means
of discussions that are organised not only by verbal resources but also by body move-
ments and gestures, by showing and exhibiting and by manipulating various artefacts.
In the case analysed here, two children are co-constructing a common shared
object (preparing dough, baking a cake), and the subjects’ orientation toward the
object largely shapes the activity. We intend to stress that the ongoing transformation
of the co-constructed object may be considered as the material instantiation of an
unpredictable transforming process that is generated by the children’s embodied
orientations to ‘what’s next’. According to this view, and relying on activity as
dialogically co-constructed in an arena of intercomprehension (Brassac 2000), the
intermediary object (here the dough) represents the semiotic materialisation of the
children’s mutually responsive speaking and acting and supports the intermediation
in the area of mutual understanding.
Accordingly we consider learning as a distributed, ongoing social process
(Jordan and Henderson 1995), and we refer to the Situated and Distributed
Cognition approach to emphasise that human cognition cannot be solely attributed
to individual heads. Rather, it lies in the relations between subjects and between
subjects and material objects in a material world (Clark 1997; Conein 2004).
In this sense, activity theory (AT) points out the central role of mediation in
human thought and action. For the purpose of our analysis, we adhere to an
AT-related ‘broadened view’ on mediating resources. We adopt a multimodal
approach to analyse joint activity. Mediated activity involves various resources. We
focus on talk, gestures, gaze, body postures and body movements, as well as spatial
and material resources mobilised by the subjects, in order to elaborate a common
object and to co-create artefacts (Mondada 2011). Thus, we consider activity as
being built by subjects through the simultaneous and continuous use of multimodal
mediating resources (Streeck, Goodwin, and LeBaron 2011). Particular regard is
given to gestures as having an organisational role in interaction (Visser 2010).
Our analyses of learning processes in joint activities also ‘focus upon
participants’ access to shared understanding’ (Crook 1994, 155). To capture the
latter we refer to Schütz (Schütz and Luckmann 2003), who emphasises the impor-
tance of ‘the reciprocity of perspectives’ in interactions (taken for granted in the
common-sense world). In describing the interchangeability of ‘here’ and ‘there’, the
reciprocity of perspectives emerges as a necessary condition for mutually shared
understanding. It means that objects of experience are intersubjectively available and
that subjects mutually grasp the simultaneity of each other and continuously and
40 B. Arend et al.
methodically coordinate their perspectives. Below, we shall set forth the scope of
the conception of perspectival reciprocity for our methodological design.
Conein (2004) argues that humans’ ability to establish and to sustain mutual
attention as well as joint foci of attention is a prerequisite to engaging in joint
activities. According to him, social – i.e. mutual – perception and the ability to
coordinate attention are key concepts for object-oriented joint activity.
Tomasello et al. (2005, 675) point out that children have ‘a species-unique
motivation to share emotions, experience, and activities with other persons’. According
to them, joint activity involving socially coordinated interactions and mutual under-
standing ‘requires a motivation to share these things in interaction with others’ (ibid.
676). Thus, we can assume that the various semiotic resources enacted by the subjects
in order to build activity also contribute to the configuration of their relationship.
Relying on the previous concepts, we shall investigate how subjects do mutual
understanding and coordination, i.e. how they communicate mutual joint
engagement in activity and make shared understanding mutually recognisable. We
use ethnomethodological inquiry (Garfinkel 1967), which aims at describing the
methods people use to account for their own actions and those of others. Garfinkel
pointed out that mutual understanding in all situations requires continuous attention
and competent use of shared methods of organising action for its achievement.
According to ethnomethodology, each action must exhibit an order that is account-
able for other subjects in order to be meaningful, and the methods for producing
mutually intelligible actions require mutual orientation and sustained trust.
Our approach requires visual access to the activity in situ and our access as
observers is ‘located in whatever discourse and action we are able to witness’
(Crook 1994, 176). Hence, the challenge is to grasp how the children methodically
coordinate and temporally organise their baking activity and to capture shared
understanding in its multimodally accountable instantiation. Accordingly, our
methodological video-based design contributes to distinguishing and identifying the
specificity of the (multimodal) utterances of every participant by making available
their coordination and their mutual synchronizations3.
3. Data construction
3.1. Setting
In the present study we focus on how children organise their joint activity. We set
up a situation in which the participating children had to cope with an open-ended
baking activity, giving them opportunities to orient their action(s) to one another. As
a location we identified a pedagogical farm with appropriate facilities such as a large
professional kitchen and a multi-purpose room. The two children engaged in the
baking activity are called Mona (6 years old) and Lisa (4 years old) (both names are
pseudonyms). In order to give them the opportunity to become familiar with the set-
ting, we visited the farm and prepared pizza dough with them. During that time we
also introduced the children to the recording devices by providing them, on request,
with explanations about the equipment. In this way we intended to familiarise them
with the presence of observers using cameras before the beginning of the actual
recording. The recording is 52 minutes long and starts when the research assistant
asks the two girls to bake a cake according to their own ideas or by relying on a
visual recipe that is at their disposal along with many ingredients and kitchen uten-
sils. The videotaped activity unfolds as the children organise ‘preparing dough’. We
Classroom Discourse 41
shall see that they do so by mobilising multimodal resources and constructing inter-
twined foci of attention on various artefacts.
adjoining professional kitchen. Thanks to the two follow cameras, at least one of the
researchers could follow the moving child without us losing track of the other.
Relying on several cameras to capture the ‘same’ event simultaneously from
different camera perspectives was an integral part of our research design from the
beginning. We therefore took into account Schütz’s assumption of the ‘interchange-
ability of standpoints’: ‘since we share a common external reality but encounter it
differently depending on our situations, if we were to change places, what is there
for you now would then become available to me, and vice versa’ (Wilson 2012,
210–11). Accordingly, we soon concluded that we had to attend to the different
perspectives at the same time in order to deepen the analysis of multimodal process
and to enhance our understanding of the ongoing interactions.
only points out that the perspectives of each observer position are interchangeable,
but also makes them simultaneously available to the analysts. Thus, joining the four
perspectives within one screen generates an ‘expanded-around’ view of how
children mutually enact and make relevant diverse multimodal resources in order to
realise a common shared object: baking a cake. In a sense, it is an attempt to
reconstruct the interactional dynamics of the (baking) activity in 3D.
4. Developing analysis
For this paper we have selected an excerpt of 23 seconds that we shall present in
three extracts. It takes place near to the beginning of our recording. The two girls
are relying on the visual recipe, and they have just put yoghurt and one egg in the
bowl (see Figure 3).
Through our analysis we shall point out how the children organise the unfolding
of the baking activity (and in that way the transformation of the dough) by
constructing divergent but intertwined foci of attention on various artefacts. The
joint screen makes available two relevant interactional spaces as mutually consti-
tuted in the reciprocity of the children’s perspectives. The dialogic cross-reading of
the four joint camera perspectives allows us to capture the children’s productive
joint engagement in activity in its temporal and phenomenal depth with regard to its
emergent re-configuration in space.
44 B. Arend et al.
Lisa gazes at the recipe, then looks around toward the desk for material resources
(in this case the whisk). By verbally addressing Mona (now is stirred Mona),8 Lisa
initiates the next step of the baking activity. By shaping her spoken utterance
(injunctive sentence, apostrophe), Lisa indicates ‘what is next’ to be done and that
Mona’s co-participation is requested. The utterance may also be considered as Lisa’s
verbally instantiated reading comprehension of the illustrated recipe. She shows that
she is able to use an ‘appropriate’ discourse to ‘translate’ the recipe at hand and
expresses her situated understanding of the artefact.
Meanwhile Mona leaves the interactional space in which her co-participant is
still engaged and walks to the adjacent kitchen. She holds eggshells in her left hand
(we see their yolks in the bowl). She addresses the research assistant who is in the
kitchen and asks where she can dispose of the eggshells (where can I throw away).
By carrying the eggshells to the kitchen she extends the previous step of the baking
activity (adding an egg) to another interactional space. At the same time she shows
that she aims at completing the previous step, by looking for a way to get rid of the
eggshells.
Classroom Discourse 45
The joint screen allows us not only to see the girls’ multimodally embodied
orientations to different foci of attention, but also displays synchronically both girls’
‘here’ and ‘there’. The combined viewing of the four joint perspectives gives us
access to the two interactional spaces as intertwined through embodied interactions.
Extract 2
Lisa now looks at the whisk and grabs it. Mona is still preoccupied by the eggshells
and in the kitchen she points at a bag.
Lisa is about to put the whisk into the bowl but she does not yet stir. The yolk in the
bowl is still whole. Mona says ‘perhaps here’, identifying the bag as a potential dustbin.
The research assistant answers Mona’s request to dispose of the eggshells
(I bring you a dustbin there). Synchronically, Lisa verbally repeats the next baking
step (now is stirred) and then turns to her left, looking in the direction to which
Mona moved. By addressing her multimodally, Lisa displays that she considers
preparing dough as a joint activity and that she identifies Mona as a co-participant.
Lisa taps on her left hand with the whisk, thereby validating this artefact as useable
and appropriate for mediating stirring. The research assistant and Lisa are
synchronically ‘voicing’ different foci of attention and contribute, in the respective
interactional spaces, to advancing the unfolding transformation of the object dough.
Mona (still keeping the eggshells in her left hand) expresses by verbal utterance
(Lisa I’m just coming) and by body movement that she will join Lisa:9 her head is
turned sideways to the back of the space, toward the research assistant who is carrying
46 B. Arend et al.
Extract 3
Classroom Discourse 47
Lisa turns back to the bowl and places the whisk in it, repeating one last time it is
stirred. But again (see Extract 2), she does not stir. Mona (still holding the eggshells
in her left hand) approaches.10 Her gaze is oriented straight ahead while Lisa gazes
at the dough. Mona is back in the baking area but the two girls do not share a
common focus.
Then Lisa lifts her head and gazes at the approaching research assistant who
walks around the panel in the symmetrically opposite direction to Mona. The
assistant brings the bin. Synchronically, Mona looks at the recipe on the desk. Joint
Screen provides here an around-view of how parallel unfolding trajectories of
embodied interaction may be intertwined. It allows us to analyse how in the ongoing
interactional organisation under discussion two different foci of attention are inter-
woven and related by a ‘double crossing symmetry’. Indeed, we can see that Mona
proceeds to a re-focalisation on what’s next to do by gazing at the recipe. We may
put forward here that this is a kind of assessment of Lisa’s elicitation to stir – that
is, of Lisa’s reading comprehension (see Extract 1). Synchronically Lisa orients her
attention to the assistant bringing the bin. The two girls are in some way momentar-
ily cross-changing the respective previous foci of attention. The joint screen
provides a view on the interactional spaces here overlapping.
Re-orienting to the construction of the object dough, Mona stops walking and
turns toward the bowl. She prompts Lisa to stir (Lisa do stir). Thus she validates
Lisa’s elicitation to stir as the next step of the baking activity. Simultaneously Lisa
begins moving the whisk slowly and displays responsive action to Mona’s elicitation.
Mona and Lisa gaze at the dough. The two girls share a common focus of
attention, but doing stirring is not yet artefactually enacted by both girls. Lisa
manipulates the whisk while Mona still holds the eggshells. Mona’s yet-unreached
purpose of disposing of them aims at completion.
As soon as the assistant has put the bin down next to the table, Mona turns to
the right, walks to the bin and throws the eggshells inside while Lisa continues
stirring and in that way transforming the dough: the yolk is actually slightly mixed
with the yoghurt.
The closing of the eggshells sequence is achieved when Mona closes the lid of
the bin. Completion is also displayed by the fact that Mona moves back to Lisa and
gazes toward the bowl (the assistant returns to the adjacent kitchen).
Both girls then hold the whisk together to mix the yoghurt and the egg. Since
Lisa’s first call to stir, the dough has taken on a different appearance. According to
our approach the intermediary object dough can be considered as a material account
of the transformatory baking process conducted as joint activity by the children.
Indeed, Mona and Lisa symmetrically cross their respective left and right arms and
hands one above the other in a harmonising movement, jointly enacting the whisk
and so transforming the dough. They both hold the bowl with their other hands.
The joint screen makes observable the co-construction of the common object as
such in the sequential flow, as well as in the coordinated synchronicity of embodied
interaction.
Acknowledgements
We thank the Fonds National de la Recherche Luxembourg for funding this research under
the CORE scheme.
Notes
1. The video recordings used in this article were made in the context of the research
project ‘COLEAP-Collaborative Learning among Peers’ (C10/LM/783921,
2011–2014). The project is supported by the Fonds National de la Recherche
Luxembourg under the CORE funding scheme.
2. See COLEAP.
3. Through repeated viewings, researchers reconstruct the meaning of the participants’
conduct – so (contrary to the participants) they do already know what is going to
happen next.
4. Knowing that although moving images make a strong claim on reality, they are not a
‘transparent window opened onto the world’ (Mondada 2009a, 75).
5. See Mondada (2009a) for a comprehensive discussion of the uses and functions of ‘split
screen’ in different domains.
6. The black frame of the image of camera 2 is due to the circumstance that the recording
mode was accidentally set to a 4:3 instead of a 16:9 ratio.
7. We are, however, aware that different areas of an image (left and right, top and bottom,
centre and margin) are endowed with specific information values (see Kress and
Leeuwen 2006).
8. Mona and Lisa are two German girls who are growing up and going to school in Luxem-
bourg. During this excerpt they speak German among themselves and Luxembourgish
with the research assistant. For reasons of readability, we chose to provide only the
English translation, which stays as close as possible to the original utterances.
9. Note that Lisa stops tapping with the whisk at the moment when Mona is finishing her
utterance.
10. Mona comes back the same way she left, although she could have followed the research
assistant and walked around the panel in the symmetrically opposed direction.
Classroom Discourse 49
Notes on contributors
Béatrice Arend is a senior lecturer in the Institute of Education & Society at the University
of Luxembourg. Her research focuses on interactional processes in mediated joint activities,
multimodally enacted literacy practices, conversation analysis.
Patrick Sunnen is an associate professor in the Institute of Education & Society at the University
of Luxembourg. His research focuses on learning processes among peers, visualizing learning
processes, interaction analysis.
Pierre Fixmer is a senior lecturer in the Institute of Education & Society at the University of
Luxembourg. His research focuses on interactional processes in arenas of mutual understand-
ing, tracing processes of joint activities at a micro level, intermediary object.
Monika Sujbert is a research associate in the research project ‘Collaborative Learning among
Peers’ (COLEAP) at the University of Luxembourg. Her research focuses focuses
References
Arend, B. 2010. “Contribution d’un Outil De Transcription Dans La Mise En Mots De
l’analyse d’un Processus De Conception Collaborative observé Dans Sa Profondeur
phénoménale.” In Recherches Qualitatives. Logiciels Pour l’analyse Qualitative:
Innovations Techniques et Sociales 9: 95–108.
Brassac, C. 2000. “Intercompréhension Et Communiaction®.” In Modèles Du Discours En
Confrontation, edited by A. C. Berthoud and L. Mondada, 219–228. Berne: Peter
Lang.
Clark, A. 1997. Being There: Puting Brain, Body and World Togehter Again. Cambridge,
MA: MIT Press.
Conein, B. 1990. “Cognition située Et Coordination De l’action, La Cuisine Dans Tous Ses
états.” Réseaux 8 (43): 99–110.
Conein, B. 2004. Les Sens Sociaux. Trois Essais De Sociologie Cognitive. Paris: Economica.
Crook, C. 1994. Computers and the Collaborative Experience of Learning. London:
Routledge.
Crook, C. 1995. “On Resourcing a Concern for Collaboration within Peer Interactions.”
Cognition and Instruction 13 (4): 541–547.
Engeström, Y., R. Miettinen, and R.-L. Punamäki, eds. 1999. Perspectives on Activity Theory.
Cambridge: Cambridge University Press.
Garfinkel, H. 1967. Studies in Ethnomethodology. Cambridge: Polity Press.
Goodwin, C. 1994. “Professional Vision.” American Anthropologist 96 (3): 606–633.
Gregori, N., and P. Fixmer. 2013. “Ce Que l’effaçage Dit De La Trace.” Intellectica 59:
243–265.
Heath, C., J. Hindmarsh, and P. Luff. 2010. Video in Qualitative Research. Analysing Social
Interaction in Everyday Life. Los Angeles, London, New Dehli, Singapore, Washington
DC: Routledge.
Jordan, B., and A. Henderson. 1995. “Interaction Analysis: Foundations and Practice.”
Journal of the Learning Sciences 4 (1): 39–103.
Kress, G., and T. van Leeuwen. 2006. Reading Images. The Grammar of Visual Design. 2nd
ed. London and New York: Routledge.
Mondada, L. 2009a. “Video Recording Practices and the Reflexive Constitution of the
Interaction: Some Systematic Uses of the Split-Screen Technique.” Human Studies
32 (1): 67–99.
Mondada, L. 2009b. “Video Recording as the Reflexive Preservation and Configuration of
Phenomenal Features for Analysis.” In Video Analysis: Methodology and Methods, edited
by H. Knoblauch, B. Schnettler, J. Raab and H.-G. Soeffner, 51–67. Frankfurt/Main:
Peter Lang.
Mondada, L. 2011. “The Organization of Concurrent Courses of Action in Surgical
Demonstrations.” In Embodied Interaction Language and Body in the Material World,
50 B. Arend et al.