Writing, screenshot or drawing? Multimodal transcription of spontaneous web-based interactions

This paper presents problems in, and possible solutions for, web-based video analyses, drawing on a video corpus of over 300 spontaneous web-based video interactions. Issues in video analysis and interpretation have been amply documented (Thibault 2000; van Leeuwen, Jewitt, 2001; Norris 2004; Pink 2007; Flewitt et al. 2009). In spontaneous web-based video interactions, participants are typically placed in different locations, whereas the screen provides the illusory perception of a shared context for participants. It follows that screen recordings are embedded in a double intersemiotic remediation: a higher level, including the analyst’s screen which captures the interaction, and a lower level, including the sum of participants’ screens, fragmented and re-coupled in the higher level screen. Furthermore, video data analyses suggest that a model entirely based on writing fails to reproduce seminal aspects, such as kinaesthetic and proxemic patterns or gaze vectors (Sindoni 2010). A more accurate model thus requires visual resources, such as screenshots and/or drawing. Research questions include: • How can this multiplicity of levels be tackled and what level needs to be fore-grounded? • What is the role of writing, screenshot and drawing in multimodal transcriptions? Building on a three-year study, this paper contends that different contexts of video interactions require transcriptions based on an ad hoc combination of writing, screenshots and drawing (Sindoni in press). A two participant Skype conversation, for example, is very different from a multi-party video chat room (Sindoni 2011). However, both of them may be transcribed, accounting for resources used by participants (e.g. speech, chat, gaze, etc.) and for the multiplicity of screen levels. Hence transcription needs to use written and/or visual descriptions accordingly. If compared with the alleged “objectivity” of screenshots, drawing preserves informants’ identities, who may feel exposed by displaying their “real selves” despite informed consent. Conversely, drawings lack important information inscribed in screenshots and may also reflect the analyst’s bias. Factors affecting transcription may also include informants’ feelings on the matter, technical issues or the analyst’s research agenda. The study thus argues the case for a flexible model that combines available resources according to pre-conditions and research goals.