Digitalizing the visible world: Scanning the environment using a mobile with an AI app
Research output: Contribution to conference › Conference abstract for conference › Research › peer-review
Standard
Digitalizing the visible world: Scanning the environment using a mobile with an AI app. / Nielsen, Ann Merrit Rikke; Lüchow, Louise; Due, Brian Lystgaard.
2021. Abstract from 17th International Pragmatics Conference .Research output: Contribution to conference › Conference abstract for conference › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - ABST
T1 - Digitalizing the visible world: Scanning the environment using a mobile with an AI app
AU - Nielsen, Ann Merrit Rikke
AU - Lüchow, Louise
AU - Due, Brian Lystgaard
N1 - Conference code: 17
PY - 2021
Y1 - 2021
N2 - Digitalizing the visible world: Scanning the environment using a mobile with an AI appKnowledge of objects in spatial environments is obtained through multisensorial resources, such as sight, hearing, touch and smell (Mondada, 2019). For most people the visual sense takes priority in the perception of objects’ spatial relation to the sensing body and the social world (Workman, 2016). We mostly use our eyes to unnoticeably perceive objects and their position in the environment (e.g. Goodwin & Goodwin, 1996). Visually impaired people (VIP), however, have no or very limited access to the visual aspects of these spatial relations, when orienting towards their immediate surroundings. The emergence of computer vision and natural language processing (NLP) in accessible mainstream technology, such as smartphones, enables the device to “translate” the visual world into language descriptions. We explore in this paper the use of the app Microsoft SeeingAI which enables VIP to receive computerized descriptions of “visual” information about objects, people and places by scanning the environment - like using a flashlight to receive information about objects in the dark. Based on a video ethnographic collection of VIP scanning the shelfs when grocery shopping, we investigate how this ‘digitalizing’ of an everyday practice is done in situ. Through ethnomethodological multimodal conversation analysis (Streeck et al., 2011) we investigate environment scanning as a specific aspect of distributed perception (Due, 2021a), that is focusing on the co-operative actions between sense-able agents. As VIP rely on other multisensorial resources than sight alone, the question is how the practice of scanning is achieved in situ. Our analysis shows, that the practice of scanning inanimate distant objects require complex coordination of the device used to scan in relation to the object(s) being scanned with regards to both distance, angle of the device, and the location of the relevant feature of the object. This paper focuses on three particular phenomena: i) scanning nearby surroundings for the location and identification of objects, ii) scanning specific objects to obtain information about the object, and iii) scanning text on the object. This paper contributes to research into blind and visually impaired people, the senses and perception in interaction, shopping activities (Due, 2017) and interactions with non-human robotic agents (cf. Due, 2021b). Due, B. L. (2017). Respecifying the information sheet: An interactional resource for decision-making in optician shops. Journal of Applied Linguistics and Professional Practice, 14(2), 127–148. https://doi.org/10.1558/jalpp.33663Due, B. L. (2021a). Distributed Perception: Co-Operation between Sense-Able, Actionable, and Accountable Semiotic Agents. Symbolic Interaction, 44(1), 134–162. https://doi.org/10.1002/symb.538Due, B. L. (2021b). RoboDoc: Semiotic resources for achieving face-to-screenface formation with a telepresence robot. Semiotica, 238, 253–278. https://doi.org/10.1515/sem-2018-0148Goodwin, C., & Goodwin, M. H. (1996). Formulating Planes: Seeing as a Situated Activity. In Cognition and Communication at Work, (eds.) David Middleton and Yrjö Engestrom (pp. 61–95). Cambridge University Press.Mondada, L. (2019). Rethinking Bodies and Objects in Social Interaction: A Multimodal and Multisensorial Approach to Tasting. In U. T. Kissmann & J. van Loon (Eds.), Discussing New Materialism: Methodological Implications for the Study of Materialities (pp. 109–134). Springer Fachmedien. https://doi.org/10.1007/978-3-658-22300-7_6Streeck, J., Goodwin, C., & LeBaron, C. D. (2011). Embodied interaction: Language and body in the material world /. University Press.Workman, J. (2016). Phenomenology and Blindness: Merleau-Ponty, Levinas, and an Alternative Metaphysical Vision. Electronic Theses and Dissertations. https://digitalcommons.du.edu/etd/1210
AB - Digitalizing the visible world: Scanning the environment using a mobile with an AI appKnowledge of objects in spatial environments is obtained through multisensorial resources, such as sight, hearing, touch and smell (Mondada, 2019). For most people the visual sense takes priority in the perception of objects’ spatial relation to the sensing body and the social world (Workman, 2016). We mostly use our eyes to unnoticeably perceive objects and their position in the environment (e.g. Goodwin & Goodwin, 1996). Visually impaired people (VIP), however, have no or very limited access to the visual aspects of these spatial relations, when orienting towards their immediate surroundings. The emergence of computer vision and natural language processing (NLP) in accessible mainstream technology, such as smartphones, enables the device to “translate” the visual world into language descriptions. We explore in this paper the use of the app Microsoft SeeingAI which enables VIP to receive computerized descriptions of “visual” information about objects, people and places by scanning the environment - like using a flashlight to receive information about objects in the dark. Based on a video ethnographic collection of VIP scanning the shelfs when grocery shopping, we investigate how this ‘digitalizing’ of an everyday practice is done in situ. Through ethnomethodological multimodal conversation analysis (Streeck et al., 2011) we investigate environment scanning as a specific aspect of distributed perception (Due, 2021a), that is focusing on the co-operative actions between sense-able agents. As VIP rely on other multisensorial resources than sight alone, the question is how the practice of scanning is achieved in situ. Our analysis shows, that the practice of scanning inanimate distant objects require complex coordination of the device used to scan in relation to the object(s) being scanned with regards to both distance, angle of the device, and the location of the relevant feature of the object. This paper focuses on three particular phenomena: i) scanning nearby surroundings for the location and identification of objects, ii) scanning specific objects to obtain information about the object, and iii) scanning text on the object. This paper contributes to research into blind and visually impaired people, the senses and perception in interaction, shopping activities (Due, 2017) and interactions with non-human robotic agents (cf. Due, 2021b). Due, B. L. (2017). Respecifying the information sheet: An interactional resource for decision-making in optician shops. Journal of Applied Linguistics and Professional Practice, 14(2), 127–148. https://doi.org/10.1558/jalpp.33663Due, B. L. (2021a). Distributed Perception: Co-Operation between Sense-Able, Actionable, and Accountable Semiotic Agents. Symbolic Interaction, 44(1), 134–162. https://doi.org/10.1002/symb.538Due, B. L. (2021b). RoboDoc: Semiotic resources for achieving face-to-screenface formation with a telepresence robot. Semiotica, 238, 253–278. https://doi.org/10.1515/sem-2018-0148Goodwin, C., & Goodwin, M. H. (1996). Formulating Planes: Seeing as a Situated Activity. In Cognition and Communication at Work, (eds.) David Middleton and Yrjö Engestrom (pp. 61–95). Cambridge University Press.Mondada, L. (2019). Rethinking Bodies and Objects in Social Interaction: A Multimodal and Multisensorial Approach to Tasting. In U. T. Kissmann & J. van Loon (Eds.), Discussing New Materialism: Methodological Implications for the Study of Materialities (pp. 109–134). Springer Fachmedien. https://doi.org/10.1007/978-3-658-22300-7_6Streeck, J., Goodwin, C., & LeBaron, C. D. (2011). Embodied interaction: Language and body in the material world /. University Press.Workman, J. (2016). Phenomenology and Blindness: Merleau-Ponty, Levinas, and an Alternative Metaphysical Vision. Electronic Theses and Dissertations. https://digitalcommons.du.edu/etd/1210
M3 - Conference abstract for conference
Y2 - 27 June 2021 through 2 July 2021
ER -
ID: 276267014