An Evolving Multimodal Sign System for the Non-visual and Non-aural Soccer Spectator

The peer-reviewed abstract below is accepted for publication and presentation at the Third Conference of the International Association for Cognitive Semiotics (IACS 2018) to be held in Toronto from July 13 to July 15. The topic is the focus of Felipe’s Major Research Project (MRP) where I am serving as Principal Advisor (PA).

Involving students and advisees in real-world projects is an important component of my experiential strategy and engenders knowledge co-construction.

To reflect the role of the Student Advisee and Faculty Principal Advisor, I employ the standard practice from the field human-computer interaction: Advisees are the first author on peer-reviewed publications and the advisor is the last author. Please note the positive peer review feedback following the abstract below that has resulted from this style of close mentorship and knowledge co-construction.

TITLE: An evolving multimodal sign system for the non-visual and non-aural soccer spectator

This paper reports on an emerging tactile gestural sign system that employs non-visual, non-aural techniques to enable a blind and deaf individual to participate as a spectator in a soccer game. Our ongoing observations and interviews reveal how tactile signs are evolving between an interpreter and spectator (who is both legally blind and deaf) to produce a capability that “translates” a soccer game in real time. We focus on three gestures that represent (a) spatial and topological relations among the ball and key players, (b) in-game faults and (c) the whistle blow.

The centerpiece is a board with the same proportions as a standard soccer field. The spectator and translator face each other with the wooden board between them (Figure 1). The positions of the translator’s index fingers on the board show the position of the two closest players to the ball (one from each team). The spectator wraps his fingers around the translator’s index fingers (Figure 2), thus perceiving relations among the ball and players through negative space.

Additional information (such as faults and referee decisions) originally required the translator to shift from their topological interpretation to sign language, encouraging gesture evolution. A gesture that represents faults resembles real faults (like shirt pulling). What began as a tactile whistle-blow-shaped gesture with air pressure (from actual blowing) simplified to more efficient air blows that freed the use of fingers and where the strength of the blow adds meaning.

Although our observations are ongoing, we consider how pressures to communicate different properties of the game (such as player-ball position versus a fault) is driving the evolution of system components in divergent directions. Building on prior work on the affordances of external representations and signs (Coppin, 2014, 2015; Coppin, Li, & Carnevale, 2016) and informed by work on artifact (Kirsh, 2010) and language evolution (Imai & Kita, 2014; Senghas, Kita, & Özyürek (2004) we will discuss how physical constraints interact with affordances of different types of signs. Specifically, we will discuss how on the one hand, communicating topological relations among players and the ball is driving evolution toward more iconic gestures that resemble the unfolding concrete situation on the field, while on the other hand, communicating such aspects of game play as player faults or whistle blows encourage more conceptual specificity, driving evolution away from more iconic gestures to more symbolic ones that correspond to more abstract categories that many concrete situations can fall under.


Figure 1: The translator (right) faces the spectator (left), The translator faces the game to interpret the game. A wooden board with raised lines depicts the soccer field and its boundaries.


Figure 2: The spectator (red) wraps his hands around the translator’s hands (blue) using his index fingers as guidelines. The translator is able to represent the movement of the closest player to the ball (one hand per team).

Anonymous Peer Review 1 — Overall evaluation: 3 (strong accept)

This is a very relevant and extremely interesting contribution, promising to result in a very good talk. This case study is extremely interesting in that it expands the range of modalities typically considered in the discipline to the tactile modality. The authors promise to discuss “how physical constraints interact with affordances of different types of signs”, and I would invite them to consider in this context (beyond the work on language evolution they cite) also the literature on experimental semiotics for added theoretical depth.

Anonymous Peer Review 2 — Overall evaluation: 3 (strong accept)

This abstract presents a case study of the evolution of a tactile communication system for a highly specific domain (football) between a translator and a blind and deaf person. The study is of high interest as it explores the affordances and evolution of a communication system in the tactile modality, which is increasingly relevant in the world (tele-presence, gaming etc).

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>