Theme 3: Multimodality in the Game

Gaming calls upon a range of senses (including audio, visual and haptic interactions), all of which contribute to and, in combination, enhance the player experience. The core question here is, How can we leverage the human perception system to improve the game experience/game design?

Projects include:

  • Fidelity and levels of realism: How much audio and visual fidelity is really needed? How does that fidelity impact our experience?
  • Study of cross-modal perception and interactions: How does sound work with image in the context of a game?
  • Aesthetic considerations in games: How do the aesthetics alter the appeal of the game, as well as our experience?
  • Embodiment and haptics (input and output modalities): How do new interfaces/interaction devices using the body as input device influence the player’s experience?

Publications & Presentations

GhasemAghaei, Reza, Ali Arya, and Robert Biddle"Design Practices for Multimodal Affective Mathematical Learning." In IEEE CSSE: 20th International Symposium on Computer Science and Software Engineering, Tabriz, Iran, 2015.Multimodality in the GameConf. PaperCarleton
Voorhees, G. "Monsters, Nazis and Tangos: The Normalization of the First-Person Shooter. "In Guns, Grenades and Grunts: First Person-Shooter Games, edited by G. Voorhees, J. Call and K. Whitlock, 89-111. NYC: Continuum International Publishing. 2012.Multimodality in the Game; Cultural and Social InteractionsBook ChapterUWaterloo
GhasemAghaei, Reza, Ali Arya, and Robert Biddle"Multimodal software for affective education: UI design." In Ed-Media:  World Conference on Educational Media and Technology, Montreal, Canada, 2015. Association for the Advancement of Computing in Education.Multimodality in the GameConf. PaperCarleton
GhasemAghaei, Reza, Ali Arya, and Robert Biddle. "The made framework: Multimodal software for affective education." In Ed-Media: World Conference on Educational Media and Technology, Montreal, Canada, 2015. Association for the Advancement of Computing in Education. Multimodality in the GameConf. PaperCarleton
Hancock, Michael. “Gamer Profiles: The Split-Screener.” Medium Difficulty. 17 April 2013. Web. July 10, 2013. Multimodality in the Game; Narrative and Dialogue in GamesJournal ArticleUWaterloo
Hancock, Michael, L. Burr, A. Houston “Hacking the Humanities: Bonfire retrospective and failed ARGs.” Digital Studies/Le Champ Numerique. Special issue “ARGs in the Real World.” Ed. By Kent Aardse. forthcoming 2016.Multimodality in the GameJournal ArticleUWaterloo
Seyed Ali Etemad and Ali Arya“Mining Expert-driven Models for Affective Motion”, ACM CHI (Workshop on Gesture-based Interaction Design: Communication and Cognition), 2014.Multimodality in the GameConf. PaperCarleton
Ehrentraut, Judy. “The New Mobile Subject: Space, Agency, and Ownership in the Techno-Utopian Age.” The Individual and Utopia: A Multidisciplinary Study of Humanity and Perfection. Eds. Clint Jones and Cameron Ellis. Ashgate Publishing: Surrey, UK, 2015. 259-278. Print. Multimodality in the GameBook ChapterUWaterloo
Collins, KarenA History of Electronic Handheld and Mobile Video Game Sound. The Oxford Handbook of Mobile Music Studies, (eds. Sumanth Gopinath and Jason Stanyek), Oxford University Press. 2014Multimodality in the GameBook ChapterUWaterloo
Dockwray R. , and K. Collins. A Symphony of Sound: Surround Sound and Formula One Racing Games. Living Stereo. Continuum International Publishing Group (ed. Paul Théberge, Kyle Devine and Tom Everett) 2015Multimodality in the GameConf. PaperMultiple Institutions
Valtchanov, D., & Ellard, C. G. Cognitive and affective responses to natural scenes: Effects of low level visual properties on preference, cognitive load and eye-movements. Journal of Environmental Psychology, 43, 184-195. 2015.Multimodality in the GameJournal ArticleUWaterloo
Voorhees, G. Criticism and Control: Procedure, Process and Possibility Space. In M. Wysocki, Ed., 9-20. Ctrl-Alt-Play: Essays on Control in Video Gaming. Jefferson, NC: McFarland. (LEAD CHAPTER). 2013.Multimodality in the GameJournal ArticleUWaterloo
Collins, KarenEntry on “Video Game Music”, Encyclopaedia of Popular Music of the World, Part III: Genres. London: Continuum (Ed. John Shephard). 2015Multimodality in the GameConf. PaperUWaterloo
Collins, KarenGrand Theft Audio?: Video Games and Licensed IP. Popular Music and Multimedia, Ed. J. McQuinn. Aldershot: Ashgate 2011. Reprint of 2008 journal article. Multimodality in the GameJournal ArticleUWaterloo
Voorhees, G. with J. Call and K. Whitlock, Eds. Guns, Grenades and Grunts: First Person-Shooter Games. NYC: Continuum International Publishing. 2012.Multimodality in the GameBookMultiple Institutions
Collins, KarenImplications of Interactivity: What does it mean for sound to be interactive? Oxford Handbook of Audio-Visual Aesthetics. (Eds. John Richardson and Claudia Gorbman). Oxford: Oxford University Press 2013.Multimodality in the GameBook ChapterUWaterloo
MacLaren, V., Harrigan, K., & Dixon, M. An introduction to video instant ticket vending machines.” Journal of Gambling Issues, vol. 30, 2015, pp. 22–34., doi:10.4309/jgi.2015.30.3.Multimodality in the Game; Interactions and Gameplay MechanicsJournal ArticleMultiple Institutions
Voorhees, G. Neoliberal Masculinity: The Government of Play and Masculinity in E-Sports. In R. Brookey and T. Oates, Eds., Perspectives on Sports and Digital Games, 63-91. Bloomington IN, Indiana University Press. 2015.Multimodality in the Game and Games that Change BehaviourJournal ArticleUWaterloo
Collins, KarenOne-Bit Wonders: Video Game Sound Before the Crash. Before the Crash: Early Video Game History (ed. Mark J. P. Wolf), 119-137. Wayne State University Press. 2012Multimodality in the GameConf. PaperUWaterloo
Voorhees, G. Online FPS Games. In Encyclopedia of Digital Communication and Society, 708-717. Hoboken, NJ: Wiley-Blackwell. 2014.Multimodality in the GameJournal ArticleUWaterloo
Collins, KarenPlaying With Sound: A Theory of Interacting with Sound and Music in Video Games. Cambridge, Mass: The MIT Press. 2013.Multimodality in the GameBookUWaterloo
Voorhees, G. Shooting Games. In M. Wolf and B. Perron, Eds., Routledge Companion to Videogames. NYC: Routledge. 2013.Multimodality in the GameJournal ArticleUWaterloo
Collins, Karen and Kapralos, B.Sound design for interactive media: Introducing students to sound. Journal of Sonic Studies 6(1), 2014.Multimodality in the GameJournal ArticleUWaterloo
Voorhees, G., J. Call and K. Whitlock Things That Go Boom: From Guns to Griefing. In Guns, Grenades and Grunts: First Person-Shooter Games, edited by Gerald Voorhees, Josh Call and Katie Whitlock, 1-21. NYC: Continuum International Publishing. 2012.Multimodality in the GameBookMultiple Institutions


Design practices for multimodal affective mathematical learning

In this paper we focus on interaction design for multimodal software in affective education, and provide a case study of our MADE (Multimodal Affect for Design and Evaluation) framework. We are considering the sensory modalities, affective and cognitive strategies and trying to solve mathematical learning difficulties such as lack of attention, distraction, stress or disabilities. Using a multimodal affective learning system will increase the encouragement in learning, and will help students develop grounded understanding of proportional equivalence e.g. 1/3 = 2/6.

Read it in full here.

The MADE Framework: Multimodal Software for Affective Education

This paper proposes a framework for multimodal educational systems, considering the affective strategies. The ability to communicate emotionally and cognitively plays an important role in human-computer interaction (HCI) and education. The challenge is how theoretical models of HCI can inform affective multimodal education.

Read it in full here.

Cognitive and affective responses to natural scenes: Effects of low level visual properties on preference, cognitive load and eye-movements

Research has shown that humans have a preference for images of nature over images of built environments, and that eye-movement behaviour and attention are significantly different across these categories. To build on these findings, we investigated the influence of low-level visual properties on scene preference, cognitive load, and eye-movements. In the present study, participants viewed a mixture of unaltered and altered photographs of nature and urban scenes to determine if low-level visual properties influenced responses to scenes. Altered versions included photographs with only low or mid-to-high visual spatial frequency information, and photographs where the phase or amplitude of visual spatial frequencies had been scrambled. We replicated past findings, demonstrating preference and longer fixation-time for nature scenes versus urban cities. We then demonstrated that the visual spatial frequencies and power spectra contained in images significantly influenced preference, cognitive load, and eye-movements, and can partially explain the restoration response to natural environments.

Read it in full here.

One-Bit Wonders: Video Game Sound Before the Crash. Before the Crash: Early Video Game History

The sound of the early video game arcades is probably embedded in the consciousness of everyone who was a child during the late 1970s and early 1980s. To walk into an arcade was to experience an overwhelming onslaught of crashes, laser guns, synthesized speech, and electronic beeping music, all competing for our attention. There have been several attempts to recreate the video game arcade atmosphere (such as Andy Hofle’s Arcade Ambience Project1), and the few existing soundscape recordings have been so popular that they were released on CDs.2 A website selling the CDs describes, “We will never hear such beautiful chaos quite the same way again.” On a Wired magazine blog entry about the site, a fan posted, “This is the greatest audio archive ever. I nominate that guy [who put together the collection] for the Nobel Prize.”3 Judging by the popularity of these arcade soundscape recordings, sound played a critical role in the enjoyment and success of early video games, and plays a powerful role in the nostalgia for the time. It is difficult to completely separate the sound of the early electronic video game arcades from the arcade industry that existed prior to the development of digital (solid-state) games. It was the electromechanical and slot machine gambling industry that gave birth to video games as commercial products. Most of the companies entering the video game market in the late 1970s already had a strong footing in the pinball and novelty arcade machine industry, as well as in slot machines. Bally Manufacturing Company, for instance, began in the 1930s with Bagatelle, a French parlor game similar to bumper-billiards that they adapted and developed into the first pinball machine. Midway Games, who worked closely with Taito to bring popular video games to the American market, was a subsidiary of Bally. Although Bally no longer creates video games (after selling the video game portion off to its rival, Williams), the company continues to create slot machines and electronic lottery machines. Williams Manufacturing had been around since the creation of 1934’s Contact game, perhaps the first pinball game to have sound, and entered the video game industry with a clone of PONG (1972) in 1973. Sega and Gottlieb likewise began as pinball and mechanical arcade companies and eventually entered the video game market. Pinball and other electromechanical arcade games included sound in the machines very early on. At first, this was bells and buzzers; later, mechanical ball-bearing chimes created simple musical tones, such as those in 1976’s Bally Freedom.4 It was not until the late 1970s that pinball machines used electronic sound components. Some electromechanical arcade games (pinball, gambling race games, shooting games, etc.) had four-track and later eight-track tape player units incorporated into them to play music and sound effects. One archivist describes the tapes: Some of the tapes are simply used for background music and/or sound effects. Others had audio tracks dedicated to particular game functions. For example, the Wild Kingdom tape just has the same jungle sound effects on every track. But the Haunted House gun game has four discrete audio tracks: one for the background and the other three for specific target sounds. Also, some Chicago Coin gun games offered an optional 8-track player but did not include a tape. They expected the operator to supply a pre-recorded 8-track with whatever music they wished. The flyer for Funland put it this way: “Musical background can be changed to fit the mood of its locale.”5 Hybrid electromechanical/solid-state games also existed, combining eight-track tapes with analog or digital sound. For instance, in Atari’s Triple Hunt (1977), the 8-track tape is used for background sound effects only. The game’s microprocessor generates all other game sounds. Hit the Bear and Raccoon Hunt use the same background sounds: 30 seconds of nighttime forest sounds-crickets, frogs and lots of background noise. Witch Hunt, on the other hand, was a much more elaborate production. [It is] 3 minutes and 26 seconds of spooky sound effects: wind, blood-curdling screams, creaking doors, pounding heartbeats, wolves, and even some speech playing backwards. It’s no wonder most Triple Hunt games on location were set up for Witch Hunt.6 The early game companies-Bally, Williams, Gottlieb, Sega, Midway, and others-had well understood the importance of sound in attracting players to the machines and keeping them interested when they entered the video game market. The very first (un-marketed) video games had no sound, but the games that were commercially developed and marketed for the arcades included sound and marketed it as a key selling point. This essay describes sound in video games up until 1983 and is divided into three parts: “How Sound Was Made” explains the technology that was being used to create sound effects and music in games in the early days; “How Sound Was Used” describes the ways in which sound was commonly employed in games; and “Key Influential Games for Sound up to 1983: A Series of Firsts” outlines some of the key sound innovations during the time period before the Great Video Game Industry Crash of 1983.

Read it in full here.

Sound design for interactive media: Introducing students to sound.

In today’s multimedia world, an understanding of non-visual modalities, particularly sound, is of great benefit to design. With the use of digital technologies within design and art practices, the affordances for sound are frequently present, but often under-used and misunderstood by those with an education that privileges the visual mode at the expense of other modalities. Despite the importance of sound within multimedia applications and the fact that digital art, video games, film, branding, and product design today all require some understanding of the sonic realm, art and design students often complete their degrees without following a single class related to sound and its perception. In this paper, we introduce several exercises undertaken in interdisciplinary sound design courses taught to undergraduate designers, artists, and game developers with the aim of illustrating a scaffolding approach to teaching about sound.

Read it in full here.