Theme 3: Multimodality in the Game

Gaming calls upon a range of senses (including audio, visual and haptic interactions), all of which contribute to and, in combination, enhance the player experience. The core question here is, How can we leverage the human perception system to improve the game experience/game design?

Projects include:

  • Fidelity and levels of realism: How much audio and visual fidelity is really needed? How does that fidelity impact our experience?
  • Study of cross-modal perception and interactions: How does sound work with image in the context of a game?
  • Aesthetic considerations in games: How do the aesthetics alter the appeal of the game, as well as our experience?
  • Embodiment and haptics (input and output modalities): How do new interfaces/interaction devices using the body as input device influence the player’s experience?

Publications & Presentations

NameTitleThemeMediumInstitution
GhasemAghaei, Reza, Ali Arya, and Robert Biddle"Design Practices for Multimodal Affective Mathematical Learning." In IEEE CSSE: 20th International Symposium on Computer Science and Software Engineering, Tabriz, Iran, 2015.Multimodality in the GameConf. PaperCarleton
GhasemAghaei, Reza, Ali Arya, and Robert Biddle"Multimodal software for affective education: UI design." In Ed-Media:  World Conference on Educational Media and Technology, Montreal, Canada, 2015. Association for the Advancement of Computing in Education.Multimodality in the GameConf. PaperCarleton
GhasemAghaei, Reza, Ali Arya, and Robert Biddle"Multimodal software for affective education: UI evaluation." In Ed-Media: World Conference on Educational Media and Technology, Montreal, Canada, 2015. Association for the Advancement of Computing in Education.Multimodality in the GameConf. PaperCarleton
GhasemAghaei, Reza, Ali Arya, and Robert Biddle. "The made framework: Multimodal software for affective education." In Ed-Media: World Conference on Educational Media and Technology, Montreal, Canada, 2015. Association for the Advancement of Computing in Education. Multimodality in the GameConf. PaperCarleton
Seyed Ali Etemad and Ali Arya“Mining Expert-driven Models for Affective Motion”, ACM CHI (Workshop on Gesture-based Interaction Design: Communication and Cognition), 2014.Multimodality in the GameConf. PaperCarleton
Dockwray R. , and K. Collins. A Symphony of Sound: Surround Sound and Formula One Racing Games. Living Stereo. Continuum International Publishing Group (ed. Paul Théberge, Kyle Devine and Tom Everett) 2015Multimodality in the GameConf. PaperMultiple Institutions
Voorhees, G. with J. Call and K. Whitlock, Eds. Guns, Grenades and Grunts: First Person-Shooter Games. NYC: Continuum International Publishing. 2012.Multimodality in the GameBookMultiple Institutions
Voorhees, G., J. Call and K. Whitlock Things That Go Boom: From Guns to Griefing. In Guns, Grenades and Grunts: First Person-Shooter Games, edited by Gerald Voorhees, Josh Call and Katie Whitlock, 1-21. NYC: Continuum International Publishing. 2012.Multimodality in the GameBookMultiple Institutions
Collins, KarenA History of Electronic Handheld and Mobile Video Game Sound. The Oxford Handbook of Mobile Music Studies, (eds. Sumanth Gopinath and Jason Stanyek), Oxford University Press. 2014Multimodality in the GameBook ChapterUWaterloo
Collins, KarenEntry on “Video Game Music”, Encyclopaedia of Popular Music of the World, Part III: Genres. London: Continuum (Ed. John Shephard). 2015Multimodality in the GameConf. PaperUWaterloo
Collins, KarenGrand Theft Audio?: Video Games and Licensed IP. Popular Music and Multimedia, Ed. J. McQuinn. Aldershot: Ashgate 2011. Reprint of 2008 journal article. Multimodality in the GameJournal ArticleUWaterloo
Collins, KarenImplications of Interactivity: What does it mean for sound to be interactive? Oxford Handbook of Audio-Visual Aesthetics. (Eds. John Richardson and Claudia Gorbman). Oxford: Oxford University Press 2013.Multimodality in the GameBook ChapterUWaterloo
Collins, KarenOne-Bit Wonders: Video Game Sound Before the Crash. Before the Crash: Early Video Game History (ed. Mark J. P. Wolf), 119-137. Wayne State University Press. 2012Multimodality in the GameConf. PaperUWaterloo
Collins, KarenPlaying With Sound: A Theory of Interacting with Sound and Music in Video Games. Cambridge, Mass: The MIT Press. 2013.Multimodality in the GameBookUWaterloo
Collins, Karen and Kapralos, B.Sound design for interactive media: Introducing students to sound. Journal of Sonic Studies 6(1), 2014.Multimodality in the GameJournal ArticleUWaterloo
Ehrentraut, Judy. “The New Mobile Subject: Space, Agency, and Ownership in the Techno-Utopian Age.” The Individual and Utopia: A Multidisciplinary Study of Humanity and Perfection. Eds. Clint Jones and Cameron Ellis. Ashgate Publishing: Surrey, UK, 2015. 259-278. Print. Multimodality in the GameBook ChapterUWaterloo
Hancock, Michael, L. Burr, A. Houston “Hacking the Humanities: Bonfire retrospective and failed ARGs.” Digital Studies/Le Champ Numerique. Special issue “ARGs in the Real World.” Ed. By Kent Aardse. forthcoming 2016.Multimodality in the GameJournal ArticleUWaterloo
Valtchanov, D., & Ellard, C. G. Cognitive and affective responses to natural scenes: Effects of low level visual properties on preference, cognitive load and eye-movements. Journal of Environmental Psychology, 43, 184-195. 2015.Multimodality in the GameJournal ArticleUWaterloo
Voorhees, G. Criticism and Control: Procedure, Process and Possibility Space. In M. Wysocki, Ed., 9-20. Ctrl-Alt-Play: Essays on Control in Video Gaming. Jefferson, NC: McFarland. (LEAD CHAPTER). 2013.Multimodality in the GameJournal ArticleUWaterloo
Voorhees, G. Online FPS Games. In Encyclopedia of Digital Communication and Society, 708-717. Hoboken, NJ: Wiley-Blackwell. 2014.Multimodality in the GameJournal ArticleUWaterloo
Voorhees, G. Shooting Games. In M. Wolf and B. Perron, Eds., Routledge Companion to Videogames. NYC: Routledge. 2013.Multimodality in the GameJournal ArticleUWaterloo
Voorhees, G. Neoliberal Masculinity: The Government of Play and Masculinity in E-Sports. In R. Brookey and T. Oates, Eds., Perspectives on Sports and Digital Games, 63-91. Bloomington IN, Indiana University Press. 2015.Multimodality in the Game and Games that Change BehaviourJournal ArticleUWaterloo