skip to main content
10.1145/3025453.3026042acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

AVUI: Designing a Toolkit for Audiovisual Interfaces

Published: 02 May 2017 Publication History

Abstract

The combined use of sound and image has a rich history, from audiovisual artworks to research exploring the potential of data visualization and sonification. However, we lack standard tools or guidelines for audiovisual (AV) interaction design, particularly for live performance. We propose the AVUI (AudioVisual User Interface), where sound and image are used together in a cohesive way in the interface; and an enabling technology, the ofxAVUI toolkit. AVUI guidelines and ofxAVUI were developed in a three-stage process, together with AV producers: 1) participatory design activities; 2) prototype development; 3) encapsulation of prototype as a plug-in, evaluation, and roll out. Best practices identified include: reconfigurable interfaces and mappings; object-oriented packaging of AV and UI; diverse sound visualization; flexible media manipulation and management. The toolkit and a mobile app developed using it have been released as open-source. Guidelines and toolkit demonstrate the potential of AVUI and offer designers a convenient framework for AV interaction design.

Supplementary Material

ZIP File (pn4846-file4.zip)
suppl.mov (pn4846-file3.mp4)
Supplemental video
suppl.mov (pn4846p.mp4)
Supplemental video

References

[1]
Paul Bach-y Rita. 1987. Brain Plasticity as a Basis of Sensory Substitution. Neurorehabilitation and Neural Repair 1, 2 (June 1987), 67--71.
[2]
Florent Berthaut, Mark T. Marshall, Sriram Subramanian, and Martin Hachet. 2013. Rouages: Revealing the Mechanisms of Digital Musical Instruments to the Audience. In Proceedings of New Interfaces for Musical Expression (NIME). Daejeon, Seoul. http://nime.org/proceedings/2013/nime2013_51.pdf
[3]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77--101.
[4]
Ana Carvalho and Cornelia Lund (Eds.). 2015. The Audiovisual Breakthrough. Fluctuating Images, Berlin. http: //www.ephemeral-expanded.net/audiovisualbreakthrough/
[5]
Karen Collins and Peter J. Taillon. 2012. Visualized sound effect icons for improved multimedia accessibility: A pilot study. Entertainment Computing 3, 1 (Jan. 2012), 11--17.
[6]
Nuno N. Correia. 2013. Interactive Audiovisual Objects. Doctoral thesis. Aalto ARTS Books, Helsinki. https://www.taik.fi/kirjakauppa/product_info.php?cPath=26&products_id=269&language=en
[7]
Nuno N. Correia and Atau Tanaka. 2014. User-Centered Design of a Tool for Interactive Computer-Generated Audiovisuals. In Proc. 2nd ICLI. Lisbon. http://www.liveinterfaces.org/proceedings2014.html
[8]
Bram Crevits. 2006. The Roots of VJing. In VJ: Audio-Visual Art and VJ Culture, Michael Faulkner (Ed.). Laurence King Publishing, London, 14--19.
[9]
Carl DiSalvo and Melissa Gregg. 2013. Issue-Oriented Hackathons. (2013). http: //carldisalvo.com/posts/issue-oriented-hackathons/
[10]
William W. Gaver. 1986. Auditory Icons: Using Sound in Computer Interfaces. Hum.-Comput. Interact. 2, 2 (June 1986), 167--177.
[11]
F. Wai-ling Ho-Ching, Jennifer Mankoff, and James A. Landay. 2003. Can you see what i hear?: the design and evaluation of a peripheral sound display for the deaf. In Proc. CHI'03 (CHI '03). ACM, NY, NY, USA, 161--168.
[12]
Lars Erik Holmquist. 2008. Bootlegging: Multidisciplinary Brainstorming with Cut-ups. In Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (PDC '08). Indiana University, Indianapolis, IN, USA, 158--161. http://dl.acm.org/citation.cfm?id=1795234.1795260
[13]
Jonathan Hook, David Green, John McCarthy, Stuart Taylor, Peter Wright, and Patrick Olivier. 2011. A VJ centered exploration of expressive interaction. In Proc. CHI'11 (CHI '11). ACM, NY, NY, USA, 1265--1274.
[14]
Jonathan Hook, John McCarthy, Peter Wright, and Patrick Olivier. 2013. Waves: exploring idiographic design for live performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 2969--2978.
[15]
Michael Xuelin Huang, Will W. W. Tang, Kenneth W. K. Lo, C. K. Lau, Grace Ngai, and Stephen Chan. 2012. MelodicBrush: A Novel System for Cross-modal Digital Art Creation Linking Calligraphy and Music. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, NY, NY, USA, 418--427.
[16]
Lilly Irani. 2015. Hackathons and the Making of Entrepreneurial Citizenship. Science, Technology & Human Values (April 2015), 1--26.
[17]
Sergi Jordà. 2003. Sonigraphical Instruments From FMOL to the reacTable*. In NIME03. Montreal. http://www.nime.org/proceedings/2003/nime2003_070.pdf
[18]
M. Komssi, D. Pichlis, M. Raatikainen, K. Kindstrom, and J. Jarvinen. 2015. What are Hackathons for IEEE Software 32, 5 (Sept. 2015), 60--67.
[19]
Michael Kubovy and Michael Schutz. 2010. Audio-Visual Objects. Review of Philosophy and Psychology 1, 1 (March 2010), 41--61.
[20]
Golan Levin. 2000. Painterly Interfaces for Audiovisual Performance. Ph.D. Dissertation. M.Sc. Dissertation. Massachusetts Institute of Technology, Cambridge, MA. http: //acg.media.mit.edu/people/golan/thesis/thesis300.pdf
[21]
Thomas James Lodato and Carl DiSalvo. 2016. Issue-oriented hackathons as material participation. New Media & Society (Feb. 2016), 1461444816629467.
[22]
John Maeda. 2006. The Laws of Simplicity. The MIT Press, Cambridge, MA.
[23]
Thor Magnusson. 2006. Affordances and Constraints in Screen-based Musical Instruments. In Proceedings of the 4th Nordic Conference on Human-computer Interaction: Changing Roles (NordiCHI '06). ACM, NY, NY, USA, 441--444.
[24]
Harry Mcgurk and John Macdonald. 1976. Hearing lips and seeing voices. Nature 264, 5588 (Dec. 1976), 746--748.
[25]
Oussama Metatla, Nuno N. Correia, Fiore Martin, Nick Bryan-Kinns, and Tony Stockman. 2016. Tap the ShapeTones: Exploring the Effects of Crossmodal Congruence in an Audio-Visual Interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 1055--1066.
[26]
William Moritz. 1997a. Digital Harmony: The Life of John Whitney, Computer Animation Pioneer. 2.5 (Aug. 1997). http://www.awn.com/mag/issue2.5/2.5pages/2. 5moritzwhitney.html
[27]
William Moritz. 1997b. The Dream of Color Music, And Machines That Made it Possible. Animation World Magazine 2.1 (April 1997). http: //www.awn.com/mag/issue2.1/articles/moritz2.1.html
[28]
Alistair Morrison, Donald McMillan, Stuart Reeves, Scott Sherwood, and Matthew Chalmers. 2012. A Hybrid Mass Participation Approach to Mobile Software Trials. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 1311--1320.
[29]
Paul Nagle. 2008. Yamaha Tenori-On Review. Sound On Sound Feb. 2008 (2008). http://www.soundonsound.com/ sos/feb08/articles/yamahatenorion.htm
[30]
I. Olowe, G. Moro, and M. Barthet. 2016. residUUm: user mapping and performance strategies for multilayered live audiovisual generation. In Proceedings of International Conference on New Interfaces for Musical Expression (NIME). Brisbane. http: //www.nime.org/proceedings/2016/nime2016_paper0053.pdf
[31]
Shin'ichi Onimaru, Taro Uraoka, Naoyuki Matsuzaki, and Michiteru Kitazaki. 2008. Cross-modal information display to improve driving performance. In Proc. ACM symposium on Virtual reality software and technology (VRST '08). ACM, NY, NY, USA, 281--282.
[32]
Chris Salter. 2010. Entangled: Technology and the Transformation of Performance. MIT Press, Massachusetts.
[33]
Guy Schofield, David Green, Thomas Smith, Peter Wright, and Patrick Olivier. 2014. Cinejack: Using Live Music to Control Narrative Visuals. In Proceedings of the 2014 Conference on Designing Interactive Systems (DIS '14). ACM, NY, NY, USA, 209--218.
[34]
Ladan Shams, Yukiyasu Kamitani, and Shinsuke Shimojo. 2000. Illusions: What you see is what you hear. Nature 408, 6814 (Dec. 2000), 788--788.
[35]
Alexander Wiethoff and Sven Gehring. 2012. Designing Interaction with Media Façades: A Case Study. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, NY, NY, USA, 308--317.

Cited By

View all
  • (2024)Musical Drawing in 2D and 3D: Dimensions and PerspectivesProceedings of the 19th International Audio Mostly Conference: Explorations in Sonic Cultures10.1145/3678299.3678317(181-188)Online publication date: 18-Sep-2024
  • (2022)AICP: Augmented Informative Cooperative PerceptionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2022.315517523:11(22505-22518)Online publication date: Nov-2022
  • (2022)Exploring visual stimuli as a support for novices’ creative engagement with digital musical interfacesJournal on Multimodal User Interfaces10.1007/s12193-022-00393-316:3(343-356)Online publication date: 1-Aug-2022
  • Show More Cited By

Index Terms

  1. AVUI: Designing a Toolkit for Audiovisual Interfaces

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. audiovisual
    2. crossmodal interaction
    3. hackathons
    4. interaction design
    5. interface builder
    6. participatory design
    7. prototyping
    8. toolkit
    9. user interface

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)50
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 15 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Musical Drawing in 2D and 3D: Dimensions and PerspectivesProceedings of the 19th International Audio Mostly Conference: Explorations in Sonic Cultures10.1145/3678299.3678317(181-188)Online publication date: 18-Sep-2024
    • (2022)AICP: Augmented Informative Cooperative PerceptionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2022.315517523:11(22505-22518)Online publication date: Nov-2022
    • (2022)Exploring visual stimuli as a support for novices’ creative engagement with digital musical interfacesJournal on Multimodal User Interfaces10.1007/s12193-022-00393-316:3(343-356)Online publication date: 1-Aug-2022
    • (2021)Preserving Hand-Drawn Qualities in Audiovisual Performance Through Sketch-Based Interaction2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC51201.2021.9576315(1-10)Online publication date: 10-Oct-2021
    • (2020)Musical Multimodal InteractionProceedings of the 2020 International Conference on Multimodal Interaction10.1145/3382507.3419444(3-3)Online publication date: 21-Oct-2020
    • (2020)10 Years of Research With and On HackathonsProceedings of the 2020 ACM Designing Interactive Systems Conference10.1145/3357236.3395543(1073-1088)Online publication date: 3-Jul-2020
    • (2020)Have a SEAT on StageProceedings of the 2020 ACM Designing Interactive Systems Conference10.1145/3357236.3395492(695-707)Online publication date: 3-Jul-2020
    • (2019)How do Dancers Want to Use Interactive Technology?Proceedings of the 9th International Conference on Digital and Interactive Arts10.1145/3359852.3359869(1-9)Online publication date: 23-Oct-2019
    • (2018)Visualisations elicit knowledge to refine citizen science technology designProceedings of the 30th Australian Conference on Computer-Human Interaction10.1145/3292147.3292171(133-144)Online publication date: 4-Dec-2018
    • (2017)The Role of Live Visuals in Audience Understanding of Electronic Music PerformancesProceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences10.1145/3123514.3123555(1-8)Online publication date: 23-Aug-2017

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media