Drawing Sound in MR Space

Drawing Sound in MR Space

Ryu Nakagawa, Ryo Komatsubara, Taku Ota, Takahisa Mitsumori
(2017 – 2019)

Concept/Creative Direction: Ryu Nakagawa
Programming/System Direction: Ryo Komatsubara
Visual Design: Taku Ota
Sound Design: Takahisa Mitsumori


What if we could draw sound in mid-air?
We present a multi- participant interactive audiovisual art experience using mixed reality (MR). In the MR space, participants can draw virtual lines in mid-air to create audiovisual effects. Through the experience, we provided a shared space for speculative thinking about all kinds of relationships such as cooperation between participants or creating new forms of expression with advanced technology, and so on.
Y. Mori proposed that our hands can indicate our emotions and intentions quite skillfully [1]. We believe that a fusion expression of drawing by hand and playing sound creates interesting and original emotional expressions. Many artists and composers have approached the concept of “drawing sound” using various techniques. For example, “UPIC” by Iannis Xenakis enable the user to draw sound using a tablet, with the X-axis representing time, and the Y-axis representing pitch [2]. Recently, we have been able to expand the drawing sound experience to three-dimensional space and a multi-participant experience with MR technology.

[1] Yoshinobu Mori. 2005. Symbolization of the human body : Sacred and symbolized hands in premodern Western Europe, Otsuma journal of social information studies 14,1- 29.
[2] G.Marino, M.H.Serra and J.M.Raczinski. 1993. The UPIC System: Origins and Innovations,Perspectives of New Music Vol. 31, No. 1 (Winter, 1993), 258-269.






SIGGRAPH 2019, Los Angeles, California USA, July – August, 2019 [ > ]

[Special members for SIGGRAPH 2019 exhibition]
Programing: Ken Sonobe (NakagawaLab)
System Direction: Katsutoshi Hata + Nangok R/Studios (Nangok R/Studios)
Sound Design: Hidefumi Ohmura (Tokyo University of Science)


Laval Virtual ReVolution 2018, Laval, France [ > ]


“N/O/W/H/E/R/E”, Art Media Center Open Laboratory 2017, Tokyo, Japan [ > ]



Published Paper

  • Drawing Sound in MR Space, Ryu Nakagawa, Ryo Komatsubara, Taku Ota, Ken Sonobe, Takahisa Mitsumori, Katsutoshi Hata, Hidefumi Ohmura, Proceedings of SIGGRAPH’19 Virtual, Augmented, and Mixed Reality, Article No. 18, 07 , Refereed  [ > ]
  • Drawing Sound in MR Space: A Multi-User Audiovisual Experience in Mixed Reality Space, Ryu Nakagawa, Ryo Komatsubara, Taku Ota, Takahisa Mitsumori, VRIC ’18 Proceedings of the Virtual Reality International Conference – Laval Virtual, Article No. 20, 04 , Refereed [ > ]

Supported by

The work was partially supported by JSPS KAKENHI Grant Number 18K00210.