Ryu Nakagawa, Masaya Furukawa, Ayano Yamanaka, Maika Yamamoto
(2023)
Concept/Creative Direction: Ryu Nakagawa
Programming/System Direction: Masaya Furukawa, Ryu Nakagawa
Haptic Device Design: Masaya Furukawa, Ayano Yamanaka
Visual Design: Maika Yamamoto
(special thanks)
Sound and visual design of Prototype (2022): Misaki Oka
(Executive Summary)
A remote, multi-participant XR audiovisual art experience combining haptic, auditory, and visual elements. Participants can interact with the hand avatars of other remote participants through musical expression in their space.
(Abstract)
In Meta Musicking, a multi-participant remote MR (Mixed Reality) experience, 1) each participant can interact with the hand avatars of other remote participants and virtual objects for musical experience within the physical space where each participant exists, and 2) by touching their own index fingers together, or touching their index finger with that of other participants, they can create note objects that generate sound, visual effects, and haptic feedback with the belt-type hot/cold haptic device when tapped, and 3) they are able to freely compose and perform with these objects in mid-air. Additionally, 4) it is possible to generate a sustained sound (drone sound) or arpeggio sound by aligning the palms of the participants’ hands, and then 5) it is possible to feel the warmth and vibrations as the presence of remote others through haptic feedback with the haptic device. In addition, participants can see cross-shadow fusion between real and virtual shadows with the XLS of the artwork. Overall, we believe that through this musical play, each participant will not only be exposed to the possibilities of experiences beyond music but will also contemplate their relationship with others through remote communication via XR.
(a) Conceptual drawing of MetaMusicking. Participants can experience it from anywhere.
(b) Installation set for exhibition and videography.