Synesthesia

Synesthesia is an immersive and interactive art installation conducted at Microsoft Campus, Hyderabad from Sept 19- Sept 25 , 2022

This piece generates visuals that are reactive to the audience's movements.

It employs the use of Azure Kinect , which relays live-feed and body tracking points in real-time, and using this data to drive interactive visuals.

TouchDesigner has been used to interface with the Azure Kinect and also generate the visuals .

This installation provides a surreal experience to the audience , by manipulating various elements of a music track by dancing to it . ( moving of hands /body).

Audience can :

Left Hand Up/Down- Controls the Synth

Left Hand Sideways : Controls the Frequency CutOff

Right Hand Up /Down : Controls the Lead

Right Hand Sideways : Controls the rise

Synesthesia is a neurological condition in which the senses are blended or "crossed," resulting in the perception of one sense (such as hearing) being linked to another sense (such as seeing). Synesthesia can take many forms, including the perception of colors or patterns associated with sounds or the sensation of tastes or textures associated with visual stimuli.

Synesthesia is a multimedia installation that combines music, visual projections, and tactile elements to evoke the synesthetic experience. This project aims to explore and celebrate the experiences of individuals with synesthesia by creating artworks that incorporate and reflect the sensory blending that occurs in synesthesia.

Act One : Movement reactive visuals

Act Two : Movement controlled Music

The aim was to create a space where the audience can immerse themselves into a synesthetic experience . A partially enclosed pod was created with reflecting sheets on three adjacent sides with a screen with projection being displayed.

The reflective surfaces can create a visually dynamic environment that changes as performers move around and interact with the space .They are also used to create a disorienting or mesmerizing environment that visitors can explore and interact with

Synesthesia is an embodiment of the multifaceted nature of perception, the interconnectedness of the senses, and the ways in which our experiences shape our shared reality. Through its combination of music, visual projections, this immersive installation invites the viewer to experience the sensory blending that characterizes synesthesia, and to explore the boundaries of traditional artistic mediums.

Synesthesia serves as a catalyst for reflection on the role of individual perspectives in shaping our understanding of reality and our collective identity. In a world that often emphasizes homogeneity and conformity, the celebration of diversity and difference is essential in promoting inclusivity and understanding. By creating a space that acknowledges and elevates the experiences of individuals with synesthesia, this installations aims to inspire empathy & deepen our appreciation for the complexity of the human experiences.

Act Three : Audio Reactive Handpan

A handpan is a type of percussion instrument that consists of a shallow, bowl-shaped metal body with a series of tuned, concave indentations on the top surface. Handpans are typically made from steel or other metals, and they produce a rich, harmonious sound when played with the hands or other implements. Handpans are often described as having a calming, meditative quality and are used in a variety of settings, including music therapy, yoga classes, and other relaxation and wellness contexts. Handpans are played by striking the indentations on the top surface with the hands or fingers, or by using other implements .

For this piece , we wanted to create a immersive connect with the instrument and the music that emanates from it . For this , we mapped the various concave indentations locations in the X-Y coordinate space using an Azure Kinect .

To create this immersive effect on the surface of the handpan, projection mapping technique was used .

Projection mapping, also known as video mapping or spatial augmented reality, is a technique that involves projecting digital images or video onto three-dimensional surfaces or objects to create the illusion of the surface or object being transformed.

The required concave indentations were manually mapped into a mask layer onto which the visuals that are created using Touchdesigner are projected .

TESTIMONIALS :

" You can now SEE music , perceive it through an another sense. The shift from the note C to C# can be visualized as the shift from Pink to Red, and likewise. I play through the aesthetic visual , thereby creating music through my eyes.... ."