An interactive installation that encourages participants to find calmness through physical movements
#physical-computing #creative-coding #machine-learning-gesture-recognition
Time
2023.11-12
Type
Class Project
Group
Katherine Liu
Kanchan Pathak
Technology
Particle Photon
C++
Rhino
p5.js
ADXL362 accelerometer
Edge Impulse
Laser Cutting
Overview
Lotus Echoes is an interactive mindfulness installation that encourages participants to find calmness through physical movements. Using the floral controller, the player can interact with the lotus on screen and enjoy an immersive experience with soothing visuals and music.
My Contribution
Designed interaction flow
Programmed visual interaction using p5.js
Programmed Particle Photon in VS Code
Trained gesture recognition ML model
Produced movement music in GarageBand
Designed and fabricated the physical controller
Interaction Flow
The experience involves two primary interaction flows. The first captures raw accelerometer data from the controller, which influences the lotus's movements on the canvas, creating ripples in the water. The second flow involves processing the raw data through a machine-learning model to interpret specific player gestures.
Challenge
As it took a long journey for our accelerometer data to arrive in the p5.js code, especially through the ML model, latency and asynchronism have been our biggest challenges for the project. We decided to send the raw data and gesture types separately. The raw data transmits faster and is used for real-time control of the lotus’ position, while the gesture types take a bit longer to arrive and will trigger non-time-sensitive interactions.
To smooth out abrupt changes and noise in raw accelerometer data, I used interpolation to create transitions between the current flower position and the target position.
I also implemented non-blocking delays using timers in both the microcontroller code and p5.js to minimize the chance that the longer task of gesture recognition blocks or delays the shorter task of transmitting/receiving raw data.
Physical Controller
Instead of a standard mechanical remote control, we decided to have an organic and gentle design of a floral-shaped controller, which can evoke a sense of calm and relaxation, aligning with the game's objective of promoting mindfulness and tranquility. I referenced this flower model and designed a base to store our Photon at the bottom and ergonomic handles that can fit comfortably in the hands.
Music
Within the interaction, each movement direction correlates with unique musical melodies, enhancing the immersive experience.The pitch and tempo of the music vary with the lotus's movement direction, simulating the sensation of pluckingstrings. For instance, when players move the lotus up-left, the music's pitch rises and tempo slows down. I crafted fourdistinct melody categories using the Chinese traditional instrument Guzheng in GarageBand.
