In spring 2024 at CMU ETC, Team Flow was able to create an interactive immersive experience that takes the guests on a spiritual journey through a Chinese folklore-inspired world with dance, music, light and color. In the experience, the guests will witness the adventure and friendship of a deer and a dancer, and help them out during important moments. Team Flow made it happen in one of the signature interactive rooms at ETC, Cavern, which contains a 270-degree curved screen for projection and multiple devices for body movement tracking. We also incorporated motion capture technology for the animated contents. Over the semester, we held more than 70 demo & playtest sessions for more than 100 guests. We kept iterating and improving based on their feedback to reach what we see in the final production.

Please enjoy our demo:

Let’s take a deep dive into our production process~

Design

The story is about a deer and a dancer meeting and growing up together. One day the dancer fell off the cliff while picking the herb. The deer eventually revive the dancer by overcoming multiple obstacles.

For a better storytelling and interactive experience, we designed with an interest curve in mind. We kept refining our storyline and interaction design with continuous playtest, and the final version of the story came into shape after multiple iterations.

For every interactable level, we divided the screen into 3 sections. We aimed to match the level layouts with the three interactions we designed and to optimize usage of the entire screen in cavern.

After numerous sessions of playtest, we received feedback that our interactions were not intuitive enough. So we decided to add a live host with a pre-show for the experience, leading the guests into the cave art and introducing them to the three types of interactions we designed for them to enjoy. By doing so, we ensure that the guests will have a better knowledge of the interactions and a more concrete context of the story before diving in.

Art

To better convey Chinese folklore, we decided on an art style inspired by Chinese ink painting. We created our original concept art and color palette after researching and studying many famous Chinese landscape paintings. 

From the concept art we were able to create some universal shaders and art assets that could be used across the entire project and reflect the art style.

In total, we built 7 scenes, including 3 interactable ones. 

Scene 3 – First Interactable Level (season changing) with Guidance (Growing up Together)

VFX were widely used in the experience to reflect the narrative. 

After receiving feedback from the guests during our playtests, we developed more visual cues to guide the focus across the screen and make the scene more dynamic.

VFX to guide the focus: guiding particles, hand particles for guests, blinking outline

VFX to make the scene more dynamic: wind, leaves, flower petals

Tech

We applied motion capture data to create the animations for the dancer. With the support of CMU motion capture lab, we were able to capture Nina’s choreography and turn it into skeletal animations. We spent a lot of time on the animation data cleanup, and matching it with the movement of the deer. After a few captures and iterations, we were able to achieve more satisfactory results than the beginning by optimizing the root motion, and adding more storytelling and interactive elements to the motion captured content.

Most of the interaction in the experience was achieved by using Kinects to track the guests’ body movements. We also incorporated a vive tracker by disguising it as a lantern to light up the virtual cave art at the very beginning of the experience. The pipeline was proven workable by the first half of the semester. In the second half, we succeeded in optimizing it by reducing the loading time by 70%, reducing the noise during body tracking by 30% and implementing multiple Kinects for future cavern projects.

Flow Beyond Spring 2024

We are definitely seeking out for any potential development for Flow outside the semester.

There has been plans for landing Flow in Shenzhen, China this summer…

Coming SOON~