Initial Idea
It started with Jing’s Monkeying Around VR climbing project made during Building Virtual Worlds, the famous ETC course focusing on rapidly iterating interactive and immersive experiences with cross-functional teams. In Monkeying Around, guests pose as a monkey stuffed animal, trying to climb back up to meet their family. To climb up, you hold on to the grab points on the wall and pull yourself up. While developing for the project, the team realized that really feeling the climbing sensation and knowing that it means falling and getting farther from family if not held tight was an experience gap that could be achieved with haptic gloves.
This initial idea outlined 2 main topics of exploration: haptics and emotional storytelling, and we decided our final deliverable will live in the intersection of both sides. This was when more team members joined:
- Winnie joined with interest in tackling the difficulty of navigating an explorational project as a producer, and is particular drawn to the research nature of working with novel technology, and the design side of immersive gameplay design.
- Alex had numerous VR development experience, including being a 3D artist and UX designer for Xhaler, a VR project investigating breathing as a mechanic. Her main goal is to continue diving into novel inputs that could innovate the VR space.
- Jack, whose goal is to become a gameplay engineer, was intrigued by the possibilities of haptics enhancing or even evoking emotional responses, which leads to interesting mechanics.
- Yufei, our 3D artist, enjoyed creating immersive environments for VR to create emotional impact.
- Michael, who wasn’t in the official pitch team at the time due to time conflicts, also joined unofficially for brainstorming meetings because as a sound designer, he was immensely interested in drawing the parallel comparisons between haptic and audio design. After all, both work with waveforms and vibrations!
What we got across during Round 1 Pitch
Our round 1 pitch was guided by the giant hourglass example:
Imagine standing inside a massive hourglass, with sands falling from above. You reach out your hand, feeling the grains of sand slip through your fingers. But it is not just your hand feeling the sand. Your heart, mind, too, also realized how time itself is slipping away. Even if you try to stop it, you can’t, and the sand continues to fall until it’s all gone. Your hand remains in the air, the sensation of sand is gone, but the feeling of loss still lingers around you.
During pitch 1, the ideas that we got across were:
Touch is a visceral sensation that is innate and carries emotional weight
A research done at the University of Regensburg in Germany found that touch leaves a memory trace that persists long after the physical sensation is gone.
Haptics for emotional storytelling is a space that hasn’t been explored as much.
Haptics in VR has been widely used for simulation and training for industrial uses, and basic sensory feedback for the entertainment world. However, both research and application for when it interacts with emotions and other senses has not come to commercial breakthrough, and more exploration is welcomed as well as needed in this space.
We will create a single coherent VR experience that ties the explored prototypes together.
We will prototype as many ways touch achieves the above, and string them together to a single coherent experience through game design. Our guiding example is Face Jumping, which uses a single mechanic of looking at a character’s eyes to change to their perspective.
Our focus will be on the experience design not hardware development but they might be expensive.
Upon a quick email to Vivian Shen, a then-SME for the team Haptic Waves, which explores haptics and audio without visuals, she recommended us to seek out high precision gloves that provide comprehensive SDK so we can focus on actually exploring user experience research and game design. The three glove options we explored was HaptX, SenseGlove, and bHaptics, with the former two costing more than $6,000 and provides force feedback, while the last option provides only vibrotactile feedback and costed only $299.
We are different from any previous ETC projects
- Haptic Waves (25sp): Strip off visuals → we are not doing so.
- VektoR (23sp): Learn from designing for the affordances of the hardware.
- Telepathway (22sp): For a good haptics experience, distinguish between the coolness of hardware itself from storytelling and emotions.
- Emotionshop (15f): Rapid prototyping of game mechanics to emotions mapping → similar workflow but less prototypes to explore deeper.
- Lights Out (16f): Nonvisual multisensory location-based experience → we have visuals and focus on touch.
- Sonology (08sp): Tactilizing sound in Whip It music world → while we might explore, our focus is on the emotional effect it brings.
Our idea got many faculty on board, some interested in the emotion aspect, some in the innovation aspect, and some thought we really did good research, so our first round pitch passed! The main aspect to get the next round pitch to pass is answer our choice of technology!
More research and outreach for round 2
To get more concrete on our plan of design and development, we continued reaching out to as many people as possible. ETC faculty, CMU main campus researchers, glove companies, and so on. Here are a list of people who gave us immense guidance!
Jessica Hammer – Jess is an ETC and HCI Institute faculty especially focusing on transformational game design and research. She showcased us a very efficient way for us to find research papers on Google Scholar. She suggested ways to find people to reach out on main campus. Her guidance was high level but very specific in direction. Because of her, we were able to find people in psychology departments!
Roberta Klatzky – Charles J. Queenan, Jr. University Professor of Psychology and Human-Computer Interaction at CMU who has done haptics and perception research for more than 30 years! Upon a quick email, she immediately met with us, shared with us how haptics research has been done (what are the parameters to experiment with – intensity, frequency, and duration), an overview of current technologies, and most importantly, how to scope our project best for our design and entertainment application project. She reviewed multiple glove choices with us, and showed us what specs to look for in to find the right gloves (actuator types, actuator intensity range, actuator locations, etc.) At the end, she suggested that we choose the bHaptics gloves for its sufficient parameters (so that we will not need to model physics). Her paper on Feel Effects, where the research gives language of what people identified as emotions was the starting point of our subsequent research!
Shirley Saldamarco – Special faculty at ETC focusing on themed entertainment design. Shirley was the hugest advocate for our emotional design direction, and gave us valuable feedback on structuring our storytelling element for an emotional journey with haptics. She said that feeling time, as well as abstract concepts, were a good choice.
Vivian Shen – Vivian, a current PhD student in the CMU Robotics Institute doing haptics and interaction research (who became our faculty advisor in the fall), met with us on zoom. She verified that our choice of bHaptics gloves was a smart choice, and shared with us her experience researching and developing a different glove (more precise, but not commercially usable for now). She walked us through the commercial glove SDKs, and looked at the specific functions that might be useful for us!
Ruth Comley – Ruth is another ETC faculty. She shared with us her extensive experience in haunted house design, talking about how an emotional experience is all about setup. Having a closed curtain and a label saying zombie gut will make wet spaghetti more disgusting and made people more scared. She told us how important context is, and this aligned with every research we have done!
Jesse Schell – Jesse Schell is a Distinguished Professor of the Practice at ETC, and CEO of Schell Games. After hearing about our project direction, we came up with two different development process, one that is bottom-up, going from many prototypes and experiments to a final experience, and one that is top-down, where we come up with an idea and start building it and having detailed playtest in between. We concluded that bottom-up exploration process is what fits our project best!
Joe Michaels – we were very fortunate to get in contact with Joe Michaels, Senior Global VP of Sales and Marketing and one of the founding members of HaptX, the company that made the glove we initially wanted, but decided not to use due to its complicated actuators pattern and heavy battery. Joe shared his enthusiasm for our project, and confirmed that, even with their precise gloves, without temperature, they still relied on visuals and audio to simulate catching water under the sink! While we did not use their gloves, their insights gave us immense confidence that we are going in the right direction!
Context matters most + Bottom up R&D process
All of these point to one single conclusion – our haptics will work with audio and visuals because they are proven to be effective. The setup and contexts are what makes the haptic memory lasting and impactful. To explore this more thoroughly, we will choose a well supported glove company that has sufficient parameters and affordable prices. Finally, we will undergo a bottom-up, prototype to final deliverable approach to ensure our process is informed and rigorous.
Finally, practicing pitch in the sound studio (using ukuleles as our microphones and having fun), each of us showed our enthusiasm in the presentation, and with our extensive research and plan of action, our pitch was approved!!
Now it’s time for summer break, some gold spike projects, design exploration, so we can get on track as fast as possible at fall!