This week, we dived into development of the chosen prototypes, but at first, we started the week with a department-wide invitation to test our haptic patterns without VR.

Haptic Patterns playtesting

Our goals are to 1. determine which haptic patterns are clearly identifiable without context. 2. what imageries are sparked by the haptic patterns itself. To do so, we generated a list of haptic patterns by categorizing parameters found in research, and conducted a playtest with playtesters trying out the 15 patterns, describing their sensational feelings (sensitivity), trying out different gestures that might go well with the patterns, and talking about their wild guesses of what those patterns are, and giving feedback on how the patterns could better represent what we intended it to be after we revealed what we were simulating.

The list of haptics patterns are: Explosion, Poke, Breathing, Slap, Tension, Stun, Crushing, Residual Impact, Heartbeat, Heat, Pinch, Train Track, Select, Chainsaw, Stroke.

What we learned

There were extremely interesting results that came out of this playtest. We’ll list it out here:

• Certain haptic events can mean multiple things because they are ambiguous enough.
  • For example, explosion and train track, while both of them included rumbling feeling (one made from a WAV file of an explosion, and one made to simulate putting your hand on the ground as the train approaches), without context, people relied on the sounds, or looked through the list of pictures, and decided that they could be absorbing energy, hand dissolving, stun, or chainsaw!
• Certain haptic events can only explicitly mean something because they match people’s sensed memories so accurately.
  • The certain haptic patterns include, heartbeat, poke/tap, pinch.
• This disparity is more on lack of context and when filled gains immediate realization.
  • Context includes movement (gestures), visual hints, and audio feedback.
  • For example, slap and crushing were immediately recognized when the movements of slapping and crushing were executed, whereas previously they were described to be “random”.
• Haptics needs events to happen in order to make the most sense for something other than basic hand actions.
  • For example, chainsaw was almost always correctly identified, but people were mostly unsure, because they have different imaginations towards how a chainsaw is (are we holding it, is it the start of a chainsaw, etc). A hand action is not sufficient for understanding.
• People move their fingers according to the vibrations.
  • In stroke and crushing, people unconsciously curl their fingers according to where the vibrations are happening. They even start doing the crushing hand movements just because the pattern intensifies.

Slap, Pinch, and Heartbeats

To streamline development, as our discussion decided during retrospectives after quarters so as not to get lost in the multiple prototypes happening everywhere and all the time (minimum 3 prototypes happening all at once in a room), we decided to use Trello as a centralized communication space to see overviews of a prototype, with a overview card added for each prototype list.

Redesigning the prototypes

After talking with many playtesters during the haptic pattern playtest, we decided our prototypes should be less arbitrary and more focused on our goal to assess the effectiveness of the pattern, as well as give space for change of contexts.

Some 3D Model Showcase

Other updates

We also finalized the design of our half sheet, poster, and logo. We also shot a social media reel to share about our project in official ETC channels.

Instagram reel

Next steps

Our next step is to bring our prototypes back to Playtest Night, but this time, testing with different contexts and how they can be tweaked for a different emotion.

This week, we started with brainstorming for haptic patterns under defined parameters of haptics. We tested our initial proof of concept prototypes at Hunt Library Playtest Night on Tuesday. We had the internal haptic patterns showcase on Thursday. Finally on Friday, we brainstormed possible contexts for each of them and prioritized those with the most potential to create a list of second tier prototypes.

Haptic Patterns Development

Parameters to Patterns

Using the parameters we found through research (focusing on repetition, location, duration, decay pattern, purpose), we mixed and matched all of them to get the haptic patterns, or the haptic primitives that we will use as basis to develop the next step contextual prototypes.

Haptic Patterns showcase

Prioritized patterns -> prototypes

On Friday, while going through what each of us brainstormed, we started ranking importance of exploration based on:

• What types of effect they will play in the experience?
  • Main interaction haptics
  • Supporting feedback
  • Background sensation
• How difficult it is to implement it?
  • Simple no setup
  • Complicated need to test

At the end, we came up with a list of 6 prototypes, those that has potential to be used as main interactions, and has sufficient complexity to test for.

---

---

---

---

---

Playtest Night

On Tuesday, we attended our first major playtest night! Our goal is to understand our pipeline for playtesting for onboarding and offboarding using the gloves, as well as try out the exploratory prototypes.

The script we followed through is the following screenshot.

Notes

Next steps

Now that we have come up with a list of haptic patterns, our next step is to test it without VR headset. In addition, we are to continuously developing the prototypes that we came up with.

This week is Quarters Walkarounds, when faculty rotate in project rooms listening to project teams share their initial research and direction. As a pitch project, our goal is to revive their memory of what we shared about our project and plan during pitch, as well as show our progress. In addition, we continued researching the problem space by reading more academic papers on how haptics and emotions create powerful interactions so our understanding became more solid. We also refined the three summer prototypes for the first Playtest Night next week, an approximately biweekly playtesting held on main campus for a space to playtest. We eventually also showed these prototypes with faculty during Quarters. Finally, with the feedback from faculty both on our project direction and prototypes, we sat down for a retrospective and next steps planning.

Continuing from last week

Quarters Walkarounds

Our plan for quarters is:

1. Refresh memory from pitch about our goals and what we plans to achieve.
2. Give a simple research finding summary on our haptics parameters, the final experience reference projects, and technology constraints.
3. Try out the exploratory prototypes, either rain or telekinesis, since haptics is very difficult to understand without actually trying them out.

Feedback

Faculty were mostly intrigued and confident in our exploration process. The main questions and suggestions raised were the following:

• Jonathan noted that there were three ways of building from exploration to delivery. Our final experience will may come from:
  • A single successful haptic pattern that we reuse and changed its contexts throughout the storytelling experience.
  • A couple of successful haptic patterns stringed through narrative, like a haptic experience playground.
  • A single successful interaction that uses different haptic patterns but has the same interaction meaning, built as a core mechanic of the final emotional experience.
• Jesse suggested us to utilize the two column method, where we:
  • Have haptic patterns on the left, and come up with as many context for it as possible, or
  • Have contexts on the left, and come up with as many different haptic representations on the right column
• All faculty echoed that testing is the most important factor of our projects. A/B testing, qualitative interview for emotions (since emotions are perceived and very hard to quantitatively measure), using the inner circle of emotion wheel so as not to prime players, think-aloud protocols to gauge how players are choosing their interactions, etc.
• Moshe suggested that we have as many faculty come in our room to try our experience as many times as possible, because haptics is very hard to test!

Retrospectives

After quarters, even though it was already Friday evening, we decided it is still extremely crucial to talk about all our feedback, refine our exploration strategy, and reflect on our collaboration in this past three weeks. It was a exhausting session, but we came out of the meeting room confident about our next steps, and ready for the next weeks to come!

1. Team direction section
   1. We should ask for Vivian and Heather’s advice on testing strategies from technical haptic patterns to emotional experience.
   2. We have been building tier-2 prototypes, or rather, the contextual prototypes, without first testing the primitive haptic patterns that the gloves supports. It’s useful to use 2 column strategy where left is tier-2, and on the right we think of possible tier-1 haptic patterns, but it’ll also be very informative vice versa where we simply do tier-1 on the left, testing with what the technology is able to do, and think of possible tier 2 prototypes through different hand movement/gestures. We decided that on Monday, we will do this to inform us what prototypes we will focus on through halves. (We define tier 1 as bHaptics designer, pure patterns, systems that support tier 2).
   3. Model of creative development, using Jonathan’s 3 methods, namely a. one single mechanic but changes haptic feedback throughout the narrative.(holding fire that changes and flickers through the whole journey) b. powerful haptic discovered discretely stringed into an experience tied by narrative. (haptic playground with story) c. create vocabulary for the haptic interactions and string them discretely through narrative. Our consensus is that we are currently doing c, and finding ways to get b, and trying to see if we can find any a. Therefore, part of our “Exploration” is this process of exploring the right pipeline.
2. Team collaboration section
   1. There are times when team members aren’t sure what to do. This is because:
      • This is an exploration phase. Artists aren’t sure how many models should they focus their attention on.
      • Programmers are implicitly expecting artists to create concept art for prototype visions, but our artistic talents are in 3D modeling and not 2D illustratioin.
   2. Therefore, we need a central asset list on Trello where we can see an overview of the prototypes, and within it, each person adds what they need according to that vision.
   3. If times come when any of us don’t know what to do, tell the team, and we can discuss!

Next steps

With all the feedback and reflection, we’re prepared for our first major playtest of the contextual prototypes at Hunt Library. It is also the crucial time of carrying out our pivot back to the simplest form of what the gloves can and cannot do through defining and creating the haptic patterns on bHaptics Designer and testing for their effectiveness and coming up with next step prototypes to test for contexts.

Week 2 was a shorter week as it was Labor Day on Monday. However, we went full on with experimenting with the bHaptic gloves and revisited our design sprint board during summer to come up with ways to categorize haptic patterns for our next step of primitive haptic patterns to test with.

Understanding Haptics

Vivian, our faculty advisor, shared with us common parameters to do categorize haptics sensation and gestures. This helped us come up with the language to talk about designing the exact haptic patterns.

• texture (x/y translation, also known as lateral motion),
• compliance (z translation, also known as hardness/pressure),
• contact,
• haptic animations (like notifications)

Psychology textbook

We read a chapter for Touch from a psychology textbook on Perception. In that, we learned that

• Human temporal sensitivity (5ms) is better than vision (25ms) but worse than audio (0.01ms)
• Intermodal feelings with other senses either: compete for brain’s attention, or complement to build up 1 representation.

Recategorizing our design sprint by haptic parameters

Revisiting our design sprint ideas, we categorized the ideas we had with parameters such as location (whole hand vs. single point), duration (transient vs. sustained/continuous), and by purpose (reactive or atmospheric).

Playing with bHaptics Designer interface

Everyone was excited to see the bHaptics Designer as it is very intuitive to work with. We were able to add haptic patterns in four ways: single actuator point activation, path drawing by defining discrete points on the hand to travel through, an actual path drawn on the hand, and waveform (wave frequency design) for even more fine-tuned vibration patterns beyond intensity on chosen actuators.

Limitations we found were:

• 0 -> 1% is a huge jump of sensation feedback and it is hard to create very subtle vibrations.
• Of course still no force feedback and temperature.

Gold spike design

In addition to the three initial proof of concept prototypes, touch grass, telekinesis, and elastic band, we decided to make a rain catching scene to explore how well single point rapid whole hand low-intensity sensations work in VR if we set the contexts well enough as a gold spike of our subsequent prototypes.

Design considerations:

• Because rain is usually a whole body experience, we limit players under an umbrella so only their hands can go out and interact with the rain.
• We wanted to create the emotion of feeling alone, just as a rainy city street in the night. We therefore created a single lamp for atmosphere.
• Also to create atmosphere, we added music that Michael composed recreating a city street that might have a jazz bar.

reference city rain scene

Next steps

Week three is quarter walkarounds, a time for teams to realign vision to present a research and plan for faculty and community in general.

Since summer came before project starts in Fall, as a pitch project, we tried to get things figured out as far down as we were able to. Originally, we pitched to explore the gloves technology during summer, and immediately start with prototyping with audio and visual contexts. However, due to logistical constraints of ordering equipment before Fall semester, we decided to focus on design sprints and getting familiar with VR haptic development using controllers and not gloves as a start! Afterwards, once Fall started, we consolidated the team with a new member joining, and came up with a initial plan for exploration during the first few weeks while establishing language to talk about this exciting space that is full of various research and frameworks.

Summer Progress

Affinity Diagramming

First, in May, we underwent 2 design sprints using affinity diagramming. In brainstorming these interactions, even though we knew there were limitations of the gloves we will be using (no temperature, no wetness, only vibrations on fingertips and back of wrist), we tried to keep our imaginations unrestrained, so that we were not restricted by the limitations, and can find opportunities for innovation.

Each teammate brainstorming interactions on a FigJam board using their designated sticky note colors. Then, we went over each them together, discussing the idea details, adding more new thoughts, and connecting similar ideas using arrows on the board. (Plus, we had fun playing with the whiteboard drawing features!)

Then, using the same sticky notes, on a different board, we started grouping sticky notes by various categories we thought of, such as abstract feelings, bouncy feelings, and time-related changes.

We did not continue brainstorming this board in the summer as many of us started our summer internships or part-time jobs as well as taking courses. However, it laid a strong foundation of a shared understanding of what might or could be within our space of exploration.

Alex Hall’s VR Design Talk

Our teammate, Alex, has been in VR research academically and in industry! To help all of our team members get into designing for VR, we gathered for a 1-hour talk about best design practices in VR. The main points we learned were:

• Having room-scaled models so players feel immersed is crucial for the UX.
• When designing the environment, clearly distinguishing interactables and non-interactables is crucial.
• For clear understanding of player experience, sufficient tutorial is crucial for VR, especially since we’re working with novel input. We should make sure everything that is not intuitive gives sufficient hints on how to interact with them.

Software Architecture

Two weeks before summer ends, the programmers Jing, Jack, and Winnie gathered to discuss software structure for our project that could help with convenience of easy haptic pattern or audio and visual context changes. Our goal is to have a simple interface so even our non-technical teammates can run playtests easily since playtest is such a central part of our project.

Proof of Concept Prototypes

In addition, out of the list of interactions on the design sprint board, we chose three to make gold spike projects for so we get familiar with the workflow working with each other but also with making a VR scene that implements the proposed software structure.

One fun detail to mention is that, while we weren’t able to collaborate using Perforce, we discovered Unity 6 Version Control, and we even had our other teammates try uploading 3D models to test out this functionality. It is indeed very user friendly and thanks to it online collaboration during summer is made possible.

Week 1

Fall finally started, and we immediately initiated the long-awaited glove order process. As mentioned in the pitch devlog, the glove we eventually settled with was the bHaptics TactGlove DK2, as it provided complete SDK support (for us to focus on design), as well as having sufficient parameters for us to test. In addition, it is 2 zeros down the price of the other gloves, and they are also incredibly comfortable. However, we weren’t able to get the gloves at the start of the week. Furthermore, all the headset we borrowed had to be returned and cannot be checked out before Friday. Instead, we focused on solidifying our ideas once again. After all, summer was long, and we were in different time zones, as well as without the newest addition to our team, Michael, the sound designer who has always been in our discord, and finally officially joined as a team member.

Finally, we also met with our assigned faculty mentor Heather Kelley, who had been our pitch champion, and Vivian Shen, who was at time of pitch, a subject matter expert (SME), but now a faculty at ETC! We were extremely lucky to have both of them on our team.

Before Your Eyes

Haptic Technology

We still weren’t able to get the gloves, so we continued refreshing our memories for haptic design, and fleshed out the exact development timeline and design ideas.

Project timeline

Composition box

What’s next

Friday after Compositional Box, we were pleasantly surprised by the 6 Quest 3 headsets, 6 gloves (finally!), and all other supporting materials like batteries, Link cables, duct tape for VR playtesting area, etc. We kneeled on the ground happily holding our precious technology we’ll be exploring and developing on and for. Next week, our focus will be on getting more research done and migrating the initial gold spike prototypes into using the bHaptic gloves!

Initial Idea

This initial idea outlined 2 main topics of exploration: haptics and emotional storytelling, and we decided our final deliverable will live in the intersection of both sides. This was when more team members joined:

• Winnie joined with interest in tackling the difficulty of navigating an explorational project as a producer, and is particular drawn to the research nature of working with novel technology, and the design side of immersive gameplay design.
• Alex had numerous VR development experience, including being a 3D artist and UX designer for Xhaler, a VR project investigating breathing as a mechanic. Her main goal is to continue diving into novel inputs that could innovate the VR space.
• Jack, whose goal is to become a gameplay engineer, was intrigued by the possibilities of haptics enhancing or even evoking emotional responses, which leads to interesting mechanics.
• Yufei, our 3D artist, enjoyed creating immersive environments for VR to create emotional impact.
• Michael, who wasn’t in the official pitch team at the time due to time conflicts, also joined unofficially for brainstorming meetings because as a sound designer, he was immensely interested in drawing the parallel comparisons between haptic and audio design. After all, both work with waveforms and vibrations!

What we got across during Round 1 Pitch

Our round 1 pitch was guided by the giant hourglass example:

During pitch 1, the ideas that we got across were:

Touch is a visceral sensation that is innate and carries emotional weight

A research done at the University of Regensburg in Germany found that touch leaves a memory trace that persists long after the physical sensation is gone.

Haptics for emotional storytelling is a space that hasn’t been explored as much.

Haptics in VR has been widely used for simulation and training for industrial uses, and basic sensory feedback for the entertainment world. However, both research and application for when it interacts with emotions and other senses has not come to commercial breakthrough, and more exploration is welcomed as well as needed in this space.

We will create a single coherent VR experience that ties the explored prototypes together.

We will prototype as many ways touch achieves the above, and string them together to a single coherent experience through game design. Our guiding example is Face Jumping, which uses a single mechanic of looking at a character’s eyes to change to their perspective.

Our focus will be on the experience design not hardware development but they might be expensive.

Upon a quick email to Vivian Shen, a then-SME for the team Haptic Waves, which explores haptics and audio without visuals, she recommended us to seek out high precision gloves that provide comprehensive SDK so we can focus on actually exploring user experience research and game design. The three glove options we explored was HaptX, SenseGlove, and bHaptics, with the former two costing more than $6,000 and provides force feedback, while the last option provides only vibrotactile feedback and costed only $299.

We are different from any previous ETC projects

• Haptic Waves (25sp): Strip off visuals → we are not doing so.
• VektoR (23sp): Learn from designing for the affordances of the hardware.
• Telepathway (22sp): For a good haptics experience, distinguish between the coolness of hardware itself from storytelling and emotions.
• Emotionshop (15f): Rapid prototyping of game mechanics to emotions mapping → similar workflow but less prototypes to explore deeper.
• Lights Out (16f): Nonvisual multisensory location-based experience → we have visuals and focus on touch.
• Sonology (08sp): Tactilizing sound in Whip It music world → while we might explore, our focus is on the emotional effect it brings.

Our idea got many faculty on board, some interested in the emotion aspect, some in the innovation aspect, and some thought we really did good research, so our first round pitch passed! The main aspect to get the next round pitch to pass is answer our choice of technology!

More research and outreach for round 2

To get more concrete on our plan of design and development, we continued reaching out to as many people as possible. ETC faculty, CMU main campus researchers, glove companies, and so on. Here are a list of people who gave us immense guidance!

Jessica Hammer – Jess is an ETC and HCI Institute faculty especially focusing on transformational game design and research. She showcased us a very efficient way for us to find research papers on Google Scholar. She suggested ways to find people to reach out on main campus. Her guidance was high level but very specific in direction. Because of her, we were able to find people in psychology departments!

Roberta Klatzky – Charles J. Queenan, Jr. University Professor of Psychology and Human-Computer Interaction at CMU who has done haptics and perception research for more than 30 years! Upon a quick email, she immediately met with us, shared with us how haptics research has been done (what are the parameters to experiment with – intensity, frequency, and duration), an overview of current technologies, and most importantly, how to scope our project best for our design and entertainment application project. She reviewed multiple glove choices with us, and showed us what specs to look for in to find the right gloves (actuator types, actuator intensity range, actuator locations, etc.) At the end, she suggested that we choose the bHaptics gloves for its sufficient parameters (so that we will not need to model physics). Her paper on Feel Effects, where the research gives language of what people identified as emotions was the starting point of our subsequent research!

Shirley Saldamarco – Special faculty at ETC focusing on themed entertainment design. Shirley was the hugest advocate for our emotional design direction, and gave us valuable feedback on structuring our storytelling element for an emotional journey with haptics. She said that feeling time, as well as abstract concepts, were a good choice.

Vivian Shen – Vivian, a current PhD student in the CMU Robotics Institute doing haptics and interaction research (who became our faculty advisor in the fall), met with us on zoom. She verified that our choice of bHaptics gloves was a smart choice, and shared with us her experience researching and developing a different glove (more precise, but not commercially usable for now). She walked us through the commercial glove SDKs, and looked at the specific functions that might be useful for us!

Ruth Comley – Ruth is another ETC faculty. She shared with us her extensive experience in haunted house design, talking about how an emotional experience is all about setup. Having a closed curtain and a label saying zombie gut will make wet spaghetti more disgusting and made people more scared. She told us how important context is, and this aligned with every research we have done!

Jesse Schell – Jesse Schell is a Distinguished Professor of the Practice at ETC, and CEO of Schell Games. After hearing about our project direction, we came up with two different development process, one that is bottom-up, going from many prototypes and experiments to a final experience, and one that is top-down, where we come up with an idea and start building it and having detailed playtest in between. We concluded that bottom-up exploration process is what fits our project best!

Joe Michaels – we were very fortunate to get in contact with Joe Michaels, Senior Global VP of Sales and Marketing and one of the founding members of HaptX, the company that made the glove we initially wanted, but decided not to use due to its complicated actuators pattern and heavy battery. Joe shared his enthusiasm for our project, and confirmed that, even with their precise gloves, without temperature, they still relied on visuals and audio to simulate catching water under the sink! While we did not use their gloves, their insights gave us immense confidence that we are going in the right direction!

Context matters most + Bottom up R&D process

All of these point to one single conclusion – our haptics will work with audio and visuals because they are proven to be effective. The setup and contexts are what makes the haptic memory lasting and impactful. To explore this more thoroughly, we will choose a well supported glove company that has sufficient parameters and affordable prices. Finally, we will undergo a bottom-up, prototype to final deliverable approach to ensure our process is informed and rigorous.

Finally, practicing pitch in the sound studio (using ukuleles as our microphones and having fun), each of us showed our enthusiasm in the presentation, and with our extensive research and plan of action, our pitch was approved!!

Now it’s time for summer break, some gold spike projects, design exploration, so we can get on track as fast as possible at fall!