Week 8 – Mar 15, 2024

Alphabet Gestures Correction

We reached out to Erik Pinter, who is part of the deaf community and the creator of the sign language course on the main campus. In addition to posing some questions, we sent him screenshots of our alphabet gestures. He provided the following corrections:

• D should be sideways with pointer finger close to your body
• E thumb should be more under the fingers
• G thumb should be parallel with the pointer finger and visible
• H is more perfectly horizontal
• K middle finger should be more pointed straight forward at 30-45 degrees from vertical
• P looks good but know that it can be pointed forward as well, same with Q, X
• S thumb should be more fully over fingers

Most issues stem from the orientation and degrees of finger positions. His guidance assists in refining the specifics of alphabet gestures, so that we’re able to enhance our model’s accuracy by feeding it data that aligns with the ground truth.

Training Data Module

After deciding to retrain part of the model and incorporate data from different hands, we created a separate training data module. This Python module allows every team member to input their data, offering a variety of hand types. It comes with straightforward instructions; we simply need to press SPACE to capture the landmarks data, which is then stored in a CSV file.

Challenges on Research Data Collection

While focusing on research data collection, we considered adding a survey page before the game begins. This would serve dual purposes: gathering the demographic information requested by our client and allowing players to select their preferred UI based on hand dominance, as well as submit a nickname for the leaderboard.

However, a significant challenge is integrating this research component without negatively affecting the gameplay experience. After consulting with our advisors, we agreed that an initial survey could deter players due to its impact on game flow. Incorporating data collection directly into the gameplay might also be problematic. We’re considering alternatives like a toggle between normal and research modes or keeping the research and game aspects separate, linking them through a code. Players participating for research purposes would receive a code after completing the survey, which they could enter in the game to link their data anonymously. We’ll have to further discuss with our client and better understand how the research data will be used before finalizing our approach.

Half Feedback from JT

Landmarks, reaction time, accuracy

Regarding the research data collected in the back end, we’ll measure the reaction time and accuracy for each letter. JT also expressed interest in having the landmarks be extractable and analyzable.

For measuring “accuracy,” we’ve proposed two methods: calculating based on the confidence score, and setting a threshold to determine the pass rate over several attempts. JT believes the first approach is preferable, provided we lock the model weights before beginning data collection.

We intend to save the data in CSV format, covering reaction time and accuracy for each mini-game. The data will be structured as follows: UserID – AttemptIdx – Level – Difficulty – SignIdx – Letter – Status – Accuracy/Reaction Time – Handedness – Hold Time Buffer.

Hint Options

For the hint images within each game, we have various settings corresponding to each difficulty level. The table below outlines how the hints are configured. All the numbers provided are temporary and may be subject to future adjustments.

Mini Game 3 Demo

We’ve integrated 3D art assets into game 3, and below is our current demo. Currently, we’re utilizing AI-generated 2D assets for various objects as placeholders for testing. We will replace them with our custom-painted objects.

Mini Game 2 Update

Demo

We’re testing the environment in mini game 2, which is designed to mimic a miniature and function as a running track. Adjustments to the proportion of the buildings and alterations to the style may be made to shift from a realistic depiction to one that more closely resembles a miniature.

Art

UI

Mini Game 1 Update

Layout

We revised the UI layout of mini game 1 to enhance its intuitiveness and tested it during Friday’s playtesting session. More details on this will be covered in the later section.

Feedback from Carl

We invited Professor Carl Rosendahl to play our mini game 1 and here’re his feedbacks:

• Provide a SKIP option for the mini game 3, show skipped words in the end
• Need more connection between gesture and letter
  • Further delay the hint image appearance
  • Put letter on hint images
• Layout could be rearranged
• Don’t make the cat too obvious
• Palm orientation could be confusing

Playtesting on Friday

This Friday, we’re focusing on testing three key aspects:

1. Mini Game 1: Assessing the intuitiveness of the updated layout.
2. Mini Game 2: Evaluating gameplay, track speed, and visibility.
3. Mini Game 3: Testing gameplay elements, including the timer and spelling mechanism.

Our playtesters, who are all middle school students, expressed great enthusiasm for the game and its technology. Overall, the core aspects of our three mini-games were deemed logical and received validation. We received some specific feedback as follows:

• The letters in Mini Game 2 could be clearer, as they currently blend into the environment somewhat.
• Initially, there was some confusion regarding palm orientation for some letters, requiring a few attempts to correct.
• The narration proved to be engaging, sparking further interest in exploration.
• The sign language tracker was found to be accurate with a quick response time.
• The updated layout in Mini Game 1 was well-received and made sense to the players.