We took our team photo!

Production:

Long Story Short: After considering time limitations, technological constraints, the availability of collectible data, and limited human resources, we have been advised to shift our focus away from the ukulele—whether it involves hand tracking, chord detection, or pitch detection—and instead explore face detection, which is much more accessible from a technological standpoint.

Programming:

Before arriving at this decision to change technologies, we thoroughly investigated the practicality of ukulele-based detection:

Hand Detection: Highly unstable when playing the ukulele (using MediaPipe)

Ukulele Detection: High cost, high risk

Chord Shape Detection: High cost, high risk

1. Hand Detection (Based on Initial Project Concept)

Our original plan followed a process similar to Gesture Quest. However, left-hand tracking is often occluded by the fretboard, while the right hand becomes unidentifiable when strumming. Additionally, training a machine learning model from scratch requires time, and we are uncertain whether it will function effectively without this extended training period.

2. Alternative Approach: Sound Detection

A. Chord Detection:

Chord detection differs significantly from pitch detection. It requires sound-related algorithms and ideally machine learning to train a model that extracts individual notes from a chord.

B. Pitch Detection:

Pitch detection is feasible; however, it would mean completely eliminating left-hand detection (chord tracking), which is a core requirement for our clients.

Decision Making

After multiple discussions and practical considerations, we have decided to spend the following week rethinking a game that utilizes face detection as an alternative controller, aiming to craft an intuitive experience for motor skill progression.

Game Design + Art Direction

Since our original plan was to brainstorm a generalized art direction that could work for both ukulele and face detection, we developed the following concept:

Game Content: Side-scrolling, 3D art, low-poly style with a strong emphasis on environmental aesthetics and stylized rendering.

Gameplay features: Environmental puzzles and platforming mechanics.

Game Controls:

Basic movement: Click or keyboard control move the character.

Puzzle mechanics: Utilize special inputs (e.g., sound, gestures, object interactions).

Game Loop:

1. Basic character movement
2. Encounter puzzle/obstacle
3. Use special input to solve the puzzle/progress
4. Resume basic movement

↓ Here is an early idea of the game layout: While the UI will vary based on different input methods, the 3D assets will be reusable. Next week, after receiving refined client requirements and direction based on face detection, we will restart the process and frame a few concept pitches for this new approach.

Team Bonding Time!

We had great BBQ and delicious chinese food.

Week 3 Quarters

Production:
1/4

For the quarters, we prepared our game design ideas, but mostly, we are here to address our challenges. The biggest challenge for this project is that we only have limited time and programmers to do a lot of technical tasks. Implementing native hand detection in Unity is more complicated than we think, especially in linking it with a hand model. After some research, we found that there aren’t any pretrained object detection models specifically for ukulele out there, so we need to train it ourselves and also embed it into Unity. WebGL adds another complexity upon model inference for managing memory and dealing with performance, also we need to figure out the data collection with external configuration pipeline to be easily manageable for our clients. There are also other challenges including finding a subject matter expert for the team and we don’t have a sound designer. The team will need to address these challenges in order to build a product that we’re satisfied with.

Feedbacks we got:

• Augmenting visuals with audio? Ensuring the right sound infers that the fingers are in the correct position?
• Motion Capture? Not easily accessible.
• Find alternatives—three plans, different walkarounds
• Proving what is working—pros and cons.
• Is it worth describing the scenario at the bare minimum? How many hand shapes need to be trained?
• Get the Unity asset for hand tracking.

Game Design:

In a parallel production, we frame out the overall game flow: picking different song, instructing corresponding chords(left hand), strumming pattern(right hand) and finally combine into a song.

Our theming initial design: Octopus cooking sushi

UI/UX

Wireframing the left hand and right hand tutorial layout, ideally, the main concept is to capture real Ukulele, assign each finger tip a color within a number, and label on the ukulele to instruct where to point at. At the same time, the fret will be shown all the time as official reference.

2D Art

Updating team poster, half sheet and logo:
We haven’t finalized our game theme yet, so the poster design isn’t tied to any specific style. I’ve approached it with an Art Deco aesthetic, aiming for a vintage-inspired poster design. Since our project is music-related—specifically focusing on chords and the ukulele—I’ve also incorporated some musical notes into the poster.

For the logo, I’m considering using the Segno sign as the base (it’s also incorporated into the last “S” of our team name, Stringers, on the poster) to highlight the connection to music learning.

Week 2

Game Design:

We prepared game pitches and presented them to the clients. We plan to organize the game in 2 stages: left hand learning, right hand learning, and a combination of both.

Attach Game Design Pitch Slides

Production:

Client Insights

Feedback from Game Pitches

1. Learn left-hand chords in a randomized order (using chords from the song but shuffled).

2. Learn right-hand strumming patterns (specific to the song).

3. Play the full song using both hands together.

Clarify the need:

1. Machine learning works well on identifying ukulele fretboard (Ideally)

2. Web App that can collect & send data for future research.

Plan for Quarters:

Overview: Problem <=> Solution (Why <=> What+How)

Giving a Hand-track demo

Present the Game design pitches, and ideally our narrowed down choice

Potentially, also present our challenges

Advices from Instructor meeting:

Gameplay Goal: Create learning scenarios involving hand-object interaction (ukulele). A simple game using the ukulele, even for those with no experience, to track their learning progression.

• What is the goal? A playground for collecting data where people are motivated to participate.
• What is good data and bad data? What is important to researchers?
• Can performance be validated with audio? Identify a beautiful chord or a bad chord through audio differences.
• Gameplay Ideas: Is it satisfying to play a song? Or is it randomized Whack-a-Mole? Or a combination of Whack-a-Mole and full song progression?
• Learning Aids: Markers or colors on the ukulele may help with learning chords.
• Documentation: Prepare a document outlining what they want.
• Game Concerns: Is an engaging environment distracting? Why are they concerned about this?

Production Advice:

• Use a spreadsheet to track features, what works, and what doesn’t.
• Prioritize tasks and check them off.

3D:

Production

In the first week, we met with our instructor, Mo as well as our team members, and decided on the team name.”Stringers”

Team meeting notes:

1. Understand why the client wants the design, not just what they want. This helps create purpose-driven solutions.
2. Prepare for the Client Meeting, Clarify goals, expectations, and priorities to align on the design vision, have a question list.
3. Deeping knowing the design content, if necessary find a Ukulele expert to interview.
4. Explore how musical instruments can be integrated into games—either as tools for teaching or as game elements.
5. Kick-off Task
   1. Research music games for inspiration.
   2. Study teaching methods for ukuleles, pedagogies
   3. Storyboards to map out possible user experience.

Insights from Client Meeting:

1. Research Question: How do humans acquire, adapt, and retain motor skills?
2. Outcome:  Develop a web-based Ukulele Training Tool that:

• Creates learning scenarios involving hand-object interaction (ukulele).
• Scales data collection using webcams (crowdsourcing).
• Functions as an open-source, research-friendly platform that is simple yet playable.
  • “Playable as a simple open-source computer web-app, amenable for research”
  • How can I experience it easily? No need for complex setups

3. Pivot:

• Simplicity Over Flashiness: Clear and easy-to-follow design is more important than visual complexity.
• Interactive but Not Overwhelming: Avoid overly abstract interactions to maintain research clarity.

4. Approaches:

1. Direct Mapping
   
   • Track user’s hands and the ukulele in real time.
   • Give instructions by directly showing where to press, no more abstraction or deciphering.
2. Why Ukulele?
   
   • Compact and fits within the frame of a webcam.
   • Simple enough to teach foundational motor skills effectively.
3. Existing Reference: Roli – Direct mapping for piano.

5. Data Type for Research

• Hand Tracking: x, y positions of both hands for accurate feedback.
• Performance Metrics: Reaction time, execution accuracy, and proficiency

6. Concerns and Open Questions

1. Is it enough to recognize hand shapes, Can the tool track exact pressing accuracy?
2. Hand Tracking Visibility can be flexible: Should users see their tracked hands during training?
3. Enabling Learning Customization: Should users choose what to learn, or should groups follow a structured learning path?
4. How can the app ensure ease of use and minimize setup complexity for users?

Based on the discussion: we will be coming up with several pitches based on following learning process:

1. Chord Learning: Teach hand positions for individual chords.
2. Strumming Pattern: Accurate strumming or plucking.

Song Integration: Piece chords and strumming into full songs.