We took our team photo!

Production:

Long Story Short: After considering time limitations, technological constraints, the availability of collectible data, and limited human resources, we have been advised to shift our focus away from the ukulele—whether it involves hand tracking, chord detection, or pitch detection—and instead explore face detection, which is much more accessible from a technological standpoint.

Programming:

Before arriving at this decision to change technologies, we thoroughly investigated the practicality of ukulele-based detection:

Hand Detection: Highly unstable when playing the ukulele (using MediaPipe)

Ukulele Detection: High cost, high risk

Chord Shape Detection: High cost, high risk

1. Hand Detection (Based on Initial Project Concept)

Our original plan followed a process similar to Gesture Quest. However, left-hand tracking is often occluded by the fretboard, while the right hand becomes unidentifiable when strumming. Additionally, training a machine learning model from scratch requires time, and we are uncertain whether it will function effectively without this extended training period.

2. Alternative Approach: Sound Detection

A. Chord Detection:

Chord detection differs significantly from pitch detection. It requires sound-related algorithms and ideally machine learning to train a model that extracts individual notes from a chord.

B. Pitch Detection:

Pitch detection is feasible; however, it would mean completely eliminating left-hand detection (chord tracking), which is a core requirement for our clients.

Decision Making

After multiple discussions and practical considerations, we have decided to spend the following week rethinking a game that utilizes face detection as an alternative controller, aiming to craft an intuitive experience for motor skill progression.

Game Design + Art Direction

Since our original plan was to brainstorm a generalized art direction that could work for both ukulele and face detection, we developed the following concept:

Game Content: Side-scrolling, 3D art, low-poly style with a strong emphasis on environmental aesthetics and stylized rendering.

Gameplay features: Environmental puzzles and platforming mechanics.

Game Controls:

Basic movement: Click or keyboard control move the character.

Puzzle mechanics: Utilize special inputs (e.g., sound, gestures, object interactions).

Game Loop:

1. Basic character movement
2. Encounter puzzle/obstacle
3. Use special input to solve the puzzle/progress
4. Resume basic movement

↓ Here is an early idea of the game layout: While the UI will vary based on different input methods, the 3D assets will be reusable. Next week, after receiving refined client requirements and direction based on face detection, we will restart the process and frame a few concept pitches for this new approach.