Our Goals

Train and deploy a machine learning model to 

• Detect left hand gesture and identify chord shapes

• Detect right hand up/down patterns

• Collect data including hand / finger position, reaction time, and accuracy

Design and implement a game that

• Includes sample lectures of songs

• Is easy to add new songs to the game with different chords & strumming patterns

• Is designed from a perspective of “Simplicity”, to not become a distraction of the research itself

Deploy a WebApp that

• Can be accessed easily by anyone with a webcam and a Ukulele

• Can send data to a server hosted by our client

Game Design (In progress): A song-driven game that has 3 phases

• Left hand / chords
  Teaching all chords used by the song first, then randomly presenting the chords to the player

• Right hand / strumming pattern
  Present a fixed (or a limited number of) strumming pattern and ask the player to practice it through a traditional rhythm game format

• Both hands / entire song
  Combine the chords and the strumming pattern together