STRINGERS

– An ETC Project website

Hi,

Design and build an extensible research tool embedded with game elements to collect data of motor skills through the process of learning Ukulele.

Our Goals

Train and deploy a machine learning model to
– Detect left hand gesture and identify chord shapes
– Detect right hand up/down patterns
– Collect data including hand / finger position, reaction time, and accuracy
Design and implement a game that
– Includes sample lectures of songs
– Is easy to add new songs to the game with different chords & strumming patterns
– Is designed from a perspective of “Simplicity”, to not become a distraction of the research itself
Deploy a WebApp that
– Can be accessed easily by anyone with a webcam and a Ukulele

Game Design (In progress): A song-driven game that has 3 phases
– Left hand / chords
– Teaching all chords used by the song first, then randomly presenting the chords to the player
– Right hand / strumming pattern
– Present a fixed (or a limited number of) strumming pattern and ask the player to practice it through a traditional rhythm game format
– Both hands / entire song
– Combine the chords and the strumming pattern together

  • Week 14 Final & Presentation

    Lab Set Up Ww went to the client’s lab (Physical Intelligence lab). Currently our game runs with another server program to split up the work between the the face recognition with data collection task and the game itself, The reason we chose this architecture is that our clients will host their own server to collect…

  • Week 13

    Tech Development We finalized the tutorial to wrap Up the game, fixing in-game small bugs, playtesting with the data collection the gives a sample for our client to review, and also as a reference of data collection sample.  UI Final refinement Problem: Difficulty Should Come from Design, Not Confusion With a designed level, another important…

  • Week 12

    Tech Development Data output from game: UI Rewiring: New iteration on UI interface, updated tutorial pages and level selection Sound BGM: We composed four different music tracks for our game, including a menu theme and multiple level themes. Sound Effects: Created custom sound effects for gameplay, covering level mechanics and character behaviors. Unity Implementation:All sound…

  • Week 11

    Playtest Goal After the official playtest day on Saturday, March 29th, we tested our game with many guests of various ages. Most of them were unfamiliar with our game and were all naïve guests who had not played it before. Our plan was to test Level 1 and Level 2, starting with the default speed.…

  • Week 10

  • Week 9

    Week 9 During GDC week, the rest of the team continued working on the project, including facial expression mapping, IK setup for character animations, Level 2 design and implementation based on the last week block out, the landing page, and level selection UI. 3D: We’re currently working on three animations for our character, including ghost…

  • Week 8

    Week 8 Gameplay: Brought in assets. Focused on scripts that control visual elements. Adjusted animations of the character model and timing of visual effects. Next week will be focusing on the mapping of face detection and animation, data collection, and fine tuning of animations UI/UX: Designed the complete game wireframe, including the landing page, in-game…

  • Week 7

    Working on Half, proceeding the space travel game Halves Frame out gameplay And bring the design concept to the game theming, we retain the original idea of a character traveling through space. Keeping in mind the constraint of constraining the path, the player Will be surrounded by a dangerous meteorite belt, navigating along a single…

  • Week 6

    Production: In the beginning of this week, we first came up with 10 pitches for a face detection game that using face as a controller. We come up with interesting idea, and also analyse the possible challenges that it will bring up: either tech challenges or does not align with client’s need.  Face Detection Game…

  • Week 5

    Week 5 Production: After the meeting last Friday and this Monday, the team and the client reached the consensus of pivoting towards facial recognition technology as our new focus. We presented four pitches to the client, and received feedback on them. The request was to design the game in a way that the players always…

STRINGERS

By Entertainment Technology Center