MIDI-Driven Boss Fight Pipeline for Action Games
Overview:
Click the below link to view our project overview slides:
https://docs.google.com/presentation/d/1ZQMpqh_vuOywrfSwS5b3oGi1PL0YCmDNtOL_ray0I_0/edit?usp=sharing
Goal Experience
We record clips and rewrite the audio exprience to show as examples.
Previous Demo(ver3)：With Attack State And Counter-Attack Sync
Previous Demo(ver2)： Attack Sync with BGM
Previous Demo(ver1)：
Background
Beat matching is an audio feature that many videos have, but not many action games do this. Some action-rhythm games like Hi-Fi Rush, BPM: Bullet Per Minute, Crypt of the Necro Dancer use rhythm gameplay to encourage the player to keep in rhythm with BGM’s BPM, and thereby create rhythmic combat experience that fascinates their audience. However, this approach will block out those players who struggle with rhythm game, those who cannot catch the beat of a metronome.
Novelty
Instead of asking the player to catch the beat, my enemies dynamically attack on beat. This would help those players who struggle with rhythm and encourage them to match the beat by just following enemy attacks like they do in any other game. The original way is like solo dance, the dancer needs to be musically trained to make movements on beat. Our approach is like partner dance: there is one person (the enemy) leading and another person (player) following. All you need to do is to look at your partner carefully and follow his movements. If you follow well enough, your movements will be synchronized with the BGM!
Approach
The overall idea is utilizing predictable player’s reactions in gameplay, for example, in our melee sword fighting game, when an enemy attack you, players are expected to parry when enemy weapons are about to reach your body. If we make the boss ‘hear’ the music like a partner dancer does and attack according to the music, the player’s parrying reaction will also sync with beat. To achieve this, we construct a MIDI-Driven Enemy AI pipeline.
To achieve the audio experience goal, different roles in a combat team: designer, programmer, sound designer, composer and animator must follow a structured pipeline. The pipeline requires a sound designer to participate in combat design from an early stage, and some steps can be done automatically with tool scripts.
Potential of the MIDI-Driven Enemy AI
- For Composer: you can not only expect the player’s movement to match a certain beat, but you can also make the enemy’s animation match to your music segment (see the Sekiro Example, when enemy thrust)
- For Sound Designer: Think parrying sound effect as a special ‘instrument’ track of the BGM, which is played by player.
- For Animator: Creating animations for different rhythm patterns.
- For Combat Designer: Creating a sequence of synchronized moment with composition.
Technical Details
The overall concept is to use beat note in MIDI tracks as a cue to trigger different Enemy AI behavior and adjust animation speed to make hitbox sync with beats.
- How to sync an attack skill with 1 hitbox?
- How to sync an attack skill with n(n>=2) hitbox?
- How to sync with a ranged skill?
- What if the boss has many skills that need beat-matching?
- How to do beat matching synchronization under interactive music context?
- How can we automate the pipeline?
Project Description：
BGM Sync is a project that is aimed to build a innovative pipeline of dynamic audio in a combat experience. This project is supported by our instructor Dave Culyba and Ricardo Washington. We also have external expert, Göksu Uğur, and Ari Winters as our external supervisor
In typical combat-oriented video games, the music adjusts based on various elements such as the presence of enemies, player actions. The most common method for implementing these changes involves using multiple soundtracks that respond to the player’s decisions during combat.
In line with this framework, we categorized these elements into distinct player states and further divided them into various sub-states. Our research is focused on creating a more dynamic background music synchronization system, where every single action within combat, whether it’s lifting a sword or slashing toward the enemy, triggers alterations in the music.
Instead of merely layering soundtracks with or without specific instruments, we’re taking a more innovative approach by crafting different melodies with seamless transitions for these sub-states, while each action aligns with the musical beat.
Goal:
We aimed for a more subtle experience, so instead of a static process that requires players to hit specific notes or match a beat to complete a single piece of music, we minimized the prominence of the metronome in gameplay. This allows players to gradually perceive the dynamic musical shifts in combat and react to them. Consequently, we can offer a higher level of detail in synchronization, granting players increased freedom and a more pronounced sense of control over the combat experience.
Contact:
Producer Email: yiquz@andrew.cmu.edu
Address: Address: 700 Technology Drive Pittsburgh, PA 15219