Introduction
Melody Painter is a god-like VR experience that allows you to create lands, trees, flowers, etc. in a world by playing different drafts of music. Made in collaboration with Vivi Yu and Vanilla Liu.
I composed the interactive progressive music, and wrote the MIDI communication and draft detection feature with Unity based on MiDiJack.
Designing game input
Since it’s in VR and hard to see where your hand is, I chose to only use five keys on a MIDI keyboard. Once you lay down your right hand to the correct keys, then starting from your thumb, they are numbered from 1-5. Inspired by a old game Loom, the mechanics is similar so that after you play the “land” draft “1-2-1-2”, a piece of land will be created on the center of your vision in the virtual world.
The MIDI communication with Unity is powered by MIDIJack.
Developing musical features
Input deciphering algorithm – improvisation friendly
Instead of checking every four notes and test if the draft is valid, notifying a lot of failure, my draft detection algorithm continuously listen to the combination of notes, until a correct draft is formed. For example, I can detect the land draft “1212” from within “51121234”, so that nothing you play is bad, giving more room to continuous improvisation.
MIDI and draft / improvisation testing
Harmonious sound effect
Also, a little trick to make the piece harmonious: originally, note 1-5 corresponds to C3 – G3. But whenever the back ground music shift its chords, I will switch the note F3 to F3# or C3 to C3#, thus whatever you play will just be harmonious to the background.
Composing progressive background music
No music -> calm music -> vibrant music -> night music
Inspire by a recent game Nier: Automata, I want the background music to be quiet at first, and as you create more things in the world, it gradually becomes more and more energetic, and the transition should be seamless.
- At first, the player faces an empty endless sea, and there should be no sound except some ocean noise.(Credit: https://freesound.org/people/amholma/sounds/372181/)
- When you successfully created your first land, the main theme will be played. It attempts to stimulate a feeling of awe and praise.
- As you more things in the world, you will eventually hear this most energetic version.
- As night comes, the night song will gradually play instead:
Result
We exhibited the project at Tech@NYU event and GSO Grant exhibition. Below are some of the feedback we received:
- “Relaxing.”
- “I feel like in heaven!”
- “Needs more visual feedback in VR for key pressing.”
- “Maybe you can change the color each time you create a new one.”
- “Improvisation should be rewarded.”
Most players were very excited after playing this game and ask if we could release it in the future.
I was also really happy to see that a player realized by himself that he could improvise, and he seemed to be enjoying that: (sadly I didn’t get the sound recorded)