Introduction

The benefits of music education, especially for children, are numerous. Time and again, research shows that learning to play an instrument and read music is correlated with improved hand-eye-coordination, pattern recognition, and problem-solving skills. Unfortunately, music and arts are often the first things to get cut from the curriculum when a school district finds itself short on funding. The VR Drums project endeavors to solve this problem by offering a tool for music education that is cheap, silent, and, most importantly, effective.

We intend to achieve this by embedding a virtualized instrument in an environment that contains all the tools necessary to practice one's technique, timing, and music literacy. We believe that by achieving a sufficient level of realism any skills learned in the virtual environment will translate to a physical drum set and vice-versa. Recent advancements in the field have yielded powerful virtual reality hardware at affordable prices. By keeping our application lightweight enough to run on a standalone VR rig, we believe we can make music education accessible at almost any budget.

While this began as a senior project from my time at Cal State, Northridge, it has evolved into a personal venture that I continue to improve.

About

The application provides the user with an eight-piece drum kit rendered in a VR space. A pair of virtual drum sticks tracks the user's hand movements, and colliders on each of the drum objects will produce a sound any time the head of a drum stick crosses the surface of a drum head. The pitch of the sound varies slightly depending on where the collision occurs. The virtual pedals of the hi-hat and bass drums are engaged when the user pulls the trigger on their left and right controller, respectively. The other objects in the scene are determined by which of the three play modes is currently active, as described below.

The heart of the VR Drums can be found in Tutorial Mode. In this mode, the drum heads are given colors and the user has a large, lined board centered in their field of view matching the format used in drum music notation. When the user presses the play button, game objects representing musical notes begin instantiating on the right side of the board and move left as if the user were reading a page of sheet music. Additionally, the notes are color coded to the drum they represent. For example, the top line of the clef represents the hi-hat, and so the notes that spawn on that line are the same color as the hi-hat itself. At the left of the board is a vertical bar which cues the user as to when a note is meant to be played. When the user strikes a drum object, the distance between the note and target bar is measured to determine how accurate the user's timing is. A score is generated based on their precision. Is the music going too fast? Just to the right of the drum set is a selector box where the user can adjust the tempo.

We believe this combination of features provides the user with an environment in which they can learn to sight-read sheet music, practice their technique, and improve their rhythm all at the same time.

Skills

When this project began, only a few members of the group had experience with Unity, and none of us had any experience with programming for virtual reality. As project lead, I identified six areas in which my team would need to become proficient:

• Designing custom game objects
• Building a user interface in a VR space
• Incorporating audio files in a way that gives the drum set a realistic sound
• Learning to use the Open XR standard and integrating VR hardware input devices
• Translating sheet music notation into a text file that could serve as an input for Tutorial Mode
• Integrating all of the above to create a fun and engaging application

My fantastic group mates took on the challenges of learning to use Blender to create realistic looking drums, sourcing sound clips, understanding how user interfaces are different in VR, and devising a system for converting sheet music into usable text files. That left me the tasks of learning Open XR and creating a cohesive program out of the assets my team created.

While Unity is known for having a thriving community of developers and excellent online documentation, learning to use Khronos's Open XR standard proved to be quite the challenge. The standard was so new at the time that I often found myself reading documentation or following tutorials that were out of date despite only being a few months old. There were times when I considered developing this project strictly for the Quest 2 and following Oculus's documentation, but ultimately I decided that cross-platform compatibility was an essential aspect of the project and stuck with Open XR. I am happy to have some experience with the standard as it seems to be the future of VR and AR development

Integrating assets between multiple programmers during a time of remote learning was unexpectedly complex as well. Of the six members in the group, only two of us had access to VR rigs and PC's capable of running them. This meant that the majority of the group was trying to develop assets for a virtual reality application without being able to test them in VR. In an effort to find ways for my team to be productive despite the obstacles caused by the pandemic, I learned how to create custom templates in Unity and how to use the XR Device Simulator. I made a template with the Universal Render Pipeline enabled and which contained XR Origin and XR Device Simulator prefabs. I then helped my team install the template on their individual computers. My teammates without VR rigs could use the XR Device Simulator to move around the VR environment and interact with VR assets, and my teammates with VR rigs could simply disable the simulator and use their hardware as normal. While this is more of a leadership success story than one of programming proficiency, I did become more familiar with Unity and Open XR in the process.

Technologies Used

Future Work

In Progress

• Adding more prefabs to represent different types of notes (quarter, eighth, etc .)
• Working to improve accuracy of scoring algorithm
• Adding haptic feedback to increase sense of realism
• Transcribing more sheet music to text files so the user has multiple options in Tutorial Mode

What's Next?

• Add more drum objects and let the user build their own drum kit
• Add more environments for the user to choose from
• Add option to turn on external cameras and make the drum set an AR element
• Add multiplayer functionality

Someday...

• Take advantage of new full-body tracking peripherals so that the user can use their feet to operate the virtual pedals
• Create custom drum stick controllers that would be tracked by the VR rig to increase realism