When Microsoft revealed the HoloLens in 2015 is was showcased in a lot of different scenarios. Some were leisure and gaming setups, others where more business oriented. Not long after that, details, including workshops at //BUILD 2015, began to come out. Most of the development tools where focused around Unity. Unity is a game engine, great for making games, but perhaps “overkill” (phun intended) when it comes to making small holographic programs that didn’t include classic game mechanics.
In this blog post we will setup and create a base application that will run on a HoloLens, and be a virtual music stand for playing an instrument.
Imagine that you’re playing a brass or wind instrument and your sheet music is visible right in front of you. By using Windows Mixed Reality, we’ll create a simple app that will display sheet music in front of you.
A real music stand has several annoying properties, they are bulky to carry around, and they create a physical barrier between the musician and the audience.  By providing a digital music stand we can also eliminate the need for printing the music first and give musicians the ability to carry unlimited amount of music with them.
A virtual music stand can have different properties than a physical one. For starters, holograms are not subject to gravity, so there is no need to “stand” on the ground, there is no need to anchor to the physical world, the important part is to be able to stay in front of the musician.
There should also be a simple way for the musician to turn pages and if our imagination starts running wild, perhaps some use of Windows Ink to make annotations on the music.
Digital sheet music is typically delivered in PDF format so we’ll also have to handle PDFs.

The tools we are going to use are:

  • Visual Studio 2015 (Support for 2017 is still coming)
  • Windows SDK version 1511 or newer
  • The UrhoSharp Hololens templates from the Visual Studio Marketplace

UrhoSharp is a great 3D library ported to DotNet by the Wizards from Xamarin. To get UrhoSharp, simple go to Extensions and Updates in Visual Studio, search the market place for Urhosharp

All we need for this set up is the templates.

After the templates are installed, create a new project using the UrhoSharp template:

As with all visual Studio templates, the first thing to do, is to check if there are NuGet packages that needs updating.
Now we are a ready, the template comes with a neat little demo showing the earth and the moon. We’ll not need that.
In UrhoSharp a Scene is the highest level, a Scene has a Node Tree that controls all the elements in the scene. In this App we’ll just have one single Node, a Plane element that is showing the music.

In the HoloApplication.Start() we define the Scene with a single Plane shape connected.

base.Start();

MusicNode = Scene.CreateChild();
MusicNode.Position = new Vector3(0, 0, 1.5f); // Lets place the music 1.5 meter away
MusicNode.Rotation = new Quaternion(0,90,-90);
MusicNode.Scale = new Vector3(0.4f * 1.414f, 0, 0.4f);
// Create a Plane component for holding the music
Music = MusicNode.CreateComponent<Urho.Shapes.Plane>();

// Override the default material (material is a set of tecniques, parameters and textures)           Music.SetMaterial(Material.FromImage(ResourceCache.GetImage(“Page_”+CurrentPage.ToString())));

After this we’ll quickly wire up some Cortana speech commands and tell the musician that we’re ready:

// requires Microphone capability enabled
await RegisterCortanaCommands(new Dictionary<string, Action> {
{ “bigger”,  () => MusicNode.Scale *= 1.2f },
{ “smaller”, () => MusicNode.Scale *= 0.8f },
{ “next”, () => NextPage() },
{ “previous”, () => PreviousPage() }
});
await TextToSpeech(“Lets play som music!”);

UWP has built-in support for PDF files, so loading a PDF file, and converting the pages to images files we can use as textures are quite easy:

private async Task LoadPdfFileAsync(StorageFile selectedFile)
{
PdfDocument pdfDocument = await PdfDocument.LoadFromFileAsync(selectedFile); ;
if (pdfDocument != null && pdfDocument.PageCount > 0)
{
PageCount = pdfDocument.PageCount;
for (int pageIndex = 0; pageIndex < pdfDocument.PageCount; pageIndex++)
{
var pdfPage = pdfDocument.GetPage((uint)pageIndex);
if (pdfPage != null)
{
MemoryStream ImageStream = new MemoryStream();
PdfPageRenderOptions pdfPageRenderOptions = new PdfPageRenderOptions();
pdfPageRenderOptions.DestinationWidth = (uint)(1024);
await pdfPage.RenderToStreamAsync(ImageStream.AsRandomAccessStream(),
pdfPageRenderOptions);
ImageStream.Position = 0;
Image i = new Image();
i.Load(new MemoryBuffer(ImageStream));
i.Name = “Page_” + pageIndex.ToString();
ResourceCache.AddManualResource(i);
pdfPage.Dispose();
}
}
}
}

A couple of house keeping methods are added to handle switching pages, wired up to the Cortana commands and air tap.

Finally, using the app looks like this through the HoloLens

 

The entire application is less than 200 lines of code, thanks to the power of the UWP framework and the UrhoSharp library. This combination is very powerful for creating apps and tools that do not require an entire gaming engine like Unity or Unreal. Since the operating system of the HoloLens is just Windows 10 that runs UWP apps, all the power of UWP is available on the HoloLens.
Grab the source from Github https://github.com/hougaard/HoloMusicStand and catch me on @eHougaard to talk mixed reality.

To learn more about Augmented and Virtual Reality you can attend the upcoming Microsoft webinar.