Artificial Intelligence Piano – My Hackweek Project

Share
Share

This article has been contributed by Lin Ma, Software Engineer and KVM Virtualization Specialist at SUSE. If you want to read more from Lin about virtualization, machine learning and artificial intelligence, have a look at the following articles:

 

With this article, I would like to introduce you to my SUSE Hackweek 19 project. If you worked on similar projects or topics, or if you would like to exchange experiences, please feel free to reach out to me.

Overview

As a Do-it-Yourself (DIY) enthusiast, I decided to have some fun during Hackweek with music and machine learning based on SUSE Linux Enterprise Server. Or, if you will, you could also think of my project as an Internet of Things (IoT) attempt based on SUSE products.

For those of you who don’t have much experience with music theory, it might be a painful process of learning how to read notes and playing them on a musical instruments. Thus, the goal of this project is to “read” a sheet music from a picture, then automatically translate it and play it on a physical musical instrument, in our case an electronic organ. And your problem is solved – thanks to our “Artificial Intelligence (AI) Piano” ;-).

 

Components of the Project

To create the AI piano, I used the following components:

Step 1 and step 2 run on the TaiShan 2280 server

Step 3 runs on Raspberry Pi 3.

  • A 61 keys electronic organ

My original idea was to buy a Yamaha P128 electric piano. But unfortunately it was too expensive for my project. Thus I used a 61 keys electronic organ instead.

  • Solenoids (model 0730), x 21

The solenoids take on the role of the fingers.

Each solenoid could be driven from a GPIO pin of the Raspberry Pi 3. However, as each model 0730 solenoid requires around 500 milliamps to actuation, this is beyond the ability of a Raspberry Pi 3. Thus I used ULN 2803 drivers to drive the solenoids.

 

 

Workflow

Regarding the workflow of how to create your AI piano, the following steps are involved:

Step 1: Use the neural network for the feature extraction of musical symbols from your sheet music image(s) serving as input.

Step 2: Use Optical Music Recognition (OMR) to recognize the musical notations (such as staves, measures, notes and so on) which were figured out by the neural network. Then save the result into a music XML file.

Step 3: Receive the music XML file via the TCP network and parse it. Then trigger the solenoids to play the piano.

 

Limitations of My Project

My piano (the electronic organ) has 61 keys and a range of 5 octaves. Based on cost-control considerations, I only bought 21 solenoids and mounted them on 21 white keys [C3 – B5] on the piano. In consequence, my AI piano can only handle 3 octaves. Moreover, it cannot play any half notes or semitones which are mapping to black keys on the piano.

This means that, while playing piano as indicated in step 3 above, any notation which raises the pitch of the note by one semitone or reduces the pitch of the note by one semitone will be ignored. Any standard pitch that doesn’t belong to these 3 octaves [C3 – B5], will forcefully be scaled up or scaled down to fit into these 3 octaves. The related part of music will seem to be out of tune in this situation.

 

Some Side Notes

  • My experience with circuit diagrams is very limited. Thus the circuit design of this project might not be perfect.
  • As the described project was a Hackweek project only, I didn’t take any action to reduce the noise generated by the solenoids.
  • Running neural network inference and OMR on the HUAWEI TaiShan 2280 ARM server is much faster than on Raspberry Pi 3. The bigger the size of the sheet music picture is, the bigger is the performance gap. As an example, I have added here a sheet music picture of the song “Twinkle Twinkle Little Star” with an image resolution of 2480*3352 and a file size 115KiB:
    • It takes approximately 30 seconds to run the above step 1 and step 2 on HUAWEI TaiShan 2280.
    • It takes approximately 8 minutes or longer to run the above step 1 and step 2 on Raspberry Pi 3.

 

Results

Below you can find a couple of pictures that show the sheet music images I used and the YouTube links with the respective final AI piano musical outputs.

Twinkle Twinkle Little Star

 

 

 

 

 

My Heart Will Go On

 

 

 

 

 

Summer

 

 

 

 

 

Kiss the Rain:

 

 

 

 

 

Overworld – Super Mario Brothers

 

 

 

 

Have a lot of fun every day – with SUSE Linux Enterprise!

Share
(Visited 14 times, 1 visits today)

Comments

  • Avatar photo victorhck says:

    Hey I can see Geeko shaking his curly tail dancing !!
    Have a lot of fun!!

  • Avatar photo chabowski says:

    Hey victorhck – nice imagination 😉 … Happy you liked the article!

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar photo
    9,111 views
    Meike Chabowski Meike Chabowski works as Documentation Strategist at SUSE. Before joining the SUSE Documentation team, she was Product Marketing Manager for Enterprise Linux Servers at SUSE, with a focus on Linux for Mainframes, Linux in Retail, and High Performance Computing. Prior to joining SUSE more than 20 years ago, Meike held marketing positions with several IT companies like defacto and Siemens, and was working as Assistant Professor for Mass Media. Meike holds a Master of Arts in Science of Mass Media and Theatre, as well as a Master of Arts in Education from University of Erlangen-Nuremberg/ Germany, and in Italian Literature and Language from University of Parma/Italy.