Activity 9 Applications Morphological Operations 2 of 3 : Playing Musical Notes though Image Processing

For this activity, we will try to read a sheet music via image processing and play the music using Scilab. A sheet music is basically black and white so it is easy to separate the notes from the background by thresholding and applying morphological operations. 

The first task for this is to find a sheet music. I used the song Clementine and the sheet music is found in Figure 1.

Figure 1. Sheet music for Celementine

To make processing easier we divide the sheet into different lines. In this case we have four lines as shown in Figure 2.

Figure 2. The 4 lines in the sheet music of Clementine were separated

 (line 1 to 4 from top to bottom)

We then binarize the images with a threshold of 0.95 and invert them such that the background is black and the foreground is white. We disregard the area of the image where the clef is located. Using morphological operations, specifically CloseImage(),OpenImage() and ErodeImage() we reduce the notes into small blobs. We then take the centroid to further reduce the blobs of the notes into points. Shown in Figure 3 are the centroids of each note's head which were dilated with a circle of radius 2 for visibility.

  

    

  

Figure 3. Blobs representing the positions of the centroids of the notes' heads

The lines of the staff are separated by 10 pixels so each pitch has an interval of 5 pixels. We assign the pitch of the notes by checking the pixel coordinates and determining the interval in which it belongs. The code used to do this is shown in Figure 4. This is in order of decreasing pitch from the G2 to C1 (middle C). After doing this, we have the list of pitches for each note in each line.

Figure 4. Code used to determine the pitch of each note; y is the list of coordinates of each note along the y axis (row) and notes is a list equal to y whose values are replaced by the corresponding pitch of each note

The next consideration is the duration of each note. There are four different notes in this piece (i.e. half note, quarter note, eighth note, and sixteenth note) with one special character (i.e.dot). These notes have different areas. In order to distinguish them we took the area of some notes and used them for thresholding. Of course, we cannot use the blob image previously used to determine pitch since we need the notes to be whole. Applying only the OpenImage() and CloseImage() operations used, we get the binarized images in Figure 5.

  

  

  

Figure 5. Binarized image of the notes

The presence of the barlines is not a concern since it can be filtered out when categorizing the notes using their areas. The code used to do this is shown in Figure 6.

Figure 6. Code used to distinguish the notes using their areas. notes is a list where the corresponding durations are saved (half note = 2, quarter note = 1, eighth note = 0.5, sixteenth note = 0.25 and dot = 1.5*preceding note's duration)

Finally, we have the notes for each line and their corresponding pitch. These were saved in text files and then loaded in Scilab and input to the function note to produce our sound file. :D

Listen here:

I give myself a grade of 10/10 for successfully producing the required output. 

I would like to thank Ms. Maria ELoisa Ventura for helpful discussions.

References:

[1] M. Soriano,Activity 9 Applications Morphological Operations 2 of 3 : Playing Musical Notes though Image Processing, AP186 Manual, 2012

[2] www.phy.mtu.edu/~suits/notefreqs.html