También disponible en Español

Inf@Vis!

The digital magazine of InfoVis.net

Visualising Music
by Juan C. Dürsteler [message nº 161]

Music has been made to hear and feel it. Musical notation added a certain way to visualise it, but we have never had so many possibilities to see its structure and the nature of the relations between its styles and individual pieces as we have today.

In issue number 37 we already talked about auralisation, the use of sound to represent information. There we spoke aboutearcons, brief audio fragments that signal certain events (a program error, for example).

Here we are more interested in the visualisation of music related information and with the visualisation of music itself than in the use of music to "hear" and ascertain the underlying patterns in information. 

WinMediaVidaFresa.gif (224838 bytes) MusicPlasmaEnyia.gif (158307 bytes) ShapeMusEnya.gif (101110 bytes)
Visual Entertainment. Many media players are accompanied by visualisation systems that "dance" following the rhythm of the music. In this case the Windows Media Player 
Source
:  Screenshot by the author
Click on the image to enlarge it
Musical similarity. As it appears at Musicplasma a visual search engine of performers and musical styles. See the text below for more information.
Source
:  Screenshot as it can be seen in the website.
Click on the image to enlarge it
Music structure. In the example the main page of "The Shape of Song" showing the structure of a piece of music, along with the value and duration of the notes. It's also possible to hear the song.
Source:  Screenshot as it can be seen in the website.
Click on the image to enlarge it

Music visualisation has several aspects of interest regarding the use that one can give to them:

  • Visual Entertainment.

  • Search for musical similarities

  • Visualisation of the structure of musical compositions.

Visual Entertainment

In this context we consider as visual entertainment the changing figures that we can find in different media players like, for example, in Winamp, Windows Media Player and other players that visually "dance" along with the music. 

WinAmp.gif (217684 bytes) WinAmp2.gif (198201 bytes) WinAmp3.gif (152436 bytes)
Some screenshots of different parts of the same song seen through different kinds of visualisations in Winamp 5.0. 
Source: Screenshot as it can be seen in the website.
Click on the image to enlarge it

Usually these visualisations use a representation of the frequency spectrum that the musical piece produces at each moment, combining it with different distorting effects. Its goal is purely aesthetical although, rigourously, the information "is there", even though we use it just for our visual pleasure.

Search for musical similarities.

MusicPlasmaEnyiaZoom.gif (173257 bytes)
Musicplasma. Once we introduce the name of our favourite singer, this application provides us with a map of the most related authors with the same.
Source
:  Screenshot by the author
Click on the image to enlarge it

The identification of the similarity between different musical styles and different pieces has a more practical interest. Let's imagine that we would like to find music similar to that of a particular performer or band. The name of a singer doesn't give us a clue if we haven't heard of her/him previously. Nevertheless different types of music share common features in many cases. 

Musicplasma presents a map where performers are grouped by ages and styles, linked by lines, so that the closer two performers are in the map the more they share a common style or pertain to the same musical age. 

Each singer or band is represented by a circle (they call it "halo") of a size proportional to the popularity or the strength with which he/she represents a particular style. 

Musicplasma gives very little information about its base technology. In fact it gives very little information on anything that could be of interest to us about it. 

According to John Battelle Musicplasma uses the results of filtraje colaborative filtering of Amazon.com (Customers who bought X also bought Y, Z and W...) to cluster and give a popularity ranking to the elements of the search.

In fact we could well say that Musicplasma is to musical search what KartOO or Grokker are to textual search.

Visualisation of the musical structure

At the other extreme of sophistication (and of power and future possibilities) lies the visualisation of the structure of music itself. A particularly appealing and elegant example is “The Shape of Song” built by Martin Wattenberg, author of the SmartMoney's Map of the Market based on the Treemap Technology

This excellent work is the application to music of Arc Diagrams. These types of diagrams have their origin in the search for strings that repeatedly appear within a sequential series of data.. Music is a clear example of a sequential series, where the variable ruling the sequence is time. Music is, moreover, a good example of how we can get complex melodies by combining repetitions of simpler musical passages (sub-structures).

In an arc diagram, the strings that are repeated, be them of one element or of many, are enclosed by an arc of a circle. The width of the arc encompasses the length of the repetition, so that long repeated strings are shown as wide arcs while short sequences, or individual notes, are shown as thinner arcs.

ShapMusEstructura.gif (7855 bytes) ShapMusMazurka.gif (22213 bytes)
Structure of an elementary piece. Notice how that arcs identify individual notes that are repeated in different parts of the score.
Source: Screenshot as can be seen in the web The Shape of Song
Click on the image to enlarge it.
Chopin's Mazurka. You can see in this case the complexity and richness of the piece along with the iterations of certain passages.
Source: Screenshot as can be seen in the web The Shape of Song
Click on the image to enlarge it.

This conceptually simple visualisation reveals, nevertheless, the structure of a melody or a complex piece such as Chopin's Mazurka and Bach's cantata or just a simple children's song.

Structure extraction, be it that way or attending to the variations of sound frequency, the spectral distribution of intensities or any other more sophisticated technique opens up another way in the search for similarities, allowing us to relate pieces with similar notational or spectral structure which, in turn, opens new doors to the visualisation of musical styles. This is what pursues, for example, the SIMAC, (Semantic Interaction with Music Audio Contents) project.

Incidentally The Shape of Song allows you to hear the music of its MIDI files catalogue online.

Music is an integral part of our life.. As it happens with so many other things, computers and Internet have changed our relationship with it. Maybe not in the deeper aspects of it, i.e. the transmission or, better said, the evocation of feelings, but in the accessibility to it and in the understanding of its structure.

Now we can also "see" the music in the structure of a song and in its relationships with the other songs.

Links of this issue:

http://www.infovis.net/printMag.php?num=37&lang=2   Number 37 Auralisation
http://www.musicplasma.com   Musicplasma's website
http://battellemedia.com/archives/001163.php   John Battelle's weblog
http://www.infovis.net/printMag.php?num=155&lang=2   Number 155 Collaborative Filtering
http://www.infovis.net/printMag.php?num=138&lang=2   Number 138 Grokker, or Visual Navigation
http://www.infovis.net/printMag.php?num=97&lang=2   Number 97 KartOO
http://www.turbulence.org/Works/song/   The Shape of Song website
http://www.bewitched.com/   Martin Wattenberg's website
http://www.infovis.net/printMag.php?num=52&lang=2   Number 52 The Evolution of Treemaps
http://domino.research.ibm.com/cambridge/research.nsf/0/e2a83c4986332d4785256ca7006cb621/$FILE/TR2002-11.pdf   Article about Arc Diagrams
http://www.semanticaudio.org/   SIMAC's website
© Copyright InfoVis.net 2000-2014