New interactive environments

Post 3: Theories and Concepts

22.12.2010 16:22 by Katri Lindau
This post tries to look into visualized music through different generative content creation theories. As I've already talked about generative art and generative literature in my previous post (available here), I try to talk a bit about music presented using generative tools.

Music as one of the fine arts

As stressed out on my previous post the generative art can be considered as an art which is represented numerically, which has modularity and can be automated as well as it can have variability. As music is one of the fine arts then it is clear to me that music can be represented using colors, shapes and other visualized media tools to make the music not only heard but seen, too.

Music is about sounds, their sequence, loudness and rhythm. Music is also about silence and pauses between sounds (as have said Arvo Pärt, the famous Estonian composer who uses in his works lots of pauses, if we are talking about music the pauses are really important).

Visualizing the music

Visualizing the music have been an issue for centuries. Firstly there were developed the musical notation system - conventional sign system which allows writing down the music and represent it in the same way as the composer have written it down. As I have studied music previously several years I can add that music isn't only about presenting right tones on the right time, it is also about the players feeling and inner understanding of it.

But it is quite clear that musical notation system have been base for most of attempts to visualize music, even if the visualization doesn't seem that way at first (but it also depends on listener and viewer background - it is clear that to understand that the person must have some kind of musical studies background to have a knowledge about notation system).

Computer-based visualization

Music visualization tools are often used in many media player's software. There are used many different ways simple to advanced, most of them are using colors and/or giving and impression of loudness and tones which are used on the same time (the more there are different tones, the more there are different colors).

In Youtube we can find the works by the Music Animation Machine.
The simple example about Johan Sebastian Bach's Toccata and Fugue in D minor, organ:

If you have studied music you'll notice that here is used the musical notation system - dots are marking the tones and their lenght. If you are familiar with organ you would be able to play this piece by this video.

The next example from Youtube is showing clearly the connection between visualizing the music using notation system. It is Pachelbel's Canon in D, performed by Voices of Music.

And if you really aren't familiar of music, then following example should give the last understanding how this type of visualizing and the notation system are used to play the piece. This is Bach's contrapunctus 9 by Art of Fugue (Kunst der Fuge).

More interesting example also from the Music Animation Machine is following nocturne by Frederic Chopin (Opus 27, number 2) which is animated graphically to show the interval type (interval is one of the many musical terms which means the determined distance between tones).

More information about The Music Animation Machine can be found on their homepage.

Music is mathematical

Actually anybody even without proper education can generate similar videos because music is amazingly mathematic! You can study a bit about it using those tools on Ricci Adams page on internet. There are several works which are analyzing music mathematically, Lindsay Grace have even generated the music box - an algorithm for producing visual music (a short overview available here). Lindsay Grace's work proposes a method for producing music via visual composition in a computer game like environment. Her software applies the visual rules of standard emergent behaviors to the algorithmic arrangement of musical tones.

So if at first the Manovich's principles of New Media (numerical representation, modularity, automation, variability and transcoding) seems a bit too mathematical and rational for describing and representing the music, it is clear that the harmony what we are hearing is actually the outcome of the mathematical formulas behind them. So if there is possible to visualize music using justified patterns there is possibility to generate musical piece.

Juan c. Dürsteler has divided visualising Music into three options - visual entertainment (here can be represented all music software which are using auralisation to represent musical information, for example Windows Media Player), tools what are searching for musical similarities (identificates the similarity between different musical styles and different pieces, it doesn't actually visualise the music itself) and visualisations of the structure of music (for example using Arc Diagrams which search for strings that repeatedly appear within a sequential series of data).

Arc diagrams are most interesting. It generalizes the musical AABB notation by using a pattern-matching algorithm to find repeated substrings and then represents them visually as trasnlucent arcs. This is quite mathematical approach.

You can see the shape of song according to arc diagram from this page.


All of those previously submitted examples are analyzing existing pieces of music but they could be used on opposite way to generate music automatically according to given algorithm. It is noticed that Mozart' music is so beautiful to us because it's harmony. As music can be represented mathematically this harmony has formula, too. If we generate software which uses those given formulas to generate the music, the outcome could be as enjoyable as Mozart's or any other composer's work.

Grace, L. Music Box: An Algorithm for Producing Visual Music. Published in the IEEE proceedings for the International Conference on Advances is Computer Human-Interaction (ACHI, St Marteen, 2010)
The Music Animation Machine. See Inside Music.
Adams, R. Musictheory.
Wattenberg, M. Arc Diagrams: Visualizing Structure in Strings. Available here.
Dürsteler, J. C. Cisualising Music. Magazine. 2005. Available here.


Post a comment