03-Machine Learning in Sound+Music
In this class we’ll look at Machine Learning as it relates to Sound and Music. Many of the principles you learned in the previous classes will apply here, but manifest themselves in different ways.

We’ll look at a few concrete examples:

Pitch Detection
Using ML5, we’ll examine a user’s voice through the computer microphone and try to correctly identify musical notes. We’ll look at an example that makes a game of it, trying to get a user to sing the right note, and create a virtual piano whose keys are highlighted according to the note sang.

AI Duet
Because ML5 is a bit limited in its Sound+Music offerings at present, we’ll look into other libraries, dissecting Google’s AI Duet examples, which, like ML5, use Tensorflow under the hood, but incorporate a project called Magenta to facilitate sound-based Machine Learning. In this example, the user will play the keyboard to make musical notes, and the AI will attempt to accompany the user’s music with its own generated notes in an AI duet - matching similar rhythms, tones, and patterns.

Current State of Machine Learning in Sound+Music
We’ll look at some up-to-date examples of what people are doing with ML in Sound+Music today.

Let’s get started…

History


You could look at something such as a Chines Wind Chimes or the Aeolian Harp as the first device to allow for automatically generated music. 


Algorithmic Music 

Music became ‘algorthmic’ in the 1700s with Musikalisches Würfelspiel , a dice game that determined the recombination of works of famous composers segment by segment. 

In 1958 Iannis Xenakis used Markov Chains, a stochastic process by which a system can make predictions on the future based on its present state, to compose Analogique. He used what he called a table of ‘coherences’ - probabilities based on present states that a subsequent note would be generated. 



Markov Chains were used by David Cope in his Experiments in Musical Intelligence (or Emmy). They were semi-automatic systems with concepts of harmonics and musical grammar. 


Extrapolation of Music

But while Markov chains can only produce subsequences of the original composition, RNNs were implemented to attempt to extrapolate beyond that. After Bach attempts to identify stylistic regularities in generating new musical sequences:


In 2002 Doug Eck upgraded this model by using LSTMs, which could retain a longer memory of a musical sequence. Here’s some LSTM Blues


Doug now leads the Magenta initiatives at Google Brain, which we’ll be looking at in a bit.

Music is a fascinating use case for machine learning, as it is somewhat of a language - it has letters, unfolding over a composition through time, but unlike a language multiple notes can be played at one time, they can be sustained for different periods, or there can be periods of silence which are important to the overall composition. Music is simultaneously art and language.