\pagestyle{fancy}
\cleardoublepage\chapter{Tasks in Music Information Retrieval}\label{tasks}
\Quote{If a composer could say what he had to say in words \\ he would not bother trying to say it in music.}{Gustav Mahler}

\noindent Among others, chord recognition\index{chord recognition} \cite{klee06, sumi08, cheng08} and onset detection\index{onset detection} \cite{lee06, duxbury03, dixon06, ewert09, grosche09} constitute topics of active research. They are important tasks in music processing\index{music processing} and music information retrieval\index{music information retrieval} (MIR) and are very helpful for the analysis of music data.

The first step in every music processing task -- and thus in music information retrieval -- is to extract suitable features to obtain key aspects while ignoring irrelevant ones. We shortly touch some key aspects of the feature extraction in the corresponding paragraph (\ref{tasks.chord_recognition} and \ref{tasks.onset_detection}).

There exist different visual representations of the audio data, such as a \emph{chroma-time representation}\index{chroma-based representation}. We do not introduce these here in detail. Instead, the interested reader is referred to the book by Meinard M{\"u}ller \cite{mueller}. Nevertheless, the basic intuition of these representations is made clear.

Throughout the thesis, we focus on the \emph{spectrogram representation}.

\section{Chord Recognition}\index{chord recognition}\label{tasks.chord_recognition}
A musical chord is a set of simultaneous tones. Looking at the chords over time, we can analyze the harmonic structure in a piece of music. Once the harmonic structure is known, a sequence of chords can give insights on higher-level structures such as phrases. The harmonic structure is also strongly related to perceived emotion. Thus, similar chord sequences can be observed in songs that are close in genre or emotion. Automatic chord labelings are very useful for people who want to do harmonic analysis of music and in applications like music search.

The basic idea behind chord recognition\index{chord recognition} is to inspect what frequencies are present at a certain time and match these to music notes. To achieve this, the audio signal is decomposed into spectral bands where each band corresponds to a pitch of the equal-tempered scale. Figure \ref{cr_frequencies_chroma} shows how such representations could look like. The first image gives a \emph{frequency-time representation}. Each row in a frequency-time representation stands for a frequency in Hz. From this representation we can go to a \emph{pitch-time representation}. In contrast to the frequency-time representation, the pitch-time representation shows the different MIDI pitches corresponding to the frequencies on the y-axis, instead of the frequencies themselves. A chroma-based representation\index{chroma-based representation} can be easily obtained from a pitch representation by adding up the subbands that correspond to the same pitch class, meaning that the chroma-time representation\index{chroma-based representation} accumulates all occurrences of C, C$^\sharp$, D, D$^\sharp$ and so on in one row per pitch class.
%For chord recognition\index{chord recognition} this is sufficient, since we only need to know which notes of a pitch class are played together and not the exact pitch. Thus, if we take the notes with the highest intensity, we can tell which chord is played at which time.
Please note that we are assuming the equal-tempered scale as used in Western music, meaning that e.g.\ the notes D$^\sharp$ and E$^\flat$ are considered the same.
In other words, by use of the frequency-time representation we are able to extract the pitches that are present at a certain time in a piece of music. Thus, it is crucial that we can clearly distinguish between different frequencies. To achieve this, we have to emphasize horizontal structures. Note that emphasizing vertical structures would result in a blurring across frequency bounds.
% Obviously, there is some further knowledge of music involved since we have to know which notes build which chord, e.g.\ that the notes E$^\flat$, G and B$^\flat$ played together build the chord E$^\flat$ Major.

Chord recognition comes with some challenges. The biggest challenge is rhythmic accompaniment in every form, be it drums or other percussion, because the sounds produced by drums or percussion contribute to every single frequency. Let us assume that the third of a chord is somehow noisy. This would prevent us from deciding between Major and Minor or -- even worse -- would lead to a wrong decision.

A description of template-based chord recognition can be found in Konz et al. \cite{konz10}.

\begin{figure}[htb]
	\centering
	\includegraphics[scale=0.75]{figures/Beatles_AllMyLoving_Beatles_1963-WithTheBeatles-03_0-25-1.eps} \\[0.3cm]
	\includegraphics[scale=0.75]{figures/Beatles_AllMyLoving_Beatles_1963-WithTheBeatles-03_0-25-pitch.eps} \\[0.3cm]
	\includegraphics[scale=0.75]{figures/Beatles_AllMyLoving_Beatles_1963-WithTheBeatles-03_0-25-3.eps}
	\caption[Three representations of ``All My Loving'' by The Beatles.]{Frequency-time, pitch-time and chroma-time representation\index{chroma-based representation} of ``All My Loving'' by The Beatles (see also number 1 in Table \ref{appendix.table_chord_files}). \textbf{Top:} Frequency-time representation. The rows correspond to frequencies in Hz. Colors correspond to the logarithmic magnitude. \textbf{Middle:} Normalized pitch-time representation. The spectrogram is cropped to show only the MIDI pitches 65 to 85 and is rescaled to fit the color coding. The MIDI pitch 69 corresponds to the note A4. \textbf{Bottom:} Normalized chroma-time representation\index{chroma-based representation}.  In this representation all the occurrences of a note (also shifted by octaves) are matched to one corresponding row (corresponding pitch classes).}
	\label{cr_frequencies_chroma}
\end{figure}

\section{Onset Detection}\index{onset detection}\label{tasks.onset_detection}
Also an interesting task in music processing\index{music processing} is onset detection\index{onset detection}. At first glance, this seems to be a well-defined task: the aim is to find the starting time of each musical note. In contrast to chord recognition, onset detection relies on vertical structures in the spectrogram. In other words, we are interested in a good isolation of events for each point in time (i.e.\ \emph{when} a contribution happens) where it is not important to know \emph{which} frequencies make a contribution. As a consequence, the goal is to enhance vertical structures.

In polyphonic music, where nominally simultaneous notes or chords might spread over a few tens of milliseconds, we see that there is some blurring involved. Another difficult scenario is whenever we have fluent note transitions which is often the case for classical music dominated by string instruments. These problems are depicted in Figure \ref{od_two_examples}. For instruments with long attack times it is difficult to define an unambiguous and precise onset time for the notes they produce.

%Usually with piano or guitar music as well as percussion, we have an increase of the energy in the spectrogram\index{spectrogram}. \emph{Short-time mean-square power features} (STMSP features) measure the local energy content of each subband and indicate the presence of certain musical notes within the audio signal. The onset features are obtained by taking the derivative of the energy curves. Due to the color coding, the increase in energy
The presence of vertical structures can be seen very well in the top spectrogram\index{spectrogram} of Figure \ref{od_two_examples}, especially in the time frame from the beginning until approximately 3 seconds. But what about the example on the bottom of Figure \ref{od_two_examples}? There we don't visually recognize any sharp boundaries. And in fact, there are no sharp boundaries. Instead, listening to this excerpt reveals that there is a melody which flows ``above'' the rhythmical accompaniment that acts like some sort of blur. Due to this fact, it is even difficult at some point to acoustically judge where an onset lies. The \emph{novelty curves} of these two examples are given in Figure \ref{od_two_examples_novelty}. A novelty curve is a representation that depicts the changes in energy and is computed e.g.\ by analyzing the signal's spectral content. As a consequence, it is a suitable indicator for note onsets. A more detailed introduction to novelty curves can be found in \cite{grosche09-2}. In Figure \ref{od_two_examples_novelty} we clearly see that in the second example (on the bottom) the novelty curve changes a lot. If we look at e.g.\ what happens around the time 5 seconds and in the time frame from about 8 seconds to 10 seconds and compare this with the manually annotated ground truth (red lines), we notice a lot of energy fluctuations -- in other words onsets -- where actually no onsets should be.

This is a perfect example that the onset detection\index{onset detection} problem is by far not easy and not well-defined. Further challenges are provided in \cite{leveau04}.

Onset detection\index{onset detection} is an essential step towards advanced tasks like tempo analysis, beat tracking, music synchronization and automatic transcription as well as non-linear time-scaling and pitch-shifting. Due to the fact that music is event-based, the segmentation into individual note events greatly helps in editing and analyzing audio.

\begin{figure}[htb]
	\centering
	\includegraphics{figures/BM02A_meinard_se-1.eps} \\[0.3cm]
	\includegraphics{figures/RWC-MDB-C-2001-M03_tr04_Borodin_Sec1-39-1.eps}
	\caption[Two frequency-time representations.]{Two frequency-time representations. \textbf{Top:} Excerpt of the piano etude Op. 100, No. 2 by Friedrich Burgm{\"u}ller (see also number 1 in Table \ref{appendix.table_onset_files}). Onsets are pretty clear to spot. \textbf{Bottom:} Excerpt of the string quartet No. 2 (Notturno) by Alexander Borodin (number 7 in Table \ref{appendix.table_onset_files}). Onsets are difficult to spot.}
	\label{od_two_examples}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics{figures/BM02A_meinard_se-3.eps} \\[0.3cm]
	\includegraphics{figures/RWC-MDB-C-2001-M03_tr04_Borodin_Sec1-39-3.eps}
	\caption[Two novelty curves.]{Novelty curves corresponding to Figure \ref{od_two_examples}. Depicted are the changes in energy as well as the manually annotated ground truth (red lines).}
	\label{od_two_examples_novelty}
\end{figure}

\section{Quality Measure}\label{tasks.quality_measurement}
In the case that there is a ground truth available, we can measure the quality of a processing step by investigating the quantities \emph{precision}\index{precision}, \emph{recall}\index{recall} and \emph{F-measure}\index{F-measure}, see also \cite{precisionrecallwiki}. We introduce the precision\index{precision} $P$ as
\begin{equation}\label{eq_precision}
	P = \frac{|\{\mbox{relevant}\} \cap \{\mbox{retrieved}\}|}{|\{\mbox{retrieved}\}|},
\end{equation}
and the recall\index{recall} $R$ as
\begin{equation}\label{eq_recall}
	R = \frac{|\{\mbox{relevant}\} \cap \{\mbox{retrieved}\}|}{|\{\mbox{relevant}\}|}.
\end{equation}

%With \eqref{eq_precision} and \eqref{eq_recall} we can formulate the F-measure\index{F-measure} $F$
Note that precision measures how specific the detection is, while recall measures its exhaustiveness. None of these aspects can be neglected in the quality measurement. To give nevertheless one single numerical measure of quality, it has been proposed to use a weighted harmonic mean of precision and recall, the so-called $\text{F}_\beta$-measure
\begin{equation}\label{eq_fmeasure}
	F_\beta = (1+\beta^2) \cdot \frac{P \cdot R}{\beta^2 \cdot P + R}.
\end{equation}
We stick to the case $\beta = 1$, i.e.\ the $\text{F}_1$-measure, because precision\index{precision} and recall\index{recall} are evenly weighted. For the sake of simplicity, we refer to the $\text{F}_1$-measure as F-measure\index{F-measure}.

Compared to the arithmetic mean, the harmonic mean exaggerates the influence of lower input values and thereby prevents a good F-measure rating when one of the two original measures, precision and recall, is particularly poor.