% Template for MIA electronic tresentation, 2002, Pavel Mrazek
\documentclass[miaitalkstyle]{cmptalk}

\hypersetup{bookmarks=false,pdfstartpage=1}%,pdfpagemode=FullScreen}
%\hypersetup{pdftitle=,pdfauthor=,pdfsubject=,pdfkeywords=}

\usepackage{bigfig} % to make images full-screen on click
\usepackage{color}
\usepackage{graphicx}
\usepackage{texpower}

\definecolor{titleblue}{rgb}{0.05,0.2,0.75}
\newcommand{\important}[1]{{\textit{\color{red}{#1}}}}
\newcommand{\comments}[1]{}
\newcommand{\stitle}[1]{{\Large \bfseries \color{miablue}{#1}}}

\author{Jan Hendrik Dithmar \\ \texttt{dithmar@mia.uni-saarland.de}}
\title{\color{miablue}{Audio Denoising Using \\ Image Processing Methods}}

\begin{document}
\maketitle
\slidetitle{\normalsize\bfseries%
    Mathematical Image Analysis Group, Saarland University\\
    Oberseminar Mathematische Bildanalyse 
%    \href{http://www.mia.uni-saarland.de}{http://www.mia.uni-saarland.de}
}

\subhead{Outline:}
\begin{enumerate} 
	\item Motivation
	\item Wav-File Structure
	\item Denoising Method, Method Noise
	\item NL-means Algorithm
	\item Ideas, Inpainting
	\item Problems / Questions
	\item Applications
\end{enumerate}

% -----------------------------------------------------------------------
\foilhead{Motivation}

\stitle{Motivation}

\begin{itemize}
	\item We have a noisy audio signal and want to denoise it.
%	\item Noise can be a cough (Husten) or different types of clicking and many more.
	\item How does, e.g. Gaussian noise look like in images?
	\item How does noise \textbf{sound}? \\
\begin{center}
	\includegraphics[width=0.3\textwidth]{Pic/couple.pdf} \quad \quad
	\includegraphics[width=0.3\textwidth]{Pic/noise.pdf} \\[0.3cm]
\textbf{Left:} Original image. \textbf{Right:} Noisy version with Gaussian noise ($\sigma = 80$).
\end{center}
\end{itemize}


% -----------------------------------------------------------------------
\foilhead{Wav-File Structure}

\stitle{Wav-File Structure}

\begin{itemize}
	\item A Wav-file is a container for audio data.
	\item Invented by Microsoft and IBM.
	\item It has no unique structure. For example it can have a lot of chunks.
	\item PCM is uncompressed audio. ADPCM and MP3 are examples for compressed audio.
%	\item We concentrate on the uncompressed case, i.e. PCM (everything else is not treated).
\end{itemize}
I will concentrate on the format specified on the next slide.

\newpage

\begin{center}
\begin{tabular}[b]{p{2.2cm}|l|p{2.2cm}|l}
	Position (in bytes) & Field Name & Field Size (in bytes) & Description \\ \hline
	0 & Chunk ID & 4 & "RIFF" \\ \hline
	4 & File Size & 4 & Entire File Size - 8 \\ \hline
	8 & File Format & 4 & "WAVE" \\ \hline
	12 & SubChunk1 ID & 4 & "fmt " (header signature) \\ \hline
	16 & SubChunk1 Size & 4 & For PCM files, this will be 16. \\ \hline
	20 & Audio Format & 2 & Should be 1 for uncompressed audio. \\ \hline
	22 & Number of Channels & 2 & Could be 1 = mono, 2 = stereo, ... \\ \hline
	24 & Sample Rate & 4 & 44100 Hz for CD quality \\ \hline
	28 & Byte Rate & 4 & Sample Rate $\times$ Block Alignment \\ \hline
	32 & Block Alignment & 2 & Channels $\times$ Bits Per Sample $\div$ 8 \\ \hline
	34 & Bits Per Sample & 2 & 8, 16 or 24 \\ \hline
	36 & SubChunk2 ID & 4 & "data" (header signature) \\ \hline
	40 & Data Size & 4 & Number of bytes following the header. \\ \hline
	\multicolumn{4}{c}{DATA} \\ \hline
\end{tabular}
\end{center}
Adapted from: \texttt{http://de.wikipedia.org/wiki/Wav}

%\newpage
%
%{\large \color{miablue}{How is the data stored?}}
%\ \\ \ \\
%\begin{center}
%	\includegraphics[width=0.8\textwidth]{Pic/WAV-frame.pdf}
%\end{center}
%Stereo signal in WAV format. The samples \texttt{ch0} and \texttt{ch1} together represent a (sample) frame. For mono audio data, one frame would be one sample. Source: \texttt{http://de.wikipedia.org/wiki/Wav}
%
\newpage

\textbf{\large \color{miablue}{Overview}}
\ \\
\begin{center}
	\includegraphics[width=0.85\textwidth]{Pic/wave-bytes.pdf}
\end{center}
First 72 bytes of an example WAV file. \\ Source: \texttt{http://ccrma.stanford.edu/courses/422/projects/WaveFormat/}


% -----------------------------------------------------------------------
\foilhead{Denoising Method}

\stitle{Denoising Method}

\begin{itemize}
	\item Formally, a denoising method $D_h$ is defined as a decomposition
		$$v = D_h v + n(D_h, v)$$
		where $v$ is the noisy image, $h$ is the filtering parameter which usually depends on the standard deviation of the noise.
	\item $D_h v$ is smoother than $v$.
	\item $n(D_h, v)$ looks like the realisation of white noise.
\end{itemize}
We know how a smooth image looks like. But how does a smooth audio signal sound? Tinny (blechern)? Flannelly (dumpf)?
%How can one apply that to audio signals? How does a smooth audio signal sound?% Is it for example blurry? How does a blurry audio signal sound?


% -----------------------------------------------------------------------
\foilhead{Method Noise}

\stitle{Method Noise}

\begin{itemize}
	\item Idea: \\
		Taking the difference of the original image and its denoised version
		$$u - D_h u$$
		where $u$ is an image and $D_h$ is a denoising operator with filtering parameter $h$ should only give you noise.
	\item Problem: \\
		In general, you get structures of the image, which give you a good measurement whether your denoising destroys too much of the original data.
	\item Goal: \\
		Port that to audio signals since it is a good quality measurement for a filter.
	\item Open: \\
		How does noise sound? Intuitively, it should not contain any structures like rhythm, for example.
\end{itemize}

\newpage

\begin{center}
	\includegraphics[height=0.38\textheight]{Pic/buades_morel_non-local_algorithm-20.pdf} \quad
	\includegraphics[height=0.38\textheight]{Pic/buades_morel_non-local_algorithm-21.pdf} \quad
	\includegraphics[height=0.38\textheight]{Pic/buades_morel_non-local_algorithm-22.pdf} \\[0.5cm]
	\includegraphics[height=0.38\textheight]{Pic/buades_morel_non-local_algorithm-23.pdf} \quad
	\includegraphics[height=0.38\textheight]{Pic/buades_morel_non-local_algorithm-24.pdf} \quad
	\includegraphics[height=0.38\textheight]{Pic/buades_morel_non-local_algorithm-25.pdf}
\end{center}
Part of the image Lena. \textbf{Top left:} Original image. \textbf{Others:} Displaying the image difference $u - D_h u$ after applying different filtering methods. $h$ is fixed in order to remove noise of standard deviation 2.5. \textbf{Authors:} A. Buades, B. Coll, J.-M. Morel.


% -----------------------------------------------------------------------
\foilhead{NL-means Algorithm}

\stitle{NL-means Algorithm}

\begin{itemize}
	\item Given a discrete noisy image $v = \{v(i) \, | \, i \in I\}$, the estimated value $NL[v](i)$ for a pixel $i$ is computed as a weighted average of all the pixels in the image
		$$NL[v](i) = \sum_{j \in I} w(i,j)v(j).$$
	\item The family of weights $\{w(i,j)\}_j$ depends on the similarity between the pixels $i$ and $j$. It satisfies the conditions $0 \leq w(i,j) \leq 1$ and $\sum_j w(i,j) = 1$.
	\item The similarity between two pixels $i$ and $j$ depends on the similarity of the intensity grey level vectors $v(\mathcal{N}_i)$ and $v(\mathcal{N}_j)$, where $\mathcal{N}_k$ denotes a square neighbourhood of fixed size and centered at a pixel $k$.
%	\item The similarity between two pixels $i$ and $j$ depends on the similarity of the intensity grey level vectors on a square neighbourhood $\mathcal{N}_k$ of fixed size and centered at a pixel $k$.
	\item The similarity is measured as a decreasing function of the weighted Euclidean distance $||v(\mathcal{N}_i) - v(\mathcal{N}_j)||_{2,a}^2, a > 0$.
\end{itemize}

\newpage

\begin{center}
	\includegraphics[height=0.8\textheight]{Pic/buades_morel_non-local_algorithm-1.pdf}
\end{center}
Scheme of NL-means strategy. Similar pixel neighbourhoods give a large weight, $w(p,q_1)$ and $w(p,q_2)$, while much different neighbourhoods give a small weight $w(p,q_3)$. \textbf{Authors:} A. Buades, B. Coll, J.-M. Morel.


% -----------------------------------------------------------------------
\foilhead{Ideas}

\stitle{Ideas}

\begin{itemize}
	\item Frequency analysis using the Short-Time-Fourier-Transform (STFT) with a Hamming window
		$$w(n) = 0.53836 - 0.46164\cos\left(\frac{2 \pi n}{N-1}\right)$$
		where $N$ is the width of  a discrete-time window function in samples (typically a power of 2) and $n$ an integer value with $0 \leq n \leq N-1$.
		\begin{center}
			\includegraphics[height=0.45\textheight]{Pic/hamming.pdf}
		\end{center}
		Hamming window. Source: \texttt{http://en.wikipedia.org/wiki/Hamming\_window}
\end{itemize}

\newpage

\begin{itemize}
	\item Apply different filtering methods, for example 
		\begin{itemize}
			\item Gaussian filtering,
			\item anisotropic filtering,
			\item total variation minimisation,
			\item neighbourhood filtering.
		\end{itemize}
	\item Synthesize audio information by using the neighbourhood.
	\item Calculate the method noise of the different filtering methods to judge whether a method is good or not. But this would only work if the intuition about the method noise is true.
\end{itemize}


% -----------------------------------------------------------------------
\foilhead{Inpainting}

\stitle{Inpainting}

\begin{itemize}
	\item First step: \\ Try to find parts of the sound file that fit in the hole to be filled.
	\item Problem: \\ You hear a digital click if the sound file is not cut properly. In general, this results in discontinuities in the audio signal. \\
		This means, one can not fill the hole by simply plugging in a sound file which has the same size. Care must be taken at the boundaries!
	\item Remedy: \\ Use another line/channel where you can fade in the block to be filled-in and fade out afterwards.
\end{itemize}


% -----------------------------------------------------------------------
\foilhead{Problems / Questions}

\stitle{Problems / Questions}

\begin{itemize}
	\item How does a smooth signal sound?
	\item How does noise sound?
	\item Not everything that sounds the same is actually the same. \\
		$\rightarrow$ need a suitable threshold!
		\begin{center}
			\includegraphics[height=0.34\textheight]{Pic/sin1.pdf} \quad
			\includegraphics[height=0.34\textheight]{Pic/sin2.pdf}
		\end{center}
	\item What does averaging do with the signal? What is removed? What is introduced? How does the signal sound afterwards?
	\item How much audio data can one synthesize based on the neighbourhood?
	\item Background sound has definitely to be kept. Removing that results in an unnatural listening experience!
\end{itemize}


% -----------------------------------------------------------------------
\foilhead{Applications}

\stitle{Applications}

\begin{itemize}
	\item Remove local distortions like a sneeze (Niesen) or a cough (Husten).
	\item Remove distortions caused by the playing technique of records (Schallplatten).
	\item Automatic cutting of music.
\end{itemize}


% -----------------------------------------------------------------------
\foilhead{Summary}

\stitle{Summary}

\begin{itemize}
	\item It looks in theory that the ideas of the NL-means algorithm work also for audio signals.
	\item There are a lot of open questions especially when it comes to smoothing or replacing parts of the signal.
	\item Some things only work because music is highly redundant.
\end{itemize}


% -----------------------------------------------------------------------
\foilhead{References}

\stitle{References}

\begin{itemize}
	\item A. Buades, B. Coll, J.-M. Morel: A non-local algorithm for image denoising. \textit{IEEE International Conference on Computer Vision and Pattern Recognition}, vol. 2, p. 60-65, 2005.
	\item A. A. Erfos, T. K. Leung: Texture Synthesis by Non-parametric Sampling. \textit{IEEE International Conference on Computer Vision}, vol. 2, p. 1033-1038, 1999.
	\item \texttt{http://de.wikipedia.org/wiki/Wav}.
	\item \texttt{http://en.wikipedia.org/wiki/Window\_function}.
\end{itemize}


% -----------------------------------------------------------------------
\foilhead{Thank you}

\ \\ \ \\ \ \\ \ \\ \ \\ \ \\ \ \\ \ \\
\begin{center}
{\Huge \color{miablue}{Thank you for your attention!}}
\end{center}


% -----------------------------------------------------------------------
%\foilhead{...}
%
%\stitle{...}

\end{document}
