Search

Research

HOME
Search
On-going Research

Topic 1. Text Query based Audio Retrieval

•
Retrieving audio signals using their sound content textual descriptions (i.e., audio captions).
•
Text query composed of manually written audio captions.
•
For each text query, the goal of this task is to retrieve audio files from a given dataset and sort them based their match with the query.

Topic 2. Automated Audio Captioning

•
The task of general audio content description using free text.
•
An inter-modal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal.
•
Modeling concepts (e.g. "muffled sound"), physical properties of objects and environment (e.g. "the sound of a big car", "people talking in a small and empty room"), and high level knowledge ("a clock rings three times").

Research 1. Audio-Text Data Augmentation

Research 2. Audio Captioning

Research 3. Audio-Text Retrieval

Audio-Language Multi-modal Learning
Multi-modal Learning

Topic : Multi-modal sensor based speech synthesis

•
Also known as “silent speech”
•
Synthesizing vocal audio by capturing the muscle signals produced during silent speech
•
Mainly use facial electromyography (EMG) sensor
•
Demo audio

Research 1. EMG-based Speech Content Representation

Research 2. Voice Conversion

Research 3. Domain Adaptation

XVoice : Multi-modal sensor based speech synthesis
Multi-modal Learning
Speech Processing

Custom Music Composition with AI Systems

The overall pipeline of generating custom music that reflects the user’s taste.
•
The "AI Producer: Developing technology of custom music composition" project is a multi-faceted initiative aimed at creating personalized multi-track music compositions based on a user's musical taste. This is achieved by using a reference song and keyword-based queries for lyric generation, leveraging artificial intelligence to create unique musical pieces.
•
The project involves multiple modules, each designed to handle a different aspect of the music generation process. These include musical instrument retrieval, which identifies and extracts desired musical instruments from a reference audio file; digital audio effects modeling, which applies audio effects and performs automatic multi-track mixing and mastering; and keyword-based lyrics generation, which creates song lyrics based on keywords provided by the user.
•
While each module operates independently, they all work in synergy to produce a complete multi-track music piece. Each of these modules relies on the reference music recording provided by the user, serving to customize the final output according to the user's taste and preferences. This combination of modules, each with its unique function, makes the project a comprehensive solution for custom music composition.

Musical Instrument Retrieval

•
Musical instrument retrieval is the task of searching and identifying specific musical instruments within mixed music recordings of a collection of different musical instruments, aiming to assist music producers in selecting appropriate virtual instruments for their projects.
•
This process addresses the time-consuming and inefficient nature of manually listening to and comparing each instrument sample to find the desired sound.
•
Performing instrument retrieval not only helps in locating the desired musical instrument but also provides valuable information for instrument recognition, enabling the identification of instruments present in a given audio mix.
related publications from MARG on musical instrument retrieval

Digital Audio Effects Modeling

AI Producer: Developing technology of custom music composition
Signal Processing
Musical Information Retrieval (MIR)
Music Production

Music

Do Sharp and Flat Keys Evoke Opposing Sensory Perception?: Focusing on the Crossmodal Influence Inherent in the Pitch and Characteristics of Enharmonic Key
•
Ahyeon Choi(chah0623@snu.ac.kr), Woojae Cho
•
Our study aimed to investigate the effects of sharp and flat keys on the perception of brightness, warmth, and sharpness, given that these keys are associated with tension and relaxation respectively in music.
•
We found that these keys significantly influenced all three sensory perceptions among 30 musicians, implying that musical keys can affect listeners' perception, thereby providing valuable insights for composers and performers to create music that triggers specific sensory responses.
Towards a New Interface for Music Listening: YouTube Case Studies
•
Ahyeon Choi(chah0623@snu.ac.kr), Eunsik Shin, Haesun Joung
•
Despite the rise of various music streaming services, an increasing number of users are choosing YouTube for music consumption due to its unique features.
•
Our research, involving semi-structured interviews with 27 active YouTube users, reveals systematic usability features of Youtube as a music streaming service such as the provision of a rich array of music-related videos and the ability to explore diverse music genres, share unique playlists, and interact with others.
•
Furthermore, we have developed the wireframe for a new youtube UI for music listening, providing practical solutions to improve user satisfaction. This work has significant implications for YouTube and other music streaming platforms aiming to enhance their user experience.

Speech

Auditory Perception & Cognition
Perception & Cognition

1) Data simulation algorithm of disaster emergency situations

Figure 1. Data simulation.
•
We build algorithms using pyroom-acoustics library that simulate the multi-channel waveform acquired on the flying drone.
Figure 2. Design of drone for test environment
•
We designed drones with a multi-channel mic array for testing environmental data acquisition and testing our model on real-world scenarios.

2) Multi-channel multi-class classification and DoA estimation

Figure 2. Classification and DoA estimation
•
We build multi-channel classification models to detect and classify emergency rescue sounds (class: male, female, baby).
•
Also, we build multi-channel Diection-of-Arrivial (DoA) models to estimate the direction of the sound sources
•
The models should be robust to drone noise and the status of the drone (hovering or moving).
•
A low-flops regime for drone-integrated model scenarios is also one of the major parts of the project. We test our algorithm on NVIDIA Jetson devices considering lightweight hardware costs.
Multi-channel DoA estimation of integrated cognitive drone for disaster emergency situations
Signal Processing

1. Introduction

‘Dance generation’ is a task of music-based motion generation that uses a rule-base or deep learning model to make realistic dance motions. In the past, rule-based models were mainly used, but recently, with the development of deep learning techniques, methods using deep learning have become a trend. This task requires musical knowledge and an understanding of human motion, especially dance. To this end, we use a model that has learned the relationship between musical features and dance movements to generate dance motion suitable for music.

2. Research

Automatic Choreography Generation with Convolutional Encoder-decoder Network

In general, when people dance, they dance to match the music. In fact, dance is closely related to music. Many elements of dance follow the temporal and spectral characteristics of music. Therefore, it is necessary to understand both music and dance in order to create a dance that matches the music. However, it is very difficult for ordinary people who do not know much about dance or music to create dance. Therefore, we propose an end-to-end dance generation model that generates appropriate dance from music without any understanding of dance or music.
Figure 1. A schematic diagram of the music-driven dance generation system.
•
An end-to-end model that takes music as an input and generates dance that matches the music should be able to identify and learn the relationship between data from two completely different areas: music and dance. That is, it should be possible to learn the relationship between a musical feature sequence according to time and a dance motion sequence according to time. Therefore, our study is based on the seq2seq model that can perform multi-modal sequence to sequence transformation.
•
In particular, music and dance have an autoregressive characteristic in which a musical features or dance features at a certain point in time is determined by the musical features or dance features of a previous point in time. Therefore, using the autoregressive seq2seq model, it is possible to effectively learn the relationship between the temporal features of music and dance.
•
To predict data at a specific point in time based on past time series data, causal dilated convolution, which is widely used in speech synthesis, is used [1]. From this, it is possible to predict the motion at a specific point in time based on past motion features.
•
In addition, in order to generate dance motions suitable for the music given as input, the local conditioning methodology, which is mainly used in fields such as TTS, is applied to the network. Based on this, it is possible to complete the seq2seq model that creates a dance sequence that matches the sequence of music.
•
To extract time-series dance motion data from dance images, pose detection algorithm such as openpose is used [2]. In this way, pose information of a motion can be extracted in 2D or 3D from the image. The extracted time series pose information is used as dance motion data.
Dance Generation
Creative AI
1) Visually Grounded Speech Representation Learning
•
Objective: Self-supervised speech representation learning without text labels through audio-image pair data
•
Dataset: SpokenCOCO, PlacesAudio
•
Baseline model: ResDAVEnet-VQ
•
Evaluation: Audio-Image retrieval(Recall @ k), zero-speech challenge metrics
•
Topics
2) Speech Generation without Reconstruction Loss
•
Objective: Modeling speech development of human using physical vocal tract model
•
Dataset: LibriSpeech, LJ Speech, Google Speech Commands
•
Physical vocal tract model: VocalTractLab, gnuspeech
•
Evaluation: WER/PER using pretrained ASR model
•
Topics
Modeling Auditory Perception and Speech Development
Speech Processing

Emotional Text-to-Speech (TTS)

Motivation

A typical speech synthesis model requires a vocoder to listen to what it sounds like through mel-spectrogram after creating mel-spectrogram in text. In recent years, the trend has shifted to studies that extract waveform from text end-to-end, rather than using pre-trained vocoder as an inference, to process speech synthesis potentially with more variance information of speech waveform and to inference directly rather than going through two steps of inference. Meanwhile, speech synthesis research that generates a speech waveform containing emotion information is in progress, and more robust performance can be demonstrated by reflecting emotion information in training. In this work, we propose a model that directly generates speech waveform containing emotion information more liveliness with our proposed technique, Speech-SECat, using conditional adversarial training.

Our Contribution

Non-Autoregressive TTS Model
End-to-end generation of waveform samples
Emotion-conditioned text-to-speech model

Method

•
A scheme of the generator architecture, which consists of the FFT Encoder, variance adaptor, Post-Net, and waveform decoder.
•
A scheme of the proposed Speech-SECat architecture. The architecture consists of three processes, squeeze, excitation, and concatenation.
Controllable Text-to-Speech
Speech Processing

1. What is singing voice synthesis?

Singing voice synthesis (SVS) is the task of generating a natural singing voice from a given musical score. With the development of various deep generative models, research on synthesizing high-quality singing voices has been emerging recently. As the performance of the SVS improves, there are increasing cases in which the technology is applied to the production of actual music content.

2. Challenges

There are various challenges to designing a singing synthesis system that can freely generate high-quality, natural-sounding singing voices.
•
First of all, the problem is that it is difficult to construct a dataset. In general, since singing is difficult to be released due to issues such as copyright, we have limitations in collecting public singing datasets. It is also difficult to access clean singing voices, as many are released with accompaniment. Lastly, singing synthesis requires not only a clean singing voice but also appropriate sheet music information corresponding to it, and the process of annotating it is time-consuming and costly.
•
In order to respond to this problem, 1) research on effectively modeling singing using only small data (LiteSing, Sinsy), 2) research on securing data sets using technologies such as sound source separation and automatic notation from various sound sources existing on the online web (DeepSinger), 3) Research on creating and presenting singing datasets free from copyright issues such as nursery rhymes is being conducted (CSD).
•
In our lab, we collect 200 songs and conduct research using them. First, we purchase an accompaniment MIDI file for K-POP music from a MIDI accompaniment producer, then hire an amateur singer to sing and record the song to the accompaniment. Later, by manually correcting minute differences in timing and pitch between the actual singing and the melody MIDI, the song and score pair data are obtained. Using this data, we are working on singing voice synthesis modeling while at the same time trying to obtain more sophisticated annotations through transcription and alignment studies.
Dataset example (audio, midi stereo)

2.2 Sound quality

•
With advances in speech synthesis studies, generating results of adequate quality in speech synthesis has advanced a lot. However, for singing synthesis technology to be used in real industry, studio quality results are required. Therefore, we are exploring different methods aimed at generating a 44 kHz sound source.
•
Unlike speech, singing voice 1) has a wide pitch range, 2) contains many notes with long duration, and 3) it is necessary to model a high sampling rate. We are trying to solve this problem based on the latest research on HiFiGAN, NSF, Parallel WaveGAN, etc. Applying the above studies focused on speech synthesis as it is causes several problems in high-quality singing modeling. (high, low pitch artifacts, glitch issue, etc.) Therefore, we are trying to develop a vocoder for singing voice that combines various GAN-based vocoders for high quality while taking the pitch robustness using the source excitation signal.
Singing Voice Synthesis
Speech Processing
Creative AI

Robust interpolation by leveraging hyper-conditioning techniques

Global HRTF Interpolation via Learned Affine Transformation of Hyper-conditioned Features (under review)

•
Authors
Abstract
•
Background
•
Proposed method
•
Ablation Study
Results
•
Coclusion

Spatial Interpolation and Abnormal Detection of HRIRs using Neural Networks

•
Background
•
Proposed method
•
Contribution
Transformation of Head-Related Transfer Functions (HRTF)
Signal Processing
Music Production is the process of managing and overseeing the recording and production of music tracks. Intelligent Music Production (IMP) aims to introduce artificial intelligence into music production, allowing for automated processes while also reflecting the user's preferences. We believe that future studies on this topic will help artists and audio engineers simplify the complex process of traditional music production and provide them with a creative workflow.
2.
new digital audio effects (DAFX) that previous approaches have not been capable of, our publication: Reverb Conversion of Mixed Vocal Tracks Using an End-to-end Convolutional Deep Neural Network.
3.
and improving the availability and accessibility of existing complex DAFX modules. our publication: Differentiable Artificial Reverberation.
Intelligent Music Production
Signal Processing
Music Production

1. Introduction

Music auto-tagging is a multi-label binary classification task that automatically predicts tags corresponding to audio content
Figure 1. Music auto-tagging
•
Note that the more specific task 'music mood tagging' has relatively high-level tags such as (groovy, dream, or sexy).

2. On-going research

2-1. Reasoning on music auto-tagging / mood-tagging

Content

2-2. Adversarial domain adaptation methods (on mood-tagging)

Content

2-3. Improving auto-tagging via cross-domain knowledge transfer

Content
Music auto-tagging
Musical Information Retrieval (MIR)
Search
Research Archive
Pop2Piano is the task of generating a piano arrangement from a given pop music audio.

1. Introduction

Arranging pop music into piano music is something that only music experts can do. This arrangement of music is used for hobbies, education, and business purposes. The MIR field has developed a lot using deep learning technology. Deep learning can analyze the genre or mood of pop music (Music Tagging, Music Mood Estimation) [1] and automatically generate pop MIDI (MIDI Generation) [2]. In addition, there has been a lot of progress in estimating piano MIDI from raw audio(Music Transcription). However, as far as we know, there was no work of arranging pop music directly into piano performance using deep learning. We call this work Pop2Piano.
Pop2Piano is the work of generating piano arrangements conditioned on pop music. Through the automated data collection/purification process, we created a dataset with thousands of pop music and piano arrangement MIDI pairs.
We are conducting research to automatically arrange pop music on the piano by learning patterns of these data using deep learning. Below is a sample of the data. (It is optimized for stereo audio. One side is the piano and the other side is the original. )
(KPOP Male Group) BTS - Butter
(KPOP Female Solo) Rosé - On the Ground
Pop2Piano
Transcription
Music Generation

What is the music performance evaluation?

•
Music performance evaluation is the feedback of judges on the parameters (i.e., tempo, timing, dynamics, intonation, and sound quality) that the performer is tasked with interpreting the musical score, and translates into an acoustic representation.
•
Evaluation of musical performance is a highly subjective task that relies on experts to evaluate aspects of a performer's technical, musical expression, and overall perception.

Piano performance evaluation

The list of 15 elements of piano performance evaluation

•
Fingering, phrasing, dynamics, timing, pedaling, interpretation, emotional expression, rubato, melodic accuracy, rhythm accuracy, articulation, tone quality, musical structure, style, and overall flow

Difference Between Hands (DBH) from the Piano Performance

•
Utilizing both hands fluently is considered to be fundamental for a pianist. Due to the nature of the piano keyboard, the balance of high and low tones reflects how well both hands are controlled.
•
A highly skilled pianist performance the accentuated notes not only louder but also approximately 20~30ms before the other notes.

Automatic Piano Performance evaluation

•
The process compares the performance with the original musical score to determine and quantify how faithfully it was played.
Automatic Piano Performance Evaluation
Performance
Symbolic music generation includes various tasks for understanding human-composed or -performed music and mimicking the process for producing them. It particularly aims to computationally model real-world music using symbolic representations, such as Musical Instrument Digital Interface (MIDI). We are interested in several tasks that have been achieved increasing attention in the MIR field.

Automatic Melody Harmonization

1. What is automatic melody harmonization?

Automatic melody harmonization is a task to find coherent chords or an arrangement for the given melody. It has been important for understanding human composition by imitating the harmonizing process of the composers. It can also reduce a barrier to creating music without expertise.
In our lab, we mainly have focused on finding a chord label sequence for the given symbolic melody notes.

2. On-going Research

1) Structured and Flexible Harmonization of Melody with Transformer

Contents

Automatic Performance Generation

1. What is automatic performance generation?

Symbolic Music Generation
Music Generation
Singing voice separation is one of the most widely studied area in the field of audio signal processing. Recent years, the performance of singing voice separation have been greatly improved with fast development of deep learning. As it seems that the separation quality is almost saturated to be improved using audio contents only, we are digging the way of using highly informative additional information for singing voice, lyrics, to get further improved model. So far, we researched the effectiveness of phonetic information of aligned lyrics for singing voice separation. With recently presented MUSDB18 lyrics dataset, we are currently working on unaligned lyrics-informed singing voice separation network.

Puplications

Lyrics-informed Singing Voice Separation

Research Overview

Automatic music transcription (AMT) aims to extract a musical notation in a form of symbolic data from a recording of a music signal automatically. It encompasses a wide range of tasks in the music signal processing, including note onset detection, pitch estimation, and multi-instrument separation. For a full transcription, by means of the complete conversion from an audio signal to a “piano-roll” representation, two essential data of “onset/offset” and “pitch” should be obtained from both temporal and spectral analysis. In addition, our recent studies have been extended to find new features that imply musical expressions, such as a singing voice and varied playing styles of musical instruments.

Publications

•
H. Heo, D. Sung, K. Lee, “Note Onset Detection based on Harmonic Cepstrum Regularity,” in Proc. IEEE Int. Conf. Multimedia and Expo (ICME), 2013.

Datasets

Project Members

Hoon Heo, Sungkyun Chang, Dooyong Sung, Yoonchang Han
Automatic Music Transcription
Transcription

Research Overview

With an extraction of some music information from an arbitrary melody input, we proposed an algorithm which generates a music accompaniment for the melody input automatically. The system calculates a optimal tempo and metrics to a free melody input (such as a humming) and then assigns a chord sequence. Accompaniment is synthesized from an assembled MIDI templates for 5 different genres. Voice inversion and rhythm suffling are implemented for diversity of musical expressions.

Project Members

Hoon Heo, Dooyong Sung, Jaehoon Kim, Gwangseok An, Sanghee Won

Funding Agency

Samsung Electronics Co., Ltd.
Automatic Music Accompaniment
Music Generation

Research Overview

The purpose of this research is to develop a hybrid music recommendation system that is customized to suit Korea’s music market and its music listeners. The technological advances in today’s hardware and the rapid development of the internet has lead to millions of songs being available online. However, paradoxically, there is less music for users now that more is available because of information overload. To solve this problem, music recommender systems filter out only the items that are relevant and of value to the user.
Figure 1. Concept of the recommender system
The basic concept of the system proposed in this research is to use experts when providing recommendations to ‘novices’. A broad definition of an expert would be a person who is knowledgeable in a certain area, while a novice is one who lacks such knowledge. Thus, it is only natural that knowledge is transferred from experts to novices. In this regard, we use users who are considered experts in their domain to provide recommendations to novices in those domains.
In this recommender system, each user has a domain in which he/she is a novice and an expert. An expert is defined as a user whose item consumption is skewed, or focused, on a certain set of similar items. Likewise, a user is a novice in areas where the consumption rate is low. Taking movies as an example, it is only reasonable that a person who enjoys watching sci-fi movies can provide helpful recommendations to a user who usually watches drama but occasionally finds some sci-fi movies engaging.
Thus, in order to find experts, the items are placed in an N-dimensional space, so that similar items are placed together and dissimilar items are apart. Similar items are then clustered, which define the areas that a user can be an expert or novice in. Next, each user is analyzed to see the distribution among the clusters, or areas, that the consumed items are in and are labeled as experts for specific clusters, accordingly, as shown in Figure 1. When providing recommendations for a novice, the experts of the cluster that the user is a novice in are used to generate novel and relevant recommendations.

Publications

•
Z. Hyung, K. Lee, K. Lee, Music Recommendation Using Text Analysis on Song Requests to Radio Stations, Expert Systems with Applications, Volume 41, Issue 5, April 2014, Pages 2608-2618, ISSN 0957-4174.
•
K. Lee and K. Lee. Using Dynamically Promoted Experts for Music Recommendation. Multimedia, IEEE Transactions on, 16(5):1-10, 2014.
•
K. Lee and K. Lee. Using Experts Among Users for Novel Movie Recommendations. JCSE, 7(1):21-29, 2013.
Music Recommender Systems
Recommendation

Research Overview

In the last decade, there has been considerable interest in digital music services that display the lyrics of songs that are synchronized with their audio. As a recent example, Sound Hound provided a live-lyrics feature using an audio fingerprint technique, with known time stamps for alignment. Although some well-known hit songs already have lyric time stamps within existing karaoke databases, more than a hundred million numbers, including new, unpopular songs YouTube cover songs, and live recordings may not. An automatic lyrics-to-audio alignment system could reduce the huge amount of time and labor required to manually construct such time stamps. Not limited to the application above, it could also be used as a front end for various purposes in content-based music information retrieval (MIR) involving lyrical rhymes and emotions.

Publication

Music Lyric Alignment

Research Overview

The purpose of this research is to personalize HRTF(Head Related Transfer Function) without directly measuring the HRTF of an individual person. HRTF is used to render 3D sound using headphones. The process of directly measuring the HRTF of a person takes more than an hour inside an anechoic chamber. This makes it almost impossible to commercialize personalized HRTF headphones. This research uses individuals body characteristics and data collected from a 3D sound hearing test to estimate personalized HRTF.
HRTF personalization using generative neural networks
HRTF

Research Overview

(According to Wikipedia,) Source separation problems in digital signal processing are those in which several signals have been mixed together into a combined signal and the objective is to recover the original component signals from the combined signal. In case of music –which is combined with many musical source- we need to separate it for the better understating of it.
There are many applications which need music source separation. For example, in the music information retrieval (MIR) scene, if we want to estimate the beat of the music, we first need to obtain the percussive sound of it, while we need to remove it if we want to estimate the chord. We need to take the vocal to estimate the lyrics, and need to separate all the instruments for the music transcription.
Music source separation is not only useful for MIR, but also by itself. Let say you want to train a singing. Vocal separation can help you by providing the vocal-removed signal from the original music file. In addition, if we can set each separated source on the various virtual location, it is possible to make the upmixed version of mono->stereo or stereo->5.1ch from it.
As many other tasks, a research for the music source separation is started from specifying the problem. What do we want to separate? How many instruments are there? Is it mono, stereo, or more? Do we have any pre-trained database? Is there any other side information? Depending on the task or the application, there are numerous problems we want to solve.

Publications

Journal Articles
•
I.-Y. Jeong, and K. Lee, “Vocal separation from monaural music using temporal/spectral continuity and sparsity constraints,” IEEE Signal Processing Letters, Vol. 21, No.10, pp. 1197-1200, Jun. 2014.
Conference Papers
•
J. Park, J. Shin, and K. Lee, “Exploiting continuity/discontinuity of Basis Vectors in spectrogram decomposition for harmonic-percussive sound separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, , Vol. 25, No. 5, pp. 1061-1074, May, 2017.
•
I.-Y. Jeong, and K. Lee, “Informed source separation from monaural music with limited binary time-frequency annotation,” submitted to IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia, 2015.
•
J. Park, and K. Lee, “Harmonic-percussive source separation using harmonicity and sparsity constraints,” in Proc. International Society for Music Information Retrieval Conference (ISMIR), Malaga, Spain, 2015.
Music Source Separation
Source Separation

Research Overview

Many users who use music streaming services want to see lyrics together when listening to music. In order to meet the needs of these users, it is important to synchronize the music with the corresponding lyrics. However, since there is no adequate performance alignment algorithm, many music streaming companies still align the lyrics information while listening music to the direct. In this research, it is aimed to develop an algorithm that automatically aligns music and lyrics when a sound information and lyric information are given. This can significantly reduce the cost of synchronizing and meet the needs of more users experience.
Figure 1. System overview
Development of Automatic Lyrics Synchronization Algorithm Using Deep Neural Network

Research Overview

This project is associated with three laboratories from KAIST and Seoul National University, and Samsung Cooperation.
This project challenges the question whether a machine can perform music in an emotionally expressive manner with a specific style, and mainly aims to develop a novel human-competitive system that generates music performance conveying an emotion and a style. The proposed system is composed of multiple neural network modules to acquire, analyze and synthesize music performance data.
Figure 1. A diagram of overall system
In order to successfully build the overall system, we divide the main goal into several subtasks; in MARG, we focus on defining useful features on analyzing emotion from acquired performance data as well as building a system that quantitatively evaluates music performance using machine learning techniques.
The outcomes from the project would provide a number of useful insights on how to represent the elements of music performance in a manner that computer can understand, and modeling the artistic process with various types of neural networks.
Synthesis of Emotionally Expressive Music Performance Using Machine Learning
Performance

Research Overview

One of the main goals of this project is to conduct a fundamental research on mathematical modeling regarding encoding and generation of sequential data or time series data in auto-regressive manner. The model would be based on generative adversarial networks(GAN) which reflects a complementary structure composed of a generator, which generates data, and a discriminator, which feeds back to the generator with evaluated cost. This model can be broadly applied to generation, recognition, and quality correction of medium such as text, audio, or video.
Figure 1. Concept of the singing synthesis model
Other major goal is to build an applied system that automatically synthesize singing human voices. The system will contain a singer model represented as a generator and a professional listener model represented as a discriminator. The project aims to generate natural singing voices using state-of-the-art sample-unit voice synthesizing method based on recent deep neural network techniques. The main functions of the system would not only be synthesizing singing voice with various timbres, but also quantitatively assessing singing proficiency and transcribing music.
Auto-regressive Generative Adversarial Network for Singing Synthesis and Evaluation

Research Overview

Human perception is a higher level process than sensation. We can get information from a variety of sensory organs, including sight, tactile, and hearing, but their perception is ultimately integrated through people’s thinking processes.
However, most artificial intelligence systems are now focusing on only one sensory information. Extract and generate information from a single modality such as image, audio, and text information. Our goal is to find associations between these multiple modalities and to convey mutual information.
Figure 1. Concept of the system
Our first approach is to create single modal information with constraints on complex information, just as it is to make another music based on music. We plan to expand into multi modality studies such as generating images from sound or creating images from sound.

Publication

•
Lee, Juheon, Seohyun Kim, and Kyogu Lee. “Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network.” arXiv preprint arXiv:1811.00818(2018).
Hear what you imagine: cross-modal audio synthesis/transformation using generative adversarial network
Multi-modal
Music Visualization for Visually-impaired Persons
Multi-modal

Research Overview

Tinnitus, also known as the phantom auditory perception; perceiving sound in the absence of an external acoustic stimulus, becomes one of popular sense organ disorders. According to the report from U.S. and Western Europe, there are at least 70 million chronic tinnitus patients in EU. Moreover, 10~15% of adults suffer chronic tinnitus and 1% of them suffer severe enough to have a significant adverse effect on their lifestyle and quality of life.
Hearing loss is known as the main cause of tinnitus. In this sense, a majority of tinnitus patients are the old since hearing loss gets worse as the people get older. Therefore, tinnitus treatment is traditionally aimed to recuperate the patient’s hearing ability. Therapies using hearing aid are adopted to handle the hearing loss based on neuroscientific backgrounds.
Hearing loss is known as the main cause of tinnitus. In this sense, a majority of tinnitus patients are the old since hearing loss gets worse as the people get older. Therefore, tinnitus treatment is traditionally aimed to recuperate the patient’s hearing ability. Therapies using hearing aid are adopted to handle the hearing loss based on neuroscientific backgrounds.
This project is aimed to develop a novel tinnitus treatment method. This method considers a preference of each patient as well as recovering the hearing function for both better treatment effect and therapy compliance. To this end, below techniques are adopted:
•
Functional imaging
•
User preference based music recommendation
•
Audio signal processing
•
Computational modeling between musical features and stimulated brain regions
Using techniques listed above, novel method can provide patient-tailored treatment and maximize remedial value using music with personal preference.

Project Members

•
Kyogu Lee, Kyungmyun Lee (Professor @ SNU)
Tinnitus Therapy based on Music and Brainwave Analysis
Neuroscience
•
Motive
•
Work in progress
Reference
[6] Cantisani, G., Trégoat, G., Essid, S., & Richard, G. (2019). MAD-EEG: an EEG dataset for decoding auditory attention to a target instrument in polyphonic music. Speech, Music and Mind (SMM), Satellite Workshop of Interspeech 2019.
[7] A. Ofner and S. Stober, “Shared Generative Representation of Auditory Concepts and EEG to Reconstruct Perceived and Imagined Music,” 19th International Society for Music Information Retrieval Conference – ISMIR 2018, pp. 392–399, 2018.
[8] Lawhern, V. J. et al. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of Neural Engineering 15, 056013 (2018).

If you have any questions, please contact the first author.

Reconstruction of Musical Features from EEG recordings during music listening
Neuroscience
Authors: Ju-Sung Ham(stephen93@snu.ac.kr), You Jeong Hong, Jeong Mi Park, Jonghwa Park, Juhan Nam, and Kyogu Lee
Keywords: Emotion communication; Expressive performance; Classical Piano Pieces; Musical training;

Backgrounds

Figure 1. Musical factors that affects the emotional expression. From Livingstone et al. (2010)
•
Communication of emotion through music is affected by the factors related to composer, performer and listener.
•
For example, Quinto & Thomson (2013) revealed that listeners’ ratings on the pleasure dimension were more changeable by the composer related factors (e.g., mode, melody), while ratings on the arousal dimension were more changeable by the performer related factors (e.g., tempo, dynamics).
•
Also, Akkermans et al. (2019) found that among several individual factors of listeners, only the level of musical training (Gold-MSI) significantly impacts listeners’ probability of correctly identifying the intended emotion.
•
However, these findings are mainly derived from monophonic melodies, which the majority of prior works have employed for experiments.

Aims

•
Therefore, our study aims to further explore the impact of factors that are related to composer, performer, and listener on emotion communication through polyphonic classical piano pieces, which are frequently consumed in our daily lives.
•
To investigate the accuracy of communication in a more quantitative way, we adopted the three-dimensional model of emotions (Mehrabian, 1996), which consists of pleasure, arousal, and dominance (PAD).
Figure 2. PAD emotional space.P for pleasure, A for arousal, and D for dominance. From Gonzalez-Sanchez et al. (2017).

Recording and Listening Experiment

Communication of Emotion Modified by the Performer in Classical Piano Pieces
Music Perception

Research Overview

The purpose of this research is to develop a real-time sound analysis technology to support unmanned surveillance systems. Industrial needs for unmanned surveillance systems are growing due to the increasing cost of labor. Today’s computer vision technology combined with deep learning algorithms is rapidly advancing to meet the goal of unmanned visual surveillance. On the other hand, auditory information is also a critical part of surveillance. Auditory information can detect abnormal events happening outside the visual angle and abnormal events which are hard to detect using visual information such as people screaming. In this research, auditory features and deep learning technology are used to achieve our objectives.
The research has three key objectives.
1. Detection and classification of abnormal sound event
2. Sound event direction of arrival(DOA) estimation
3. Robot motor noise cancellation

Publications

•
Wansoo Kim and Kyogu Lee, “Sound Event Localization and Detection Using Modular Neural Network Structure”, 대한전자공학회 추계 학술대회, 대한전자공학회, 2018.
Real-time sound analysis technology development
Sound Detection

Research Overview

A variety of household appliances we use in our lives generate noise. If there is an abnormal noise in the appliance, we can guess whether the product is malfunctioning. The purpose of this research is to develop an algorithm to detect the abnormal noise generated in household appliances and to diagnose the fault by using it. If the failure can be diagnosed with only noise without additional inspection, the repair process can be more effectively carried out.
Figure 1. Concept of the system
Development of Abnormal Sound Detection Algorithm for Home Appliances using Deep Neural Network
Sound Detection

Research Overview

The purpose of this project is to develop an interactive/smart music education system especially when learning instrument. For the purpose of this project, we proposed a real-time musical performance feedback system which is combination of common MIR tasks such as onset detection, Pitch detection and Score following.
This system detects pitch and onset from live audio stream and make note sequence with previous detected note events. The system aligns two note sequence which is from live audio stream and score to calculate score position, wrong note. Figure 1. shows the overview of the system. An example of rendered music note is shown in Figure 2. The music notation wass rendered using VexFlow with HTML5. The information from the backend, such as the incorrect notes and current position, are sent to VexFlow to be rendered accordingly. This HTML page is displayed on the iOS through the UIWebView class, which is used to embed web content in applications. Any additional buttons used for user interactions are coded natively through Xcode.
Figure 1.
Figure 2.

Publications

•
S. Kwon, B. Han, Y. Han and K. Lee, “A Musical Performance Feedback System Based on Real-time Score-Following”, HCI Korea 2013, 2013.
•
Y. Han, S. Kwon, K. Lee and K. Lee, “A Real-time Musical Performance Feedback System for Beginner Musician”, NIME 2013, 2013.

Project Members

Kibeom Lee, Yoonchang Han, Sungkyun Jang, Sejun Kwon

Funding Agency

Smart Interactive Music Edutainment

Research Overview

In recent years, automatic and objective classification of mood disorders, such as depression using various signals (speech, face, brain wave) has got popularity.Clinical depression is a severe mental health disorder and it has a wide range of potential symptoms including like strong negative conceptualization and observable psychomotor retardation.
Current diagnosis tool for depression depends almost exclusively on self-report and clinical opinion by psychiatrist. But it has critical weakness that monitoring patient’s status is very difficult. Most of people suffer from depression, which also has no strong will to revisit hospital for medical care. So developing automatic monitoring/classification system for patients with depression can have a major impact on society.
Speech has strong cues for classifying patient’s status. Speech in patients with depression has reflection that often described as being stagnant with diminished prosody. Paralinguistic analysis of depressed speech has shown that physiological symptoms relating to depression affects vocal tract properties.
The aim of this project is to investigate the utility of affective sensing methods for automated depression analysis, which can assist psychiatrist in depression diagnosis and monitoring. Remote depression diagnosis/monitoring system can cover a large number of depressive patients.

Project Members

Subin Lee, Ziwon Hyung, Jongkyu Shin, Joengsoo Park

Funding Agency

Seoul National University, Seoul National University Bundang Hospital
Development of Voice-based Diagnostic Marker for Depression

Research Overview

The purpose of this research is to collect respiratory sound including snoring among biological sounds generated during sleep, to be used for the development of a personalized sleep management device for the management of routine sleep in everyday life.
The research will include an analysis of sound information generated during sleep, selection of acoustic biomarker, development of a medical algorithm of a new concept based on the acoustic marker which can measure the non-invasive sleep status, development of a smart sleep mask through fusion research, and evaluation of the accuracy and efficacy of these methods using the results of sleep polyvalence test and sleep apnea test as well as a clinical trial and pilot application.
In addition, the goal of this research includes development of a smart sleep mask to a higher level, to be used for the development of various sleep-related health management scenarios using this smart sleep mask and trial application.

Publications

•
Jaepil Kim, Taehoon Kim, Jeong-Whun Kim, and Kyogu Lee, “An Investigation of the Snoring Sound Classification Using the Cyclostationary Attributes”, in Proc. of International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, 2016
•
Jaepil Kim, Taehoon Kim, Jeong-Whun Kim, and Kyogu Lee, “CYCLOSTATIONARITY-BASED ANALYSIS OF NOCTURNAL RESPIRATION SOUNDS”, in Proc. of International Congress on Sound and Vibration (ICSV), Athens, Greece, 2016
AcoustoSleepMask: Development of smart sleep mask using acoustic biomarker and machine learning

Research Overview

This project is supported by Urban Data Science Laboratory under the Seoul Metropolitan Government and uses big data to solve various problems in the city. In particular, the Platform part creates a platform for collecting and analyzing multiple pieces of data in the city and solves problems based on data collected by this platform.
Currently, research on real-time noise value collection and analysis in Seoul is underway, and research on real-time detection of abnormal noises that may occur in traffic and crime prevention environments in cities such as screams, glass breaking sounds, and horn sounds is underway.
Figure 1. platform device; audio analysis module

Publications

Urban Data Science Laboratory Project for Future City with Data Foundation
Research Overview
The purpose of this research is to monitor the condition of a baby using the sounds that the baby makes. Childcare experts with sufficient experience claim that they can predict the condition of a baby such as the reason for crying with the sound that the baby is making. The goal of this research is to transfer the knowledge of childcare experts to a deep learning model. This deep learning model will benefit new parents struggling to figure out the reason why their baby is crying. The first step of this research is to collect sufficient baby sound database with appropriate tags. The collected database will be used to train out deep learning model.
Baby sound database collection and baby condition monitoring using machine learning

Obstructive Sleep Apnea Screening Test Techniques Development using Analysis and Classification of Snoring Sound

Research Overview

Obstructive Sleep Apnea (OSA) is the most common type of sleep disorder and is caused by obstruction of the upper airway during sleep. This symptom arouses pauses in breathing which lead to deoxygenation and consequent arousals.
Figure 1. Obstructive sleep apnea (from wikipedia)
The effects of sleep disorders are extensive, impacting sufferers physically, psychologically and financially. Up to 40% of the US adults experience problems with falling asleep or daytime sleepiness. (Hossain and Shapiro 2002)
Figure 2. Effects of sleep disorders (from Ronald Kessler, Harvard Medical School 2006)
Moreover, diagnosis rate of OSA is very low on account of the diagnostic complexity and heavy expenditure although this sleep disorder has high fatality and prevalence.Recently, interests about wearable device and telemedicine have been increasing and market requires high clinical accuracy in personalized handheld devices for mobile healthcare. Especially, OSA is typically cannot be aware of the disease in the single person households. So if we can develop simple and accurate algorithm which can be an OSA screening test, this is a good solution for national health promotion and creates values for next generation mobile healthcare system.
In our study, we only use snoring sound in the polysomnography, although other studies typically used various physiological parameters (oxygen saturation, body movement, EEG, EOG, EMG etc.). We extract various audio features in the perspectives of temporal structures and raw signal processing. Or we may conduct feature learning which is a set of techniques in machine learning that learn a transformation of raw inputs to a representation that can be effectively exploited in a supervised learning task such as classification.
Through this research, we may obtain good classification or regression models to inform the patients’ OSA status.
This research is underway. So, if we get a significant good performance, we can update this page immediately.

Project Members

Taehoon Kim, Yoonchang Han, Ziwon Hyung
Development of Obstructive Sleep Apnea (OSA) Screening Algorithm through Acoustic Analysis

An Ultra-real New Media Art Using 3D Virtual Reality and Precise Real-time Position Tracking System

Research Overview

In this project, a.k.a Blind Perspective, we wanted to integrate two different perspectives in one same space. The two different perspectives are assigned as the real vision, and the vision seen through virtual reality. The idea is to provide a highly immersive virtual reality experience by physically matching the virtual world space dimensions with those in the real world.
The idea became possible by integrating head-mounted displays (HMDs) with accurate indoor-position tracking technology. As the proliferation of stereoscopic 3D virtual reality devices is reshaping the user experience in immersive virtual reality methods, we believed that these low-cost HMD devices are the perfect 3D vision interpreter among other kinds. The system is designed to have six degrees of freedom in the virtual world, which allows users to physically move around in the real world while wearing the wireless system with an HMD. To eliminate the potential safety risk factors such as a user being bumped into an obstacle caused by the mismatched reciprocal positions between the real and the VR or the cybersickness which mainly causes dizziness and nausea due to the sensory gap between the motor and the visual cortex, our main task was to establish a high standard in terms of positioning accuracy and steadiness.
We have utilized the ultrasound as the underlying technology for our position tracking algorithm as it enables the finest accuracy and the fast computational speed. One omni-directional ultrasound transducer mounted on the HMD generates a burst tone at 20 Hz at carrier frequency of 40 kHz. The burst signal is transmitted wirelessly from the signal generator to HMD via 5.8 GHz RF range transmission. The receivers mounted on the ceiling of the room calculate the position by triangulating the signal. A total of 16 signal receivers were used to cover the corridor space of 35 meters in Contemporary Art Gallery in Art Tower Mito.
After refining our position tracking algorithm, the test results show that the system has the accuracy of less than 10mm on average. Apart from developing algorithms, we have also implemented dedicated devices which are necessary for the integration of the HMD and the positioning system. Additionally, our system was designed to be wearable, requiring no external power or signal cables. The system is consisted of two main parts, the user position tracking part with the data acquisition equipment, and the HMD integration part with the unity 3d engine. The diagram of all the components is shown in Figure 1.
Figure 1. The System components diagram
As the purpose of our system is to provide immersive virtual experience with natural interaction/control, we are proud to be a part of the exhibition, yet hope to improve the system by covering a larger space with higher accuracy and immersion.

Publications

Blind Perspective

Research Overview

With the increasing number and popularity of new motion-sensing hardware, and touch screen mobile devices improving the usability of the new interfaces is becoming more important. However, people do not accept the new interfaces as comfortably as traditional ones because they lack in tactile or haptic feedback that traditional input methods such as mice or keyboards provide. However, having tactile feedback in these interfaces is not possible unless users are wearing a device with actuators. Therefore, auditory feedback is an appropriate and unique alternative for assisting visual feedback.
At MARG, we explored various types of novel auditory feedback methods and explore their effects as secondary feedback on complementing visual feedback. We performed a user study for a menu selection task and experimental results show that the proposed auditory feedback is significantly more efficient and more effective than visual-only feedback or conventional auditory feedback in terms of time and accuracy.
Conventional Auditory Feedback on Menu Selection in Gesture Interface
In most gesture interfaces, auditory feedback is used to indicate the transitions between menu items.Transitional auditory feedback is usually a short and percussive clicking sound which triggers on the boundary or center of a menu item. This auditory feedback has shown to be helpful for separating items on a circular menu.

Auditory Feedback for Browsing Assistance

•
Pre-emptive Auditory Feedback
Weaknesses of the transitional auditory feedback lies on the fact that it provides limited information about the position of the gesture in the circular menu. In gesture interfaces, providing single auditory feedback at the transition between menu items is not sufficient to alert the user about the change of the states. Pre-emptive auditory feedback provides additional auditory feedback just before the transitional auditory feedback which triggers sound at the boundary between menu items. It also informs the user that the transition between menu items is imminent. The pre-emptive auditory feedback has additional auditory feedback ten degrees before the triggering point of the transitional auditory feedback. This additional feedback, supplementing conventional transitional auditory feedback, will aid users in noticing the completion of browsing to the desired menu item.
•
Continuous Auditory Feedback
New Interfaces with Auditory Feedback