Search

On-going Research

Search

Topic 1. Text Query based Audio Retrieval

•
Retrieving audio signals using their sound content textual descriptions (i.e., audio captions).
•
Text query composed of manually written audio captions.
•
For each text query, the goal of this task is to retrieve audio files from a given dataset and sort them based their match with the query.

Topic 2. Automated Audio Captioning

•
The task of general audio content description using free text.
•
An inter-modal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal.
•
Modeling concepts (e.g. "muffled sound"), physical properties of objects and environment (e.g. "the sound of a big car", "people talking in a small and empty room"), and high level knowledge ("a clock rings three times").

Research 1. Audio-Text Data Augmentation

Research 2. Audio Captioning

Research 3. Audio-Text Retrieval

Audio-Language Multi-modal Learning
Multi-modal Learning

Topic : Multi-modal sensor based speech synthesis

•
Also known as “silent speech”
•
Synthesizing vocal audio by capturing the muscle signals produced during silent speech
•
Mainly use facial electromyography (EMG) sensor
•
Demo audio

Research 1. EMG-based Speech Content Representation

Research 2. Voice Conversion

Research 3. Domain Adaptation

XVoice : Multi-modal sensor based speech synthesis
Multi-modal Learning
Speech Processing

Custom Music Composition with AI Systems

The overall pipeline of generating custom music that reflects the user’s taste.
•
The "AI Producer: Developing technology of custom music composition" project is a multi-faceted initiative aimed at creating personalized multi-track music compositions based on a user's musical taste. This is achieved by using a reference song and keyword-based queries for lyric generation, leveraging artificial intelligence to create unique musical pieces.
•
The project involves multiple modules, each designed to handle a different aspect of the music generation process. These include musical instrument retrieval, which identifies and extracts desired musical instruments from a reference audio file; digital audio effects modeling, which applies audio effects and performs automatic multi-track mixing and mastering; and keyword-based lyrics generation, which creates song lyrics based on keywords provided by the user.
•
While each module operates independently, they all work in synergy to produce a complete multi-track music piece. Each of these modules relies on the reference music recording provided by the user, serving to customize the final output according to the user's taste and preferences. This combination of modules, each with its unique function, makes the project a comprehensive solution for custom music composition.

Musical Instrument Retrieval

•
Musical instrument retrieval is the task of searching and identifying specific musical instruments within mixed music recordings of a collection of different musical instruments, aiming to assist music producers in selecting appropriate virtual instruments for their projects.
•
This process addresses the time-consuming and inefficient nature of manually listening to and comparing each instrument sample to find the desired sound.
•
Performing instrument retrieval not only helps in locating the desired musical instrument but also provides valuable information for instrument recognition, enabling the identification of instruments present in a given audio mix.
related publications from MARG on musical instrument retrieval

Digital Audio Effects Modeling

AI Producer: Developing technology of custom music composition
Signal Processing
Musical Information Retrieval (MIR)
Music Production

Music

Do Sharp and Flat Keys Evoke Opposing Sensory Perception?: Focusing on the Crossmodal Influence Inherent in the Pitch and Characteristics of Enharmonic Key
•
Ahyeon Choi(chah0623@snu.ac.kr), Woojae Cho
•
Our study aimed to investigate the effects of sharp and flat keys on the perception of brightness, warmth, and sharpness, given that these keys are associated with tension and relaxation respectively in music.
•
We found that these keys significantly influenced all three sensory perceptions among 30 musicians, implying that musical keys can affect listeners' perception, thereby providing valuable insights for composers and performers to create music that triggers specific sensory responses.
Towards a New Interface for Music Listening: YouTube Case Studies
•
Ahyeon Choi(chah0623@snu.ac.kr), Eunsik Shin, Haesun Joung
•
Despite the rise of various music streaming services, an increasing number of users are choosing YouTube for music consumption due to its unique features.
•
Our research, involving semi-structured interviews with 27 active YouTube users, reveals systematic usability features of Youtube as a music streaming service such as the provision of a rich array of music-related videos and the ability to explore diverse music genres, share unique playlists, and interact with others.
•
Furthermore, we have developed the wireframe for a new youtube UI for music listening, providing practical solutions to improve user satisfaction. This work has significant implications for YouTube and other music streaming platforms aiming to enhance their user experience.

Speech

Auditory Perception & Cognition
Perception & Cognition

1) Data simulation algorithm of disaster emergency situations

Figure 1. Data simulation.
•
We build algorithms using pyroom-acoustics library that simulate the multi-channel waveform acquired on the flying drone.
Figure 2. Design of drone for test environment
•
We designed drones with a multi-channel mic array for testing environmental data acquisition and testing our model on real-world scenarios.

2) Multi-channel multi-class classification and DoA estimation

Figure 2. Classification and DoA estimation
•
We build multi-channel classification models to detect and classify emergency rescue sounds (class: male, female, baby).
•
Also, we build multi-channel Diection-of-Arrivial (DoA) models to estimate the direction of the sound sources
•
The models should be robust to drone noise and the status of the drone (hovering or moving).
•
A low-flops regime for drone-integrated model scenarios is also one of the major parts of the project. We test our algorithm on NVIDIA Jetson devices considering lightweight hardware costs.
Multi-channel DoA estimation of integrated cognitive drone for disaster emergency situations
Signal Processing

1. Introduction

‘Dance generation’ is a task of music-based motion generation that uses a rule-base or deep learning model to make realistic dance motions. In the past, rule-based models were mainly used, but recently, with the development of deep learning techniques, methods using deep learning have become a trend. This task requires musical knowledge and an understanding of human motion, especially dance. To this end, we use a model that has learned the relationship between musical features and dance movements to generate dance motion suitable for music.

2. Research

Automatic Choreography Generation with Convolutional Encoder-decoder Network

In general, when people dance, they dance to match the music. In fact, dance is closely related to music. Many elements of dance follow the temporal and spectral characteristics of music. Therefore, it is necessary to understand both music and dance in order to create a dance that matches the music. However, it is very difficult for ordinary people who do not know much about dance or music to create dance. Therefore, we propose an end-to-end dance generation model that generates appropriate dance from music without any understanding of dance or music.
Figure 1. A schematic diagram of the music-driven dance generation system.
•
An end-to-end model that takes music as an input and generates dance that matches the music should be able to identify and learn the relationship between data from two completely different areas: music and dance. That is, it should be possible to learn the relationship between a musical feature sequence according to time and a dance motion sequence according to time. Therefore, our study is based on the seq2seq model that can perform multi-modal sequence to sequence transformation.
•
In particular, music and dance have an autoregressive characteristic in which a musical features or dance features at a certain point in time is determined by the musical features or dance features of a previous point in time. Therefore, using the autoregressive seq2seq model, it is possible to effectively learn the relationship between the temporal features of music and dance.
•
To predict data at a specific point in time based on past time series data, causal dilated convolution, which is widely used in speech synthesis, is used [1]. From this, it is possible to predict the motion at a specific point in time based on past motion features.
•
In addition, in order to generate dance motions suitable for the music given as input, the local conditioning methodology, which is mainly used in fields such as TTS, is applied to the network. Based on this, it is possible to complete the seq2seq model that creates a dance sequence that matches the sequence of music.
•
To extract time-series dance motion data from dance images, pose detection algorithm such as openpose is used [2]. In this way, pose information of a motion can be extracted in 2D or 3D from the image. The extracted time series pose information is used as dance motion data.
Dance Generation
Creative AI
1) Visually Grounded Speech Representation Learning
•
Objective: Self-supervised speech representation learning without text labels through audio-image pair data
•
Dataset: SpokenCOCO, PlacesAudio
•
Baseline model: ResDAVEnet-VQ
•
Evaluation: Audio-Image retrieval(Recall @ k), zero-speech challenge metrics
•
Topics
2) Speech Generation without Reconstruction Loss
•
Objective: Modeling speech development of human using physical vocal tract model
•
Dataset: LibriSpeech, LJ Speech, Google Speech Commands
•
Physical vocal tract model: VocalTractLab, gnuspeech
•
Evaluation: WER/PER using pretrained ASR model
•
Topics
Modeling Auditory Perception and Speech Development
Speech Processing

Emotional Text-to-Speech (TTS)

Motivation

A typical speech synthesis model requires a vocoder to listen to what it sounds like through mel-spectrogram after creating mel-spectrogram in text. In recent years, the trend has shifted to studies that extract waveform from text end-to-end, rather than using pre-trained vocoder as an inference, to process speech synthesis potentially with more variance information of speech waveform and to inference directly rather than going through two steps of inference. Meanwhile, speech synthesis research that generates a speech waveform containing emotion information is in progress, and more robust performance can be demonstrated by reflecting emotion information in training. In this work, we propose a model that directly generates speech waveform containing emotion information more liveliness with our proposed technique, Speech-SECat, using conditional adversarial training.

Our Contribution

Non-Autoregressive TTS Model
End-to-end generation of waveform samples
Emotion-conditioned text-to-speech model

Method

•
A scheme of the generator architecture, which consists of the FFT Encoder, variance adaptor, Post-Net, and waveform decoder.
•
A scheme of the proposed Speech-SECat architecture. The architecture consists of three processes, squeeze, excitation, and concatenation.
Controllable Text-to-Speech
Speech Processing

1. What is singing voice synthesis?

Singing voice synthesis (SVS) is the task of generating a natural singing voice from a given musical score. With the development of various deep generative models, research on synthesizing high-quality singing voices has been emerging recently. As the performance of the SVS improves, there are increasing cases in which the technology is applied to the production of actual music content.

2. Challenges

There are various challenges to designing a singing synthesis system that can freely generate high-quality, natural-sounding singing voices.
•
First of all, the problem is that it is difficult to construct a dataset. In general, since singing is difficult to be released due to issues such as copyright, we have limitations in collecting public singing datasets. It is also difficult to access clean singing voices, as many are released with accompaniment. Lastly, singing synthesis requires not only a clean singing voice but also appropriate sheet music information corresponding to it, and the process of annotating it is time-consuming and costly.
•
In order to respond to this problem, 1) research on effectively modeling singing using only small data (LiteSing, Sinsy), 2) research on securing data sets using technologies such as sound source separation and automatic notation from various sound sources existing on the online web (DeepSinger), 3) Research on creating and presenting singing datasets free from copyright issues such as nursery rhymes is being conducted (CSD).
•
In our lab, we collect 200 songs and conduct research using them. First, we purchase an accompaniment MIDI file for K-POP music from a MIDI accompaniment producer, then hire an amateur singer to sing and record the song to the accompaniment. Later, by manually correcting minute differences in timing and pitch between the actual singing and the melody MIDI, the song and score pair data are obtained. Using this data, we are working on singing voice synthesis modeling while at the same time trying to obtain more sophisticated annotations through transcription and alignment studies.
Dataset example (audio, midi stereo)

2.2 Sound quality

•
With advances in speech synthesis studies, generating results of adequate quality in speech synthesis has advanced a lot. However, for singing synthesis technology to be used in real industry, studio quality results are required. Therefore, we are exploring different methods aimed at generating a 44 kHz sound source.
•
Unlike speech, singing voice 1) has a wide pitch range, 2) contains many notes with long duration, and 3) it is necessary to model a high sampling rate. We are trying to solve this problem based on the latest research on HiFiGAN, NSF, Parallel WaveGAN, etc. Applying the above studies focused on speech synthesis as it is causes several problems in high-quality singing modeling. (high, low pitch artifacts, glitch issue, etc.) Therefore, we are trying to develop a vocoder for singing voice that combines various GAN-based vocoders for high quality while taking the pitch robustness using the source excitation signal.
Singing Voice Synthesis
Speech Processing
Creative AI

Robust interpolation by leveraging hyper-conditioning techniques

Global HRTF Interpolation via Learned Affine Transformation of Hyper-conditioned Features (under review)

•
Authors
Abstract
•
Background
•
Proposed method
•
Ablation Study
Results
•
Coclusion

Spatial Interpolation and Abnormal Detection of HRIRs using Neural Networks

•
Background
•
Proposed method
•
Contribution
Transformation of Head-Related Transfer Functions (HRTF)
Signal Processing
Music Production is the process of managing and overseeing the recording and production of music tracks. Intelligent Music Production (IMP) aims to introduce artificial intelligence into music production, allowing for automated processes while also reflecting the user's preferences. We believe that future studies on this topic will help artists and audio engineers simplify the complex process of traditional music production and provide them with a creative workflow.
2.
new digital audio effects (DAFX) that previous approaches have not been capable of, our publication: Reverb Conversion of Mixed Vocal Tracks Using an End-to-end Convolutional Deep Neural Network.
3.
and improving the availability and accessibility of existing complex DAFX modules. our publication: Differentiable Artificial Reverberation.
Intelligent Music Production
Signal Processing
Music Production

1. Introduction

Music auto-tagging is a multi-label binary classification task that automatically predicts tags corresponding to audio content
Figure 1. Music auto-tagging
•
Note that the more specific task 'music mood tagging' has relatively high-level tags such as (groovy, dream, or sexy).

2. On-going research

2-1. Reasoning on music auto-tagging / mood-tagging

Content

2-2. Adversarial domain adaptation methods (on mood-tagging)

Content

2-3. Improving auto-tagging via cross-domain knowledge transfer

Content
Music auto-tagging
Musical Information Retrieval (MIR)