Search

Research Archive

Search
Pop2Piano is the task of generating a piano arrangement from a given pop music audio.

1. Introduction

Arranging pop music into piano music is something that only music experts can do. This arrangement of music is used for hobbies, education, and business purposes. The MIR field has developed a lot using deep learning technology. Deep learning can analyze the genre or mood of pop music (Music Tagging, Music Mood Estimation) [1] and automatically generate pop MIDI (MIDI Generation) [2]. In addition, there has been a lot of progress in estimating piano MIDI from raw audio(Music Transcription). However, as far as we know, there was no work of arranging pop music directly into piano performance using deep learning. We call this work Pop2Piano.
Pop2Piano is the work of generating piano arrangements conditioned on pop music. Through the automated data collection/purification process, we created a dataset with thousands of pop music and piano arrangement MIDI pairs.
We are conducting research to automatically arrange pop music on the piano by learning patterns of these data using deep learning. Below is a sample of the data. (It is optimized for stereo audio. One side is the piano and the other side is the original. )
(KPOP Male Group) BTS - Butter
(KPOP Female Solo) Rosé - On the Ground
Pop2Piano
Transcription
Music Generation

What is the music performance evaluation?

•
Music performance evaluation is the feedback of judges on the parameters (i.e., tempo, timing, dynamics, intonation, and sound quality) that the performer is tasked with interpreting the musical score, and translates into an acoustic representation.
•
Evaluation of musical performance is a highly subjective task that relies on experts to evaluate aspects of a performer's technical, musical expression, and overall perception.

Piano performance evaluation

The list of 15 elements of piano performance evaluation

•
Fingering, phrasing, dynamics, timing, pedaling, interpretation, emotional expression, rubato, melodic accuracy, rhythm accuracy, articulation, tone quality, musical structure, style, and overall flow

Difference Between Hands (DBH) from the Piano Performance

•
Utilizing both hands fluently is considered to be fundamental for a pianist. Due to the nature of the piano keyboard, the balance of high and low tones reflects how well both hands are controlled.
•
A highly skilled pianist performance the accentuated notes not only louder but also approximately 20~30ms before the other notes.

Automatic Piano Performance evaluation

•
The process compares the performance with the original musical score to determine and quantify how faithfully it was played.
Automatic Piano Performance Evaluation
Performance
Symbolic music generation includes various tasks for understanding human-composed or -performed music and mimicking the process for producing them. It particularly aims to computationally model real-world music using symbolic representations, such as Musical Instrument Digital Interface (MIDI). We are interested in several tasks that have been achieved increasing attention in the MIR field.

Automatic Melody Harmonization

1. What is automatic melody harmonization?

Automatic melody harmonization is a task to find coherent chords or an arrangement for the given melody. It has been important for understanding human composition by imitating the harmonizing process of the composers. It can also reduce a barrier to creating music without expertise.
In our lab, we mainly have focused on finding a chord label sequence for the given symbolic melody notes.

2. On-going Research

1) Structured and Flexible Harmonization of Melody with Transformer

Contents

Automatic Performance Generation

1. What is automatic performance generation?

Symbolic Music Generation
Music Generation
Singing voice separation is one of the most widely studied area in the field of audio signal processing. Recent years, the performance of singing voice separation have been greatly improved with fast development of deep learning. As it seems that the separation quality is almost saturated to be improved using audio contents only, we are digging the way of using highly informative additional information for singing voice, lyrics, to get further improved model. So far, we researched the effectiveness of phonetic information of aligned lyrics for singing voice separation. With recently presented MUSDB18 lyrics dataset, we are currently working on unaligned lyrics-informed singing voice separation network.

Puplications

Lyrics-informed Singing Voice Separation

Research Overview

Automatic music transcription (AMT) aims to extract a musical notation in a form of symbolic data from a recording of a music signal automatically. It encompasses a wide range of tasks in the music signal processing, including note onset detection, pitch estimation, and multi-instrument separation. For a full transcription, by means of the complete conversion from an audio signal to a “piano-roll” representation, two essential data of “onset/offset” and “pitch” should be obtained from both temporal and spectral analysis. In addition, our recent studies have been extended to find new features that imply musical expressions, such as a singing voice and varied playing styles of musical instruments.

Publications

•
H. Heo, D. Sung, K. Lee, “Note Onset Detection based on Harmonic Cepstrum Regularity,” in Proc. IEEE Int. Conf. Multimedia and Expo (ICME), 2013.

Datasets

Project Members

Hoon Heo, Sungkyun Chang, Dooyong Sung, Yoonchang Han
Automatic Music Transcription
Transcription

Research Overview

With an extraction of some music information from an arbitrary melody input, we proposed an algorithm which generates a music accompaniment for the melody input automatically. The system calculates a optimal tempo and metrics to a free melody input (such as a humming) and then assigns a chord sequence. Accompaniment is synthesized from an assembled MIDI templates for 5 different genres. Voice inversion and rhythm suffling are implemented for diversity of musical expressions.

Project Members

Hoon Heo, Dooyong Sung, Jaehoon Kim, Gwangseok An, Sanghee Won

Funding Agency

Samsung Electronics Co., Ltd.
Automatic Music Accompaniment
Music Generation

Research Overview

The purpose of this research is to develop a hybrid music recommendation system that is customized to suit Korea’s music market and its music listeners. The technological advances in today’s hardware and the rapid development of the internet has lead to millions of songs being available online. However, paradoxically, there is less music for users now that more is available because of information overload. To solve this problem, music recommender systems filter out only the items that are relevant and of value to the user.
Figure 1. Concept of the recommender system
The basic concept of the system proposed in this research is to use experts when providing recommendations to ‘novices’. A broad definition of an expert would be a person who is knowledgeable in a certain area, while a novice is one who lacks such knowledge. Thus, it is only natural that knowledge is transferred from experts to novices. In this regard, we use users who are considered experts in their domain to provide recommendations to novices in those domains.
In this recommender system, each user has a domain in which he/she is a novice and an expert. An expert is defined as a user whose item consumption is skewed, or focused, on a certain set of similar items. Likewise, a user is a novice in areas where the consumption rate is low. Taking movies as an example, it is only reasonable that a person who enjoys watching sci-fi movies can provide helpful recommendations to a user who usually watches drama but occasionally finds some sci-fi movies engaging.
Thus, in order to find experts, the items are placed in an N-dimensional space, so that similar items are placed together and dissimilar items are apart. Similar items are then clustered, which define the areas that a user can be an expert or novice in. Next, each user is analyzed to see the distribution among the clusters, or areas, that the consumed items are in and are labeled as experts for specific clusters, accordingly, as shown in Figure 1. When providing recommendations for a novice, the experts of the cluster that the user is a novice in are used to generate novel and relevant recommendations.

Publications

•
Z. Hyung, K. Lee, K. Lee, Music Recommendation Using Text Analysis on Song Requests to Radio Stations, Expert Systems with Applications, Volume 41, Issue 5, April 2014, Pages 2608-2618, ISSN 0957-4174.
•
K. Lee and K. Lee. Using Dynamically Promoted Experts for Music Recommendation. Multimedia, IEEE Transactions on, 16(5):1-10, 2014.
•
K. Lee and K. Lee. Using Experts Among Users for Novel Movie Recommendations. JCSE, 7(1):21-29, 2013.
Music Recommender Systems
Recommendation

Research Overview

In the last decade, there has been considerable interest in digital music services that display the lyrics of songs that are synchronized with their audio. As a recent example, Sound Hound provided a live-lyrics feature using an audio fingerprint technique, with known time stamps for alignment. Although some well-known hit songs already have lyric time stamps within existing karaoke databases, more than a hundred million numbers, including new, unpopular songs YouTube cover songs, and live recordings may not. An automatic lyrics-to-audio alignment system could reduce the huge amount of time and labor required to manually construct such time stamps. Not limited to the application above, it could also be used as a front end for various purposes in content-based music information retrieval (MIR) involving lyrical rhymes and emotions.

Publication

Music Lyric Alignment

Research Overview

The purpose of this research is to personalize HRTF(Head Related Transfer Function) without directly measuring the HRTF of an individual person. HRTF is used to render 3D sound using headphones. The process of directly measuring the HRTF of a person takes more than an hour inside an anechoic chamber. This makes it almost impossible to commercialize personalized HRTF headphones. This research uses individuals body characteristics and data collected from a 3D sound hearing test to estimate personalized HRTF.
HRTF personalization using generative neural networks
HRTF

Research Overview

(According to Wikipedia,) Source separation problems in digital signal processing are those in which several signals have been mixed together into a combined signal and the objective is to recover the original component signals from the combined signal. In case of music –which is combined with many musical source- we need to separate it for the better understating of it.
There are many applications which need music source separation. For example, in the music information retrieval (MIR) scene, if we want to estimate the beat of the music, we first need to obtain the percussive sound of it, while we need to remove it if we want to estimate the chord. We need to take the vocal to estimate the lyrics, and need to separate all the instruments for the music transcription.
Music source separation is not only useful for MIR, but also by itself. Let say you want to train a singing. Vocal separation can help you by providing the vocal-removed signal from the original music file. In addition, if we can set each separated source on the various virtual location, it is possible to make the upmixed version of mono->stereo or stereo->5.1ch from it.
As many other tasks, a research for the music source separation is started from specifying the problem. What do we want to separate? How many instruments are there? Is it mono, stereo, or more? Do we have any pre-trained database? Is there any other side information? Depending on the task or the application, there are numerous problems we want to solve.

Publications

Journal Articles
•
I.-Y. Jeong, and K. Lee, “Vocal separation from monaural music using temporal/spectral continuity and sparsity constraints,” IEEE Signal Processing Letters, Vol. 21, No.10, pp. 1197-1200, Jun. 2014.
Conference Papers
•
J. Park, J. Shin, and K. Lee, “Exploiting continuity/discontinuity of Basis Vectors in spectrogram decomposition for harmonic-percussive sound separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, , Vol. 25, No. 5, pp. 1061-1074, May, 2017.
•
I.-Y. Jeong, and K. Lee, “Informed source separation from monaural music with limited binary time-frequency annotation,” submitted to IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia, 2015.
•
J. Park, and K. Lee, “Harmonic-percussive source separation using harmonicity and sparsity constraints,” in Proc. International Society for Music Information Retrieval Conference (ISMIR), Malaga, Spain, 2015.
Music Source Separation
Source Separation

Research Overview

Many users who use music streaming services want to see lyrics together when listening to music. In order to meet the needs of these users, it is important to synchronize the music with the corresponding lyrics. However, since there is no adequate performance alignment algorithm, many music streaming companies still align the lyrics information while listening music to the direct. In this research, it is aimed to develop an algorithm that automatically aligns music and lyrics when a sound information and lyric information are given. This can significantly reduce the cost of synchronizing and meet the needs of more users experience.
Figure 1. System overview
Development of Automatic Lyrics Synchronization Algorithm Using Deep Neural Network

Research Overview

This project is associated with three laboratories from KAIST and Seoul National University, and Samsung Cooperation.
This project challenges the question whether a machine can perform music in an emotionally expressive manner with a specific style, and mainly aims to develop a novel human-competitive system that generates music performance conveying an emotion and a style. The proposed system is composed of multiple neural network modules to acquire, analyze and synthesize music performance data.
Figure 1. A diagram of overall system
In order to successfully build the overall system, we divide the main goal into several subtasks; in MARG, we focus on defining useful features on analyzing emotion from acquired performance data as well as building a system that quantitatively evaluates music performance using machine learning techniques.
The outcomes from the project would provide a number of useful insights on how to represent the elements of music performance in a manner that computer can understand, and modeling the artistic process with various types of neural networks.
Synthesis of Emotionally Expressive Music Performance Using Machine Learning
Performance

Research Overview

One of the main goals of this project is to conduct a fundamental research on mathematical modeling regarding encoding and generation of sequential data or time series data in auto-regressive manner. The model would be based on generative adversarial networks(GAN) which reflects a complementary structure composed of a generator, which generates data, and a discriminator, which feeds back to the generator with evaluated cost. This model can be broadly applied to generation, recognition, and quality correction of medium such as text, audio, or video.
Figure 1. Concept of the singing synthesis model
Other major goal is to build an applied system that automatically synthesize singing human voices. The system will contain a singer model represented as a generator and a professional listener model represented as a discriminator. The project aims to generate natural singing voices using state-of-the-art sample-unit voice synthesizing method based on recent deep neural network techniques. The main functions of the system would not only be synthesizing singing voice with various timbres, but also quantitatively assessing singing proficiency and transcribing music.
Auto-regressive Generative Adversarial Network for Singing Synthesis and Evaluation

Research Overview

Human perception is a higher level process than sensation. We can get information from a variety of sensory organs, including sight, tactile, and hearing, but their perception is ultimately integrated through people’s thinking processes.
However, most artificial intelligence systems are now focusing on only one sensory information. Extract and generate information from a single modality such as image, audio, and text information. Our goal is to find associations between these multiple modalities and to convey mutual information.
Figure 1. Concept of the system
Our first approach is to create single modal information with constraints on complex information, just as it is to make another music based on music. We plan to expand into multi modality studies such as generating images from sound or creating images from sound.

Publication

•
Lee, Juheon, Seohyun Kim, and Kyogu Lee. “Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network.” arXiv preprint arXiv:1811.00818(2018).
Hear what you imagine: cross-modal audio synthesis/transformation using generative adversarial network
Multi-modal
Music Visualization for Visually-impaired Persons
Multi-modal

Research Overview

Tinnitus, also known as the phantom auditory perception; perceiving sound in the absence of an external acoustic stimulus, becomes one of popular sense organ disorders. According to the report from U.S. and Western Europe, there are at least 70 million chronic tinnitus patients in EU. Moreover, 10~15% of adults suffer chronic tinnitus and 1% of them suffer severe enough to have a significant adverse effect on their lifestyle and quality of life.
Hearing loss is known as the main cause of tinnitus. In this sense, a majority of tinnitus patients are the old since hearing loss gets worse as the people get older. Therefore, tinnitus treatment is traditionally aimed to recuperate the patient’s hearing ability. Therapies using hearing aid are adopted to handle the hearing loss based on neuroscientific backgrounds.
Hearing loss is known as the main cause of tinnitus. In this sense, a majority of tinnitus patients are the old since hearing loss gets worse as the people get older. Therefore, tinnitus treatment is traditionally aimed to recuperate the patient’s hearing ability. Therapies using hearing aid are adopted to handle the hearing loss based on neuroscientific backgrounds.
This project is aimed to develop a novel tinnitus treatment method. This method considers a preference of each patient as well as recovering the hearing function for both better treatment effect and therapy compliance. To this end, below techniques are adopted:
•
Functional imaging
•
User preference based music recommendation
•
Audio signal processing
•
Computational modeling between musical features and stimulated brain regions
Using techniques listed above, novel method can provide patient-tailored treatment and maximize remedial value using music with personal preference.

Project Members

•
Kyogu Lee, Kyungmyun Lee (Professor @ SNU)
Tinnitus Therapy based on Music and Brainwave Analysis
Neuroscience
•
Motive
•
Work in progress
Reference
[6] Cantisani, G., Trégoat, G., Essid, S., & Richard, G. (2019). MAD-EEG: an EEG dataset for decoding auditory attention to a target instrument in polyphonic music. Speech, Music and Mind (SMM), Satellite Workshop of Interspeech 2019.
[7] A. Ofner and S. Stober, “Shared Generative Representation of Auditory Concepts and EEG to Reconstruct Perceived and Imagined Music,” 19th International Society for Music Information Retrieval Conference – ISMIR 2018, pp. 392–399, 2018.
[8] Lawhern, V. J. et al. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of Neural Engineering 15, 056013 (2018).

If you have any questions, please contact the first author.

Reconstruction of Musical Features from EEG recordings during music listening
Neuroscience
Authors: Ju-Sung Ham(stephen93@snu.ac.kr), You Jeong Hong, Jeong Mi Park, Jonghwa Park, Juhan Nam, and Kyogu Lee
Keywords: Emotion communication; Expressive performance; Classical Piano Pieces; Musical training;

Backgrounds

Figure 1. Musical factors that affects the emotional expression. From Livingstone et al. (2010)
•
Communication of emotion through music is affected by the factors related to composer, performer and listener.
•
For example, Quinto & Thomson (2013) revealed that listeners’ ratings on the pleasure dimension were more changeable by the composer related factors (e.g., mode, melody), while ratings on the arousal dimension were more changeable by the performer related factors (e.g., tempo, dynamics).
•
Also, Akkermans et al. (2019) found that among several individual factors of listeners, only the level of musical training (Gold-MSI) significantly impacts listeners’ probability of correctly identifying the intended emotion.
•
However, these findings are mainly derived from monophonic melodies, which the majority of prior works have employed for experiments.

Aims

•
Therefore, our study aims to further explore the impact of factors that are related to composer, performer, and listener on emotion communication through polyphonic classical piano pieces, which are frequently consumed in our daily lives.
•
To investigate the accuracy of communication in a more quantitative way, we adopted the three-dimensional model of emotions (Mehrabian, 1996), which consists of pleasure, arousal, and dominance (PAD).
Figure 2. PAD emotional space.P for pleasure, A for arousal, and D for dominance. From Gonzalez-Sanchez et al. (2017).

Recording and Listening Experiment

Communication of Emotion Modified by the Performer in Classical Piano Pieces
Music Perception

Research Overview

The purpose of this research is to develop a real-time sound analysis technology to support unmanned surveillance systems. Industrial needs for unmanned surveillance systems are growing due to the increasing cost of labor. Today’s computer vision technology combined with deep learning algorithms is rapidly advancing to meet the goal of unmanned visual surveillance. On the other hand, auditory information is also a critical part of surveillance. Auditory information can detect abnormal events happening outside the visual angle and abnormal events which are hard to detect using visual information such as people screaming. In this research, auditory features and deep learning technology are used to achieve our objectives.
The research has three key objectives.
1. Detection and classification of abnormal sound event
2. Sound event direction of arrival(DOA) estimation
3. Robot motor noise cancellation

Publications

•
Wansoo Kim and Kyogu Lee, “Sound Event Localization and Detection Using Modular Neural Network Structure”, 대한전자공학회 추계 학술대회, 대한전자공학회, 2018.
Real-time sound analysis technology development
Sound Detection

Research Overview

A variety of household appliances we use in our lives generate noise. If there is an abnormal noise in the appliance, we can guess whether the product is malfunctioning. The purpose of this research is to develop an algorithm to detect the abnormal noise generated in household appliances and to diagnose the fault by using it. If the failure can be diagnosed with only noise without additional inspection, the repair process can be more effectively carried out.
Figure 1. Concept of the system
Development of Abnormal Sound Detection Algorithm for Home Appliances using Deep Neural Network
Sound Detection

Research Overview

The purpose of this project is to develop an interactive/smart music education system especially when learning instrument. For the purpose of this project, we proposed a real-time musical performance feedback system which is combination of common MIR tasks such as onset detection, Pitch detection and Score following.
This system detects pitch and onset from live audio stream and make note sequence with previous detected note events. The system aligns two note sequence which is from live audio stream and score to calculate score position, wrong note. Figure 1. shows the overview of the system. An example of rendered music note is shown in Figure 2. The music notation wass rendered using VexFlow with HTML5. The information from the backend, such as the incorrect notes and current position, are sent to VexFlow to be rendered accordingly. This HTML page is displayed on the iOS through the UIWebView class, which is used to embed web content in applications. Any additional buttons used for user interactions are coded natively through Xcode.
Figure 1.
Figure 2.

Publications

•
S. Kwon, B. Han, Y. Han and K. Lee, “A Musical Performance Feedback System Based on Real-time Score-Following”, HCI Korea 2013, 2013.
•
Y. Han, S. Kwon, K. Lee and K. Lee, “A Real-time Musical Performance Feedback System for Beginner Musician”, NIME 2013, 2013.

Project Members

Kibeom Lee, Yoonchang Han, Sungkyun Jang, Sejun Kwon

Funding Agency

Smart Interactive Music Edutainment

Research Overview

In recent years, automatic and objective classification of mood disorders, such as depression using various signals (speech, face, brain wave) has got popularity.Clinical depression is a severe mental health disorder and it has a wide range of potential symptoms including like strong negative conceptualization and observable psychomotor retardation.
Current diagnosis tool for depression depends almost exclusively on self-report and clinical opinion by psychiatrist. But it has critical weakness that monitoring patient’s status is very difficult. Most of people suffer from depression, which also has no strong will to revisit hospital for medical care. So developing automatic monitoring/classification system for patients with depression can have a major impact on society.
Speech has strong cues for classifying patient’s status. Speech in patients with depression has reflection that often described as being stagnant with diminished prosody. Paralinguistic analysis of depressed speech has shown that physiological symptoms relating to depression affects vocal tract properties.
The aim of this project is to investigate the utility of affective sensing methods for automated depression analysis, which can assist psychiatrist in depression diagnosis and monitoring. Remote depression diagnosis/monitoring system can cover a large number of depressive patients.

Project Members

Subin Lee, Ziwon Hyung, Jongkyu Shin, Joengsoo Park

Funding Agency

Seoul National University, Seoul National University Bundang Hospital
Development of Voice-based Diagnostic Marker for Depression

Research Overview

The purpose of this research is to collect respiratory sound including snoring among biological sounds generated during sleep, to be used for the development of a personalized sleep management device for the management of routine sleep in everyday life.
The research will include an analysis of sound information generated during sleep, selection of acoustic biomarker, development of a medical algorithm of a new concept based on the acoustic marker which can measure the non-invasive sleep status, development of a smart sleep mask through fusion research, and evaluation of the accuracy and efficacy of these methods using the results of sleep polyvalence test and sleep apnea test as well as a clinical trial and pilot application.
In addition, the goal of this research includes development of a smart sleep mask to a higher level, to be used for the development of various sleep-related health management scenarios using this smart sleep mask and trial application.

Publications

•
Jaepil Kim, Taehoon Kim, Jeong-Whun Kim, and Kyogu Lee, “An Investigation of the Snoring Sound Classification Using the Cyclostationary Attributes”, in Proc. of International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, 2016
•
Jaepil Kim, Taehoon Kim, Jeong-Whun Kim, and Kyogu Lee, “CYCLOSTATIONARITY-BASED ANALYSIS OF NOCTURNAL RESPIRATION SOUNDS”, in Proc. of International Congress on Sound and Vibration (ICSV), Athens, Greece, 2016
AcoustoSleepMask: Development of smart sleep mask using acoustic biomarker and machine learning

Research Overview

This project is supported by Urban Data Science Laboratory under the Seoul Metropolitan Government and uses big data to solve various problems in the city. In particular, the Platform part creates a platform for collecting and analyzing multiple pieces of data in the city and solves problems based on data collected by this platform.
Currently, research on real-time noise value collection and analysis in Seoul is underway, and research on real-time detection of abnormal noises that may occur in traffic and crime prevention environments in cities such as screams, glass breaking sounds, and horn sounds is underway.
Figure 1. platform device; audio analysis module

Publications

Urban Data Science Laboratory Project for Future City with Data Foundation
Research Overview
The purpose of this research is to monitor the condition of a baby using the sounds that the baby makes. Childcare experts with sufficient experience claim that they can predict the condition of a baby such as the reason for crying with the sound that the baby is making. The goal of this research is to transfer the knowledge of childcare experts to a deep learning model. This deep learning model will benefit new parents struggling to figure out the reason why their baby is crying. The first step of this research is to collect sufficient baby sound database with appropriate tags. The collected database will be used to train out deep learning model.
Baby sound database collection and baby condition monitoring using machine learning

Obstructive Sleep Apnea Screening Test Techniques Development using Analysis and Classification of Snoring Sound

Research Overview

Obstructive Sleep Apnea (OSA) is the most common type of sleep disorder and is caused by obstruction of the upper airway during sleep. This symptom arouses pauses in breathing which lead to deoxygenation and consequent arousals.
Figure 1. Obstructive sleep apnea (from wikipedia)
The effects of sleep disorders are extensive, impacting sufferers physically, psychologically and financially. Up to 40% of the US adults experience problems with falling asleep or daytime sleepiness. (Hossain and Shapiro 2002)
Figure 2. Effects of sleep disorders (from Ronald Kessler, Harvard Medical School 2006)
Moreover, diagnosis rate of OSA is very low on account of the diagnostic complexity and heavy expenditure although this sleep disorder has high fatality and prevalence.Recently, interests about wearable device and telemedicine have been increasing and market requires high clinical accuracy in personalized handheld devices for mobile healthcare. Especially, OSA is typically cannot be aware of the disease in the single person households. So if we can develop simple and accurate algorithm which can be an OSA screening test, this is a good solution for national health promotion and creates values for next generation mobile healthcare system.
In our study, we only use snoring sound in the polysomnography, although other studies typically used various physiological parameters (oxygen saturation, body movement, EEG, EOG, EMG etc.). We extract various audio features in the perspectives of temporal structures and raw signal processing. Or we may conduct feature learning which is a set of techniques in machine learning that learn a transformation of raw inputs to a representation that can be effectively exploited in a supervised learning task such as classification.
Through this research, we may obtain good classification or regression models to inform the patients’ OSA status.
This research is underway. So, if we get a significant good performance, we can update this page immediately.

Project Members

Taehoon Kim, Yoonchang Han, Ziwon Hyung
Development of Obstructive Sleep Apnea (OSA) Screening Algorithm through Acoustic Analysis

An Ultra-real New Media Art Using 3D Virtual Reality and Precise Real-time Position Tracking System

Research Overview

In this project, a.k.a Blind Perspective, we wanted to integrate two different perspectives in one same space. The two different perspectives are assigned as the real vision, and the vision seen through virtual reality. The idea is to provide a highly immersive virtual reality experience by physically matching the virtual world space dimensions with those in the real world.
The idea became possible by integrating head-mounted displays (HMDs) with accurate indoor-position tracking technology. As the proliferation of stereoscopic 3D virtual reality devices is reshaping the user experience in immersive virtual reality methods, we believed that these low-cost HMD devices are the perfect 3D vision interpreter among other kinds. The system is designed to have six degrees of freedom in the virtual world, which allows users to physically move around in the real world while wearing the wireless system with an HMD. To eliminate the potential safety risk factors such as a user being bumped into an obstacle caused by the mismatched reciprocal positions between the real and the VR or the cybersickness which mainly causes dizziness and nausea due to the sensory gap between the motor and the visual cortex, our main task was to establish a high standard in terms of positioning accuracy and steadiness.
We have utilized the ultrasound as the underlying technology for our position tracking algorithm as it enables the finest accuracy and the fast computational speed. One omni-directional ultrasound transducer mounted on the HMD generates a burst tone at 20 Hz at carrier frequency of 40 kHz. The burst signal is transmitted wirelessly from the signal generator to HMD via 5.8 GHz RF range transmission. The receivers mounted on the ceiling of the room calculate the position by triangulating the signal. A total of 16 signal receivers were used to cover the corridor space of 35 meters in Contemporary Art Gallery in Art Tower Mito.
After refining our position tracking algorithm, the test results show that the system has the accuracy of less than 10mm on average. Apart from developing algorithms, we have also implemented dedicated devices which are necessary for the integration of the HMD and the positioning system. Additionally, our system was designed to be wearable, requiring no external power or signal cables. The system is consisted of two main parts, the user position tracking part with the data acquisition equipment, and the HMD integration part with the unity 3d engine. The diagram of all the components is shown in Figure 1.
Figure 1. The System components diagram
As the purpose of our system is to provide immersive virtual experience with natural interaction/control, we are proud to be a part of the exhibition, yet hope to improve the system by covering a larger space with higher accuracy and immersion.

Publications

Blind Perspective

Research Overview

With the increasing number and popularity of new motion-sensing hardware, and touch screen mobile devices improving the usability of the new interfaces is becoming more important. However, people do not accept the new interfaces as comfortably as traditional ones because they lack in tactile or haptic feedback that traditional input methods such as mice or keyboards provide. However, having tactile feedback in these interfaces is not possible unless users are wearing a device with actuators. Therefore, auditory feedback is an appropriate and unique alternative for assisting visual feedback.
At MARG, we explored various types of novel auditory feedback methods and explore their effects as secondary feedback on complementing visual feedback. We performed a user study for a menu selection task and experimental results show that the proposed auditory feedback is significantly more efficient and more effective than visual-only feedback or conventional auditory feedback in terms of time and accuracy.
Conventional Auditory Feedback on Menu Selection in Gesture Interface
In most gesture interfaces, auditory feedback is used to indicate the transitions between menu items.Transitional auditory feedback is usually a short and percussive clicking sound which triggers on the boundary or center of a menu item. This auditory feedback has shown to be helpful for separating items on a circular menu.

Auditory Feedback for Browsing Assistance

•
Pre-emptive Auditory Feedback
Weaknesses of the transitional auditory feedback lies on the fact that it provides limited information about the position of the gesture in the circular menu. In gesture interfaces, providing single auditory feedback at the transition between menu items is not sufficient to alert the user about the change of the states. Pre-emptive auditory feedback provides additional auditory feedback just before the transitional auditory feedback which triggers sound at the boundary between menu items. It also informs the user that the transition between menu items is imminent. The pre-emptive auditory feedback has additional auditory feedback ten degrees before the triggering point of the transitional auditory feedback. This additional feedback, supplementing conventional transitional auditory feedback, will aid users in noticing the completion of browsing to the desired menu item.
•
Continuous Auditory Feedback
New Interfaces with Auditory Feedback