Speech-to-Singing Conversion in an Encoder-Decoder Framework

Jayneel Parekh       Preeti Rao       Yi-Hsuan Yang

ICASSP 2020 Project Webpage


Abstract

In this paper our goal is to convert a set of spoken lines into sung ones. Unlike previous signal processing based methods, we take a learning based approach to the problem. This allows us to automatically model various aspects of this transformation, thus overcoming dependence on specific inputs such as high quality singing templates or phoneme-score synchronization information. Specifically, we propose an encoder–decoder framework for our task. Given timefrequency representations of speech and a target melody we learn encodings that enable us to synthesize singing that preserves the linguistic content and timbre of the speaker while adhering to the target melody. We also propose a multi-task learning based objective to improve intelligibility. We present a thorough quantitative and qualitative analysis of our framework.




Code

Code is uploaded on github and can be found here..




Audio Samples

Chrome is the preferred browser. The samples may not always run in Firefox. Use of headphones/earphones recommended.

Sample A (Outputs of all systems)

Input speech:             Target melody:    
Baseline 1 (No Melody Prediction):    
Baseline 2 (No IN layers, ResBlocks):    
Proposed-MSE:    
Proposed-MTL:    



Sample B (Outputs of all systems)

Input speech:             Target melody:    
Baseline 1 (No Melody Prediction):    
Baseline 2 (No IN layers, ResBlocks):    
Proposed-MSE:    
Proposed-MTL:    



Sample C (Proposed-MSE vs Proposed-MTL)

Input speech:             Target melody:    
Proposed-MSE:    
Proposed-MTL:     Better pronunciation of multiple consonants



Sample D (Proposed-MSE vs Proposed-MTL)

Input speech:             Target melody:    
Proposed-MSE Prediction:    
Proposed-MTL Prediction:     Notice the better pronunciation of 'just' and 'girl'



Sample E (Proposed-MSE vs Baseline 2)

Input speech:             Target melody:    
Baseline 2 (No IN layers, ResBlocks):    
Proposed-MSE:    



Supplementary

Network Architecture