MIP-Frontiers

Future steps in version identification

2022-03-30T00:00:00+00:00

As my final blog post for the MIP-Frontiers project, I have chosen to give my two cents on the future of musical version identification (VI) research and technologies. The topics that I elaborate on below are all included in our survey article and my dissertation. My goal here is to give a brief summary of the selected set of future work mentioned in those two works.

As opposed to the categories me and my colleagues used in our previous work, I will now attempt to categorize the potential strategies for the VI research into two groups: short-term and medium-term strategies. For a more detailed perspective on the issues below and for some other future work, our survey article would be a great place to start.

1. Strategies for short-term gains

Considering that the main use case of VI systems is digital rights management, the short-term strategies I mention below focus more on developing reliable systems that can satisfy the current needs of the relevant stakeholders. Assuming that an interest from the industry may give a quick boost to the speed of VI research and development, unleashing the full potential of such systems would be beneficial to the overall VI ecosystem.

1.1 Incorporating lyrics into the workflow

Historically, VI systems have used harmonic and melodic information from audio signals by extracting chroma and dominant melody features, respectively. The experimental results have shown that systems that use such information can demonstrate reliable performances for many mainstream use cases. Apart from the fact that harmonic and melodic characteristics are expected to show a certain degree of similarity between versions, another reason why those features were the popular choice among VI researchers is that they are relatively easy to compute and process.

In the last year, VI research has witnessed an important development in going beyond the harmonic and melodic characteristics. Vaglio et al. used a system that extracts lyrics from audio signals and proposed to use them for VI. Since it is a very intuitive idea, it may surprise people that it hadn’t been tried before. However, the main obstacle was having a reliable system that can extract lyrics directly from the audio signals. Luckily, the research community has come up with interesting solutions to this problem, and developing VI systems that use lyrics in their workflow is a very viable and promising idea today.

1.2 Using automatically generated tags/metadata

Another interesting and not-yet-tested idea is to use a music classification system to obtain some information about the audio track that may help the VI system. Such an “auto-tagging” system may detect the genre of a track, or whether it has vocals or is completely instrumental. Such side information can be used to steer the VI system on the right path. Exploring potential synergies between VI and other MIR tasks would absolutely be a worthy effort.

One of the major topics of my dissertation was the scalability perspective of VI systems. To be taken seriously by the industrial stakeholders, this is an important aspect to take into account. However, these days, many information retrieval systems and machine learning research, in general, seem to put enough emphasis on this issue. Therefore, I think VI research just needs to be aware of general tendencies that going on in the related research fields, rather than devoting a considerable amount of effort into this aspect.

1.4 Handling short queries

Historically, VI systems have always been designed to process and compare entire tracks or long segments of audio (e.g., 1-2 minutes). Although this is desirable behavior for many of the VI use cases, querying short audio segments (e.g., 5-15 seconds) may facilitate exploring some new commercial use cases that so far have not been studied. Although it sounds like a straightforward extension to the current capabilities of VI systems, the wide range of musical characteristics that can differ between versions make it drastically more difficult to solve this problem using only a short snippet of audio. In fact, even humans may require a bit more than just a few seconds when they are asked to decide whether two tracks are versions of each other. Therefore, although intriguing, this use case requires serious efforts to yield reliable systems.

1.5 Postprocessing operations

VI systems try to return the most similar item for each query independently, and because of this, the relations among a group of items are often discarded. However, by considering the versions that originate from the same musical work as members of the same set, certain postprocessing operations can be designed. Such operations may take advantage of the cases where VI systems are more confident in order to revise decisions in cases where the systems can’t point to a clear solution. Such postprocessing operations are generally low-cost but highly effective.

2. Issues to address for medium-term benefits

The points I explain below are related more to some of the current issues with the VI research. Although such issues are mostly of a conceptual nature and may not directly affect the performance of VI models, keeping them in sight while working on the short-term strategies outlined above may result in a healthier and more sustainable development of VI technologies.

2.1 Definition of musical versions and musical work

Throughout my research, I have repeatedly mentioned that the main use case of VI systems is digital rights management and detecting cases of copyright infringement. An important point for achieving this goal is to correctly define what constitutes a copyright infringement. For this, we have to think about several questions: “what is a musical work?”, “what is a composition copyright?”, “what commonalities do two tracks need to share so that they can be considered as versions?”, etc. To study VI at any degree, one must answer such questions.

In our research, we have favored a quite permissive definition: a version can be considered as any reinterpretation of an existing musical work. Although such a definition allows us to continue our research from an academic perspective, it may not be practical for industrial applications. In the music ecosystem, oftentimes, two tracks are considered as versions not because of their musical similarities but because of the agreements between rightsholders. This introduces a complexity into our efforts of defining versions that requires us to consult legal entities. Therefore, to design and evaluate VI systems that can seamlessly be used in industrial use cases, one must revise the definitions of musical version and musical work to comply with those of legal perspective.

Apart from the discrepancies between academic and legal definitions of versions, we must think about how to define versions in non-Western musical traditions. The assumptions that we have while designing our systems are likely to fail when applying them to other cultures.

Lastly, one may need to consider musical tracks that are far from general tonal conventions. In Western music, we have a good idea of how similar two tracks should be to be considered as versions of each other; however, what about ambient music or soundscapes? Or when the identity of a track heavily depends on its timbral properties rather than harmonic or melodic ones. Although these are mostly edge cases that VI systems may not encounter frequently, thinking about the extent of musical content those systems should handle can be useful in the future.

2.2 Evaluation metrics

Because they are examples of information retrieval applications, VI systems are mostly evaluated using metrics from other information retrieval frameworks. Although this is pleasant from an academic perspective where we mostly use carefully curated datasets, potential issues in industrial datasets may create some unexpected results on how these metrics work. One such issue is the presence of near-duplicates, which may drastically change the outcome of evaluation metrics like precision and recall (see our survey article). Other than potential unexpected behavior, some current metrics may not correspond to how such systems are evaluated in industrial contexts. Therefore, a careful study on designing or choosing evaluation metrics for VI may result in future benefits.

2.3 Genre distributions

The behavior of VI systems on different genres has been underexplored. It is quite intuitive that systems that process only harmonic or melodic information cannot perform equally well on a wide range of genres; however, only a handful of previous research has explicitly tried to quantify any potential discrepancies. Developing systems that perform well for most popular genres first requires an analysis of the drawbacks of current systems. By detecting the cases where the current systems perform badly, specific measures can be developed in order to obtain high-performance VI systems.

MIP-Frontiers Final Workshop

2021-10-11T00:00:00+00:00

Date: Thursday 15th - Friday 16th of October 2021
Time Zone: UTC+01 (UK summer time)
Location: Online

Nature and Purpose of the Workshop

MIP-Frontiers project (New Frontiers in Music Information Processing) has supported 15 PhD students at four universities working with a range of industry and cultural partners. This final workshop allows the fellows to present the main results of their research over the last three years. We have also invited several distinguished researchers in the field to give keynote presentations.

Thursday 14th October 2021

9:00	Welcome
	Simon Dixon, Queen Mary University of London
	Domna Paschalidou, European Research Executive Agency
9:15	Data-Driven Musical Version Identification: Accuracy, Scalability, and Bias Perspectives
	Furkan Yesiler, Universitat Pompeu Fabra
	Deep Learning Methods for Musical Instrument Separation and Recognition
	Carlos Lordelo, Queen Mary University of London & DoReMIR
10:30	Informed Audio Source Separation with Deep Learning in Limited Data Settings
	Kilian Schulze-Forster, Telecom Paris
	Deep Neural Networks for Automatic Lyrics Transcription
	Emir Demirel, Queen Mary University of London
11:45	Data-Driven Approaches for Query by Vocal Percussion
	Alejandro Delgado, Queen Mary University of London & Roli
12:15	Lunch Break
14:00	*Keynote 1: Applications of Machine Intelligence in Computational Acoustics*
	Augusto Sarti, Politecnico di Milano
15:00	Towards Neural Context-Aware Performance-Score Synchronization
	Ruchit Agrawal, Queen Mary University of London
	Autonomous and Robust Live Tracking of Complete Opera Performances
	Charles Brazier, Johannes Kepler University Linz
16:15	Large-Scale Multi-Modal Music Search and Retrieval without Symbolic Representation
	Luís Carvalho, Johannes Kepler University Linz
	Investigating the Behaviour of Audio Classification Models through Adversarial Attacks and Gradient based Interpretability Methods
	Vinod Subramanian, Queen Mary University of London

Friday 15th October 2021

9:00	*Keynote 2: Source Separation Metrics: What are they really capturing?*
	Rachel Bittner, Spotify
10:00	Exploration of Music Collections with Audio Embeddings
	Philip Tovstogan, Universitat Pompeu Fabra
	Audio Auto-tagging as Proxy for Contextual Music Recommendation
	Karim Ibrahim, Telecom Paris
11:15	Deep Learning Methods for Music Style Transfer
	Ondřej Cífka, Telecom Paris
	Exploring Generative Adversarial Networks for Controllable Musical Audio Synthesis
	Javier Nistal, Telecom Paris & Sony CSL
12:15	Lunch Break
14:00	*Keynote 3: Generative Modeling Meets Deep Learning: A Modern Statistical Approach to Music Signal Analysis*
	Kazuyoshi Yoshii, Kyoto University
15:00	Neuro-Steered Music Source Separation
	Giorgia Cantisani, Telecom Paris
	Automatic Characterization and Generation of Music Loops and Instrument Samples for Electronic Music Production
	António Ramires, Universitat Pompeu Fabra
16:00	Closing
	Domna Paschalidou, European Research Executive Agency

Please remember to download zoom app and if you have any issues, please email a.bort@qmul.ac.uk

Keynote Speakers

Augusto Sarti: Applications of Machine Intelligence in Computational Acoustics

Information retrieval and machine learning have grown to the point of playing an leading role in all aspects of sound analysis. There are some research areas, however, in which the potential of information retrieval techniques is only now beginning to have an impact on the research community. One of these areas is musical acoustics. In this area, in fact, machine intelligence and information retrieval have been widely used only for timbral analysis. Machine learning is now beginning to play a relevant role also in the analysis of vibrational and acoustic properties of musical instruments. With this talk I would like to offer an overview on some emerging methodologies for vibrational, acoustic and timbral analysis based on machine intelligence.

Rachel Bittner: Source Separation Metrics: What are they really capturing?

The task of source separation has relied on what has become a standard set of metrics for the past 15 years. These metrics have been the basis for evaluating the latest “state-of-the-art”, and for determining when design choices are good or bad. Yet a number of articles have shown that these metrics are limited in how well they correlate with human perception. This talk gives a history of the metrics, their evolution, implementations, and what we use today. Then, we dive into the mathematical properties of these metrics and give illustrative examples. We conclude with a short survey of alternative metrics from the literature, and criterion for what we might consider when defining a new standard.

Kazuyoshi Yoshii: Generative Modeling Meets Deep Learning: A Modern Statistical Approach to Music Signal

I will present a modern statistical approach to automatic audio-to-score transcription based on an effective combination of an acoustic model, a language model, and an inference model. These models are implemented with classical probabilistic models (e.g., HMM) or with deep neural networks (e.g., LSTM) for richer expression capabilities if necessary. As subtasks of music transcription based on this approach, I will introduce music structure analysis and automatic transcription of singing voice, chords, keys, drums for popular music. I will also introduce the state-of-the-art automatic piano transcription system that can yield decent symbolic piano scores.

Fellows Presentations

Furkan Yesiler: Data-Driven Musical Version Identification: Accuracy, Scalability, and Bias Perspectives

This thesis aims at developing audio-based musical version identification (VI) systems for industry-scale corpora. To employ such systems in industrial use cases, they must demonstrate high performance on large-scale corpora while not favoring certain musicians or tracks above others. Therefore, the three main aspects we address in this thesis are accuracy, scalability, and algorithmic bias of VI systems.

We first propose a deep learning-based model that incorporates domain knowledge in its network architecture and training strategy. We design explicit modules to handle common transformations between musical versions (e.g., key, tempo, structure, etc.). We then take two main directions to further improve our model. Firstly, we investigate embedding distillation techniques to reduce the size of the embeddings produced by our model, which reduces the requirements for data storage and, more importantly, retrieval time.

Secondly, we experiment with data-driven fusion methods to combine information from models that process harmonic and melodic information, which greatly enhances identification accuracy. After exploring potential improvements in accuracy and scalability, we analyze the algorithmic biases of our systems and point out the impact such systems may have on various stakeholders in the music ecosystem (e.g., musicians, composers) when used in an industrial context. We conclude our research by analyzing the performance of our proposed systems on two industrial use cases, in collaboration with a broadcast monitoring company.

Overall, our work addresses the research challenges of the next generation of VI systems. We show the feasibility of developing systems that are both accurate and scalable at the same time by carefully combining domain knowledge into data-driven workflows. We believe that our contributions will accelerate the integration of VI systems into industrial scenarios, and, thus, the impact of VI research on musicians and listeners will be more eminent than ever.

Carlos Lordelo: Deep Learning Methods for Musical Instrument Separation and Recognition

Automatically transcribing polyphonic music to a score is a challenging task and one of the most discussed topics in the Music Information Retrieval (MIR) community. In particular, when analysing recordings with multiple instruments, the transcription process becomes even more complex, because not only each note should have its pitch and duration properly estimated, but the information regarding the timbre of sounds should also be correctly processed. It is mandatory to have a way of recognising the instrument that played each note and associate each sound to the correct voice in the final staff notation. Moreover, when dealing with signals with pitched and non-pitched sounds (drum kits), drum detection and classification is often performed independently because the sound characteristics of drum instruments differ in many aspects from pitched instruments that constitute the melodic and harmonic nature of music.

With the final objective of allowing a more general multi-instrument automatic music transcription, where not only harmonic, but also percussive instruments could be present in the signal, in this project we propose and deeply investigate deep-learning-based harmonic-percussive source separation and instrument recognition methods. We show that by musically motivated architectures can improve the performance of each of those tasks and with the usage of the Tap & Fiddle Dataset, a dataset curated as part of this project containing 28 stereo recordings of traditional Scandinavian fiddle tunes with accompanying foot-tapping, we show that unsupervised domain adaptation methods can help in cases where no labelled data is available for specific instruments. Regarding instrument recognition, we investigate the pitch streaming task and propose novel deep-learning-based methods that can perform the task when any multi-pitch estimations are provided as input to the system.

Kilian Schulze-Forster: Informed Audio Source Separation with Deep Learning in Limited Data Settings

Audio source separation is the task of estimating the individual signals of several sound sources when only their mixture can be observed. State-of-the-art performance for musical mixtures is achieved by Deep Neural Networks (DNN) trained in a supervised way. They require large and diverse datasets of mixtures along with the target source signals in isolation. However, it is difficult and costly to obtain such datasets because music recordings are subject to copyright restrictions and isolated instrument recordings may not always exist.

In this dissertation, we explore the usage of prior knowledge for deep learning based source separation in order to overcome data limitations.

First, we focus on a supervised setting with only a small amount of available training data. We investigate to which extent singing voice separation can be improved when it is informed by lyrics transcripts. To this end, a novel deep learning model for informed source separation is proposed. It aligns text and audio during the separation using a novel monotonic attention mechanism. The lyrics alignment performance is competitive with state-of-the-art methods while a smaller amount of training data is used. We find that exploiting aligned phonemes can improve singing voice separation, but precise alignments and accurate transcripts are required.

Finally, we consider a scenario where only mixtures but no isolated source signals are available for training. We propose a novel unsupervised deep learning approach to source separation. It exploits information about the sources’ fundamental frequencies (F0). The method integrates domain knowledge in the form of parametric source models into the DNN.

Experimental evaluation shows that the proposed method outperforms F0-informed learning-free methods based on non-negative matrix factorization and a F0-informed supervised deep learning baseline. Moreover, the proposed method is extremely data-efficient. It makes powerful deep learning based source separation usable in domains where labeled training data is expensive or non-existent.

Emir Demirel: Deep Neural Networks for Automatic Lyrics Transcription

Lyrics are an essential building block for the representing, understanding and appreciation of singing performances. Therefore, the automatic retrieval of lyrics from singing performances, or automatic lyrics transcription, has a number of potential industrial applications, though the performance of such systems had not reached to a level to be used in industrial applications. In our project, we develop the first automatic lyrics transcriber module that is integrated with an automatic music transcription system to be used in large scale applications. To achieve this, we propose a number of novel methods for an improved lyrics transcriber, such as a compact multistreaming neural networks architecture, cross-domain training, singing adapted pronunciation dictionary and music informed silence modeling. In addition, we introduce a new evaluation set for this task that is much larger than the existing benchmark test sets. Finally, we provide a quantitative comparison between the state-of-the-art DNN-HMM and end-to-end methods within this context. While our best performing model sets the new state-of-the-art in lyrics transcription, it is going to be included as a novel feature of the new ScoreCloud - Songwriter app which is planned to be released in the upcoming months. We will finalize our talk with an initial demo for the industrial application of the automatic lyrics transcription technology.

Alejandro Delgado: Data-Driven Approaches for Query by Vocal Percussion

The imitation of percussive sounds via the human voice is a natural and effective tool for communicating rhythmic ideas on the fly. Query by Vocal Percussion (QVP) is a subfield in Music Information Retrieval (MIR) that explores techniques to query percussive sounds using vocal imitations as input, usually plosive consonant sounds. In this way, fully automated QVP systems can help artists prototype drum patterns in a comfortable and quick way, smoothing the creative workflow as a result. This project focuses on applying data-driven approaches to the two most important tasks in QVP. On the one hand, the task of Drum Sample Retrieval by Vocal Imitation (DSRVI) aims at picking different-sounding samples by timbral similarity with a given vocal imitations. This is a problem of correspondence between two acoustic spaces, the one for the real drum samples and the one for the vocal imitations, and thus the main objective is to learn the set of audio features that best link them. On the other hand, the task of Vocal Percussion Transcription (VPT) works by identifying distinct vocal percussion utterances that trigger individual drum samples. This problem, in contrast, is one of correspondence between a sound and a label (classification), and the relevant set of audio features is the one that best separates all classes, independently of how the triggered drum samples sound like. In this study, we try to give robust solutions to these two problems using recent deep learning techniques so that music producers can have a more pleasant experience when searching for sounds and composing beats.

Ruchit Agrawal: Towards Neural Context-Aware Performance-Score Synchronization

Music synchronization aims at providing a way to navigate among multiple representations of music in a unified manner, lending itself applicable to a myriad of domains like music education, performance, enhanced listening, automatic accompaniment and so on. This project focuses on improved music synchronization in real life settings. This entails developing robust alignment methods which have significant domain coverage and can adapt to the setting they are being employed in. This project develops synchronization methods applicable to both audio-to-audio and audio-to-score alignment and addresses important challenges that make up limitations of traditional alignment algorithms.

Charles Brazier: Autonomous and Robust Live Tracking of Complete Opera Performances

This project addresses the challenging task of tracking complete opera performances in real-time along with their respective scores. So far, existing approaches have proven their efficiency at tracking full orchestral works with accuracy and robustness. However, these approaches fail at tracking operas. Such trackers must not only deal with a continuous musical recording, but also have to consider a complex mixture of polyphonic music and singing voice, acting sounds, interludes, breaks, and also applause from the audience. All those parameters interfere with the tracking process and tend to provoke failures where the tracker is lost in the score. To address this issue, we propose to develop new methods to integrate different extra-musical knowledge sources (e.g. acoustic event detection, speech-specific features, acoustic model) into state-of-the-art music score following algorithms to achieve robust and accurate opera tracking during live opera tracking in real conditions.

Luís Carvalho: Large-Scale Multi-Modal Music Search and Retrieval without Symbolic Representation

The goal of this project is to propose methods for the automatic structuring and cross-linking of large multi-modal music collections, with focus on audio recordings and sheet music images, and without the need for symbolic representation. These methods should support tasks such as the retrieval of one modality based on another one, identification of different versions of the same material, and piece identification in unknown recordings. We have then identified two main trends for this research. First we focus on how to learn better audio-to-sheet music correspondences, following recent advances in deep neural networks. Our approach consists of learning similarities between short snippets of audio spectrograms and staff-wise unrolled sheet music pages. Second, we build upon the learned correspondences and investigate how to best exploit them for identification and retrieval tasks on real multimodal archives of music, aiming for fast and scalable methods.

Vinod Subramanian: Investigating the Behaviour of Audio Classification Models through Adversarial Attacks and Gradient based Interpretability Methods

In this presentation we will explore different methods that help us understand how audio classification models work. The two main methods are adversarial attacks and interpretability. Adversarial attacks allow us to perturb the input to fool the classifier and interpretability uses the gradients of the classifier to show what parts of the input are important for the model prediction. We focus on motivating our research in the audio domain, tackling challenges unique to audio, and exploring directions that the community must take to mature the field.

Philip Tovstogan: Exploration of Music Collections with Audio Embeddings

Music recommendation systems are an integral part of modern music streaming services. To balance user retention and diversity of recommendations, most industrial systems utilize exploit vs explore model. In this thesis, we focus on music exploration as opposed to exploitation, as this area of research is less developed and better suited to the academic environment as opposed to industry, which mostly focuses on improving the exploitation performance. We propose a novel approach to music exploration that utilizes visualization of music in continuous semantic latent space instead of browsing through artists, genres or moods.

We release MTG-Jamendo - a new open-source auto-tagging dataset that provides full audiotracks under Creative Commons license with tags categorized between genres, moods, and themes and instruments that are useful for research on music exploration. We utilize state-of-the-art deep auto-tagging systems to perform the evaluation of the dataset.

We present a novel web interface to visualize music collections using the audio embeddings extracted from music tracks. The system allows exploring the relationship between music tracks from multiple perspectives and on different levels (segments vs full tracks), displaying embedding spaces extracted by music auto-tagging models, trained using different architectures and datasets, coupled with various 2D projection algorithms. We conduct a user study to analyze the appropriateness of different embeddings and visualizations on the music collections, particularly for playlist creation and music library navigation and rediscovery. Our results show that such an interface provides a good alternative to standard hierarchical library organization by metadata. Furthermore, we provide the analysis of the participants’ preference for different audio embeddings and visualization algorithms.

Karim Ibrahim: Audio Auto-tagging as Proxy for Contextual Music Recommendation

The exponential growth in volume of online services and user data changed how we interact with various service providers, and how we explore and select new products. Hence, there is a growing need for methods to recommend the appropriate items for each user. In the case of music, it is more important to recommend the right items at the right time. It has been well documented that the context, i.e. the listening situation of the users, strongly influences their listening preferences. Hence, there has been an increasing attention towards developing context-aware systems. State-of-the-art approaches are sequence-based models aiming at predicting the tracks in the next session using available contextual information. However, these approaches lack interpretability and serve as a hit-or-miss with no room for user involvement. Additionally, few of previous approaches focused on studying how the audio content relates to these situational influences, and even to a lesser extent making use of the audio content in providing the contextual recommendations. Hence, these approaches suffer from both lack of interpretability and the cold-start problem.

Ondřej Cífka: Deep Learning Methods for Music Style Transfer

In music, composers, arrangers, performers and producers often adapt existing pieces to different contexts and audiences. Recently, deep learning methods have enabled transforming musical material in a data-driven manner, setting the ground for tools which could partially automate this process. The research performed in this area so far has focused largely on conversion between a small set of musical genres or instrument timbres, and on tasks that involve completing a partial arrangement in a desired style. The focus of this thesis, on the other hand, is on a family of tasks which we refer to as (one-shot) music style transfer, where the goal is to transfer the style of one musical piece or fragment onto another. We propose two specific tasks in this direction: (1) accompaniment style transfer for symbolic music representations (i.e. digital scores or MIDI files), and (2) timbre transfer for audio recordings. For each of these tasks, we propose novel methods based on deep learning, as well as evaluation protocols. Additionally, we present a broader contribution related to the processing of sequences (music or otherwise) using Transformer neural networks.

In the first part of this work, we focus on supervised methods for symbolic music accompaniment style transfer, aiming to transform a given piece by generating a new accompaniment for it in the style of another piece. The method we have developed is based on supervised sequence-to-sequence learning using recurrent neural networks (RNNs) and leverages a synthetic parallel (pairwise aligned) dataset generated for this purpose using existing accompaniment generation software. We propose a set of objective metrics to evaluate the performance on this new task and we show that the system is successful in generating an accompaniment in the desired style while following the harmonic structure of the input.

In the second part, we investigate a more basic question: the role of positional encodings in music generation using Transformers. In particular, we propose stochastic positional encoding (SPE), a novel form of positional encoding capturing relative positions while being compatible with a recently proposed family of efficient Transformers. The main theoretical contribution of this work is to draw a connection between positional encoding and cross-covariances of correlated stochastic processes. We demonstrate that SPE allows for better extrapolation beyond the training sequence length than the commonly used absolute positional encoding.

Finally, in the third part, we turn from symbolic music to audio and address the problem of timbre transfer. Specifically, we are interested in transferring the timbre of an audio recording of a single musical instrument onto another such recording while preserving the pitch content of the latter. We present a novel method for this task, based on an extension of the vector-quantized variational autoencoder (VQ-VAE), along with a simple self-supervised learning strategy designed to obtain disentangled representations of timbre and pitch. As in the first part, we design a set of objective metrics for the task. We show that the proposed method is able to outperform existing ones.

Javier Nistal: Exploring Generative Adversarial Networks for Controllable Musical Audio Synthesis

In this thesis, we study Generative Adversarial Networks (GANs) for musical audio synthesis. We explore various sources of conditional information in order to shape the synthesized sounds according to high-level features. Additionally, we also address a fundamental problem originated from applying image-based GAN architectures to the audio domain: the generation of sounds with variable duration.

Giorgia Cantisani: Neuro-Steered Music Source Separation

In this project, we address the challenge of integrating BCI and music technologies on the specific application of music source separation, which is the task of isolating individual sound sources that are mixed in the audio recording of a musical piece. This problem has been investigated for decades, but never considering BCI as a possible way to guide and inform separation systems. Specifically, we explored how the neural activity characterized by electroencephalographic signals (EEG) reflects information about the attended instrument and how we can use it to inform a source separation system.

First, we studied the problem of EEG-based auditory attention decoding of a target instrument in polyphonic music, showing that the EEG tracks musically relevant features which are highly correlated with the time-frequency representation of the attended source and only weakly correlated with the unattended one. Second, we leveraged this “contrast” to inform an unsupervised source separation model based on a novel non-negative matrix factorisation (NMF) variant, named contrastive-NMF (C-NMF) and automatically separate the attended source.

Unsupervised NMF represents a powerful approach in such applications with no or limited amounts of training data as when neural recording is involved. Indeed, the available music-related EEG datasets are still costly and time-consuming to acquire, precluding the possibility of tackling the problem with fully supervised deep learning approaches. Thus, we explored alternative learning strategies to alleviate this problem. Specifically, we propose to adapt a state-of-the-art music source separation model to a specific mixture using the time activations of the sources derived from the user’s neural activity. This paradigm can be referred to as one-shot, as the adaptation acts on the target song instance only. We conducted an extensive evaluation of both the proposed system on the MAD-EEG dataset which was specifically assembled for this study obtaining encouraging results, especially in difficult cases where non-informed models struggle.

António Ramires: Automatic Characterization and Generation of Music Loops and Instrument Samples for Electronic Music Production

Repurposing audio material to create new music - also known as sampling - was at the foundation of electronic music and is a fundamental component of this practice. Loops are audio excerpts, usually of short duration, that can be played repeatedly in a seamless manner. These loops can serve as the basis for songs that music makers can combine, cut and rearrange and have been extensively used in Electronic Dance Music (EDM) tracks. Similarly, the so-called “one-shot sounds” are smaller musical constructs that are not meant to be looped but that are also typically used in EDM production. These might be sound effects, drum sounds or even melodic phrases. Both loops and one-shot sounds have been made available for amateur and professional music makers since the early ages of electronic music. Currently, large-scale databases of audio offer huge collections of audio material for users to work with. Significant research has focused on easing the navigation of one-shot sounds in these databases, either through similarity search, clustering, high-level description or recommendation of sounds. Loops however have not yet been given such attention.

In our work, we address two fundamental methods for navigating sounds: characterization and generation. Characterizing loops and one-shots in terms of their instruments or instrumentation (e.g. drums, harmony, melody) allows organizing unstructured collections and a faster retrieval for music-making. Generation enables the creation of new sounds which are not present on the database through interpolation or modification of the existing material. To achieve this, we employ deep-learning-based data-driven methodologies for classification (e.g. Convolutional Neural Networks) and generation (Wave-U-Net and Generative Adversarial Networks).

How Multidisciplinary Researchers Come Together to Solve a Problem

2021-07-30T00:00:00+00:00

Multidisciplinary research is described as a technique of research in which the tools of different sciences and disciplines are utilized to study a specific problem. In this article, we examine how this is done through the multidiscplinary case of recommending contextual music. Recommendation systems are one of the most useful tools for online browsing. With huge catalogues growing everyday, a good recommendation system can save the users hours of search by directly proposing items that are likley to interest the user. This is the case for almost all online catalogues including merchandise, movies, books, and music. Those recommendation systems are being continously developed to accomodate the different user needs. However, each domain requires different designs based on the recommended media.

Music, in particular, is one of the most challenging media to recommend. Music tracks are often short, spanning a duration of few minutes, yet quite complex in their content. People often spend hours listening to music one track after the other. While users have their own listening preferences, they could also enjoy exploring new styles. Hence, music recommenders are designed to provide fast and coherent items, while also infusing new items for exploration. All while being personalized to each user. A pretty complex task.

To make it even harder, music is one of the media most influenced by the user situations/conditions, specially since the development of portable music players. For example, we listen to different music if we are accompanied by friends if we are alone. It is also different music when we are sad or happy, or when we are doing sports or studying. The list of influencing situations is long and can keep going on if we start to think about it. Researchers at crossroad of psychology, information retrieval, and machine learning have been trying to identify these influencing situations and its effects on the listener preferences for music content in order to provide better recommendations.

Crossroad of domains

In the psychology domain, researchers focus on understanding how people use music in their daily life. They try to understand how much of our daily engagement with music is attributable to individual characteristics and how much it depends on the listening situations, and which are these situations. Early before the progress in information retrieval, they identified the influence of the listening situation on the listener preferences and tried to categorize it. For example, situations could be categorized as either environment-related or user-related. Environment situations reflect the location, time, weather, season, among others. While user situations included the user activity, mood, social setting, etc. These situations are still independent from the listener’s personal preferences shaped by their personality and culture, which is also a research question being studied. Those studies enable us to narrow down the potential situations and study their impact on our preferences. As an example, the following table shows the results from one of the studies ¹ indicating the frequency of different situations that influence our listening preference.

Example of music-listening situations and their frequency of occurance from the study by Greb et al. ¹

Another dimension of studying music daily use is understanding the listening intent. The intent refers to a specific function of music that the listener is seeking. While the listening intent can be influenced by the situation, it can also be different for the same situation. Understanding the listening intent can help us comprehend the music style/content the listener is seeking. An example of different functions of music reported by listeners in one of the studies ² can be found in the following table. While these studies show the level of complexity in selecting music, it helps to narrow down the problem to tangible factors. For those interested in this line of research, here are some references for further readings ¹ ² ³.

Reported statements of participants on their intent from their favorite music. From the study by Schäfer et al. ²

However, to properly study this influence on the music content, we needed to make progress in information retrieval tools. In the information retrieval domain, researchers have been working on analyzing the music content and finding means to describe and categorize it. One of the prevalent descriptors of music is what is called the valence-arousal scale. This scale enables us to narrow down a “music purpose”, i.e. the function of a music piece, by describing its emotional effect through two dimensions: arousal and valence. Researchers develop intelligent models that can rate a music piece on this scale by analysing its audio content. The figure below shows this valence-arousal space. However, we must remember that this is among the simplest ways of describing music through only two descriptors.

The Valence-Arousal scale. Figure from Du et al. ⁴

More complex methods rely on embedding music, either through content similarity or through frequent co-occurances, into a high-dimensional space. This representation enables us to describe and explore music using more abstract features. While less interpretable, those representation embeddings have proven to be a useful tool to describe the content of the music based on certain criteria. An example of an embedding space projected down to 2-D showing how similar music is clustered together is shown in the figure below. Another way we recently proposed is to learn this representation of the music according to each user independently, by learning how different listeners “use” the music ⁵. Now that we have a proper way to represent and retrieve music through its contents, and have an idea of how listening preferences are influenced by different factors, we can put our findings in use to provide contextual recommendations!

Visualization of music representation and their reported feelings by Cowen et al. ⁶

Finally, in the machine learning domain, researchers have been trying to develop intelligent systems that take all this different factors in consideration and provide relevant and timely recommendations. With such complex requirements, studies have been continuously proposing more sophistcated recommendation systems. Since we do not have space to describe all of the methods, here are some examples of specialized contextual recommenders and generic recommenders. From specialized recommenders, one approach ⁷ uses the emotion descriptors to match music to places-of-interest, which could be of use for a mobile travel guide, for example.

A more complex and generic contextual system uses all data available in an online streaming service to predict the content of a new listening session ⁸. This system considers both the current user situation, through the used device and time of the day, and the user’s listening history and global preferences to retrieve tracks most suitable for the new session. These tracks are represented in an embedding space similar to the one described earlier. These models are trained on millions of data samples and different users to better learn how to model a contextual session for any user. Those more interested in contextual recommendation systems can refer to this detailed review ⁹.

Conclusion

In this article, we explored how one very common problem requires efforts from multiple different domains put together to tackle it. This is multidisciplinary research.

References

F. Greb, W. Schlotz and J. Steffens, “Personal and situational influences on the functions of music listening.” in Psychology of Music 46.6 (2018): 763-794. ↩ ↩² ↩³
A. North and D. Hargreaves, “Situational influences on reported musical preference.” in Psychomusicology: A Journal of Research in Music Cognition 15.1-2 (1996): 30. ↩ ↩² ↩³
T. Schäfer, and P. Sedlmeier, “From the functions of music to music preference.” in Psychology of Music 37.3 (2009): 279-300. ↩
P., Du, X. Li, and Y. Gao. “Dynamic Music emotion recognition based on CNN-BiLSTM.” in IEEE 5th Information Technology and Mechatronics Engineering Conference (ITOEC), IEEE, (2020). ↩
K. Ibrahim et al. “Should we consider the users in contextual music auto-tagging models?” in International Society for Music Information Retrieval Conference (ISMIR), (2020). ↩
A. Cohen et al. “What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures.” in Proceedings of the National Academy of Sciences, 117.4 (2020): 1924-1934. ↩
M. Kaminskas and F. Ricci, “Emotion-based matching of music to places.” in Emotions and Personality in Personalized Services, Springer, Cham, (2016). 287-310. ↩
C. Hansen et al. “Contextual and sequential user embeddings for large-scale music recommendation.” in 14-th ACM Conference on Recommender Systems, (2020). ↩
A. Murciego et al. “Context-Aware Recommender Systems in the Music Domain: A Systematic Literature Review.” in Electronics, 10.13 (2021): 1555. ↩

The Science Behind Music Technology: an animated story

2021-07-28T00:00:00+00:00

Science dissemination

Sharing your research with the rest of the world can be very challenging. Sometimes you may need to target a broader audience than simply the colleagues in your particular research field. Colleagues in other communities or disciplines are already less likely to read about your work. When it comes to sharing your research with the general public, things become even more difficult.

There are several reasons why we all should aim to disseminate our research beyond our universities and scientific communities. For instance, it might be essential to explain your research to a general audience because you are doing it thanks to some public funding. In such a case, it is a social duty to inform the citizens about your findings and make your research comprehensible. It’s a virtuous circle that produces culture and participation, and in return, can pay for new investments in research.

Another reason is to attract the next generation towards science and your specific research field. This is an aspect that is often underrated because it hasn’t an immediate economic and/or social recognition return, but that is critical in the long term. Undergraduate students can orient their education choices and be our future colleagues and enlarge our research community. It’s vital then to let them know that your research exists and might be interesting for them. This would also benefit and increase diversity in the community and reach all those students for whom computer science is not among the options because of societal, demographic, or socioeconomic factors.

In this context, it is still tough for scientists to involve the uninitiated on very specific topics that seem to have almost no connection with their everyday lives. However, many different techniques, tools, and languages have been studied and gradually refined over time. With the increasing amount of information available online, it is becoming more and more important to be concise and attract the audience’s attention from the very beginning. Video might be one of the ways to go.

Number of views per day (log) x video length (min). Plot courtesy of “Communicating Science With YouTube Videos: How Nine Factors Relate to and Affect Video Views” by Velho et al.

Videos about science

Videos about science have become more and more popular over the last decade as they are a low-barrier medium to communicate ideas efficiently and effectively. Short videos from 3 to 5 minutes are ideal because they are long enough to explain a concept and sufficiently short for viewers to decide if they are interested. We all have learned about the advantages and disadvantages of this medium during the last year of the pandemic. The format of the conferences has changed, and video abstracts are now a standard. However, video abstracts are intended for peers and not for a broader audience. When disseminating science, complex concepts should be made accessible for the largest audience possible. In such a case, motion graphics and animated storytelling can be a possible solution. Through the process of abstraction in an animated representation, we can effectively simplify the concept we want to transmit. The style, colour palette, transitions, aesthetic and functional choices can all concur to convey the main message.

Examples of scientific dissemination projects. Image courtesy of Scienseed.

Now, I can’t say this process of abstraction is easy. It takes time, many iterations over the script and many drafts before coming up with something good. You have to learn to work with visual designers who do not know anything about your research. We experienced this when working on the MIP-frontiers video communication project, meant to attract young researchers in our research field. It’s very hard to simplify and abstract things you work on every day. It feels like sacrificing many details which are essential to you for the sake of simplicity. Because of that, you have to always keep in mind who’s your target audience. In the specific case of this video, there was an additional problem: we needed to cover the most possible areas in music information processing (MIP), which was quite hard. The trick we found was to trace back the history of a song that an imaginary inhabitant of the future is listening to. We managed to derive a circular story following the song from composition to recording and from distribution to the user experience. Therefore, the music is the backbone of the video, and its choice was crucial.

Making-of

When preparing a motion graphic, you need to provide to the visual designers a script (description of the scenes), the voiceover (text that an actor needs to read and which describes the scene), and the background music. With those three elements, the visual designers built an animation on which you can then give feedback and adapt the voiceover and the music again. This process is reiterated repeatedly until convergence, when everyone is happy with the result.

Draft of the initial script of the MIP-frontiers animation

In our case, an additional difficulty was that the music wasn’t just some “background” music. It was, on the contrary, the absolute protagonist that mainly contributes to conveying the main message. The music evolves throughout the video and changes according to the MIR application we wanted to illustrate. All of this needs a not negligible effort of synchronization and composition.

Regarding the voiceover, we quickly realized how few words can fit a 3-minutes-long video. More importantly, we learned how hard it can be to summarize the vast diversity of research in our community. Moreover, there are synchronization constraints that impose a fixed number of words to express complex concepts. In the end, we reached a compromise trying to represent as extensively as possible some MIP applications.

Once the voiceover, the animation and the music are done, it is not trivial to create the final video anyway. In fact, in addition to a temporal synchronization of events, automation on the volume of the various instruments and the voice are necessary. This operation is always necessary for video production, and the role of a sound engineer is essential for an optimal result. Especially in this work, where music and its evolving parts are the protagonists, this professional figure had a particularly central role in glueing all the components.

Special Thanks

In general, it was a great experience! I learned a lot, and I’ve spent some time doing something that is not strictly related to my research, but that is a fundamental part of the scientist job. We really thank Mandela (music) and Scienseed (animation) and Alberto Di Carlo (sound engineer) for their great work!

Mandela is an Italian instrumental jazz band from Vicenza. The sound of the band is characterized by a fusion of jazz idioms, rock, world music, psychedelic, and funk. Over the years, the band has performed in several festivals and venues and released 3 full-length albums. These recordings are all available on the major streaming services. Their last release was presented at the festival Rimusicazioni (Bolzano, Italy) and consists of an original soundtrack for “Grass: A Nation's Battle for Life” -- one of the earliest documentaries ever produced (1925).

For this video, the track Simple from the album Mandela s.t. was used. The song was remixed and remastered by Alberto Di Carlo.

Scienseed is a multifunctional agency for the dissemination of scientific findings. Its founding goal is to promote public engagement in science through all available tools in the Era of IT. We are specialized in the translation of scientific data into different accessible products and activities, aimed at either the scientific community (peers) or the general public (society). We provide support to academic laboratories, research institutes, universities and private institutions to raise public awareness and increase the repercussion of their contribution to science.

Segmental DTW

2021-06-24T00:00:00+00:00

From June 6 to 11, I attended the International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2021, which was held online, as a fully virtual conference. ICASSP is a huge conference that presents lots of papers with various topics, and that specifically dedicates sections to Audio and Signal Processing works. In this blog post, I would like to showcase the paper entitled “Segmental DTW: A parallelizable Alternative to Dynamic Time Warping” that has been presented by its author Timothy J. Tsai, a researcher at Harvey Mudd College, Claremont, CA, USA.

This work is in a continuity of several previous works from the author that aim at exploring variants of the standard Dynamic Time Warping (DTW) to propose a variety of applications that cannot be answered with the initial DTW algorithm, such as an offline accompaniment generator from solo recordings for known pieces, or a piano score following video maker. Here, the author proposes a method for parallelizing the initial DTW to speed up the alignment process.

The algorithm

Segmental DTW is a variant of DTW. DTW aims at aligning pairs of sequences. The sequences are time series, that are mostly represented by feature sequences extracted from the audio signal. At first, DTW calculates distances, or costs, between each pair of features. Then, it computes the cumulative costs of each possible alignment with dynamic programming. Finally, it retrieves the optimal alignment in following the path given by the lowest cumulative cost.

The previously described algorithm presents two main limitations. At first, the time complexity of the algorithm is quadratic. It depends directly on the length of each time sequence. Additionally, the algorithm is based on dynamic programming, whose alignment is calculated incrementally from the first to the last frame. Thus, the algorithm is not parallelizable.

In the presented paper, the author aims at proposing a variant of DTW, Segmental DTW, that can quickly compute an alignment that closely matches the baseline DTW alignment. Instead of computing the alignment between two full sequences directly, the proposed algorithm breaks down the problem into subalignments between one full sequence and all non-overlapped segments extracted from the other sequence.

Figure 1: Segmental DTW and its four main steps.

The algorithm is illustrated in Figure 1. The first step is described at the bottom left-hand corner. In this matrix, both time sequences are represented respectively along with the horizontal and vertical axis, the beginning being at the bottom left-hand corner. The vertical sequence is divided into segments of the same length. In the example, four segments are represented, from C₁ to C₄. For each segment, the algorithm applies Subsequence DTW that calculates the cumulative frame-level cost matrix between the segment and each possible subsequence in the vertical axis. The algorithm focuses only on subsequences that have a length in between half and double of the segment length. Musically speaking, this corresponds to tempo variations included between 0.5 and 2 times the reference tempo in the segment. At this point, the minimal value of all cumulative costs present in the last row of the cumulative cost matrix represents the best match between the segment and a subsequence ending at this index. Each cost matrix can be computed independently, i.e. in parallel.

As the second step, the algorithm creates the C_seg matrix in concatenating all lasts rows of previously calculated cost matrices. C_seg represents a segment-level cost matrix. It has the advantage of being smaller than the initial frame-level cost matrix.

The third step aims at connecting the segments between themselves and retrieving the global segment-level optimal path. This is done by applying dynamic programming with two allowable transitions. The first transition (0, 1) allows navigating between optimal subsequences in the horizontal axis for a given segment. Its weight is fixed to 0, which means that we do not accumulate costs in a segment, the goal is to select only one subsequence per segment. The second transition (1, N/(2K)) corresponds to the transition between a selected (the weight is set to 1) subsequence and the starting search point of the next segment. Because we search for subsequences that have a minimum distance of half of the segment length, we start the search from a length corresponding to N/(2K), where N/K is the length of the segment for an initial vertical sequence of length N and a fixed number of segments K. This strategy aims at finding the optimal path through C_seg. The path is composed of one element per row. The elements are in increasing order and at least separated by a minimal distance of N/(2K) in the vertical axis.

The fourth and last step retrieves the frame-level optimal path for the segment-level path. Each selected element in C_seg indicates the final index of the corresponding subsequence. This is done by backtracing each cumulative cost matrix from the selected element. This step is also computed in parallel.

The resulting path given by Segmental DTW can be discontinuous at segment borders. The author investigates the restriction to be strictly monotonic with the help of T_seg, the matrix shown at the top left-hand corner of Figure 1. In this matrix, each row corresponds to the starting indexes of all optimal subsequences in the horizontal sequence ending at the corresponding index. Thus, this is computed in backtracing each cumulative cost matrix of each segment from every possible ending location. This extension requires lots of additional computation and from experiments, the resulting optimal path is less accurate than the previous method. Thus, this method is no more detailed here.

Accuracy and speed

The following experiments are evaluated on the Chopin Mazurka dataset. It gathers pairs of audio recordings annotated at the beat level.

Segmental DTW and DTW do not give the same alignment path. This is due to the allowed discontinuities at segment boundaries. The more segments there are, the more different the paths are. In Figure 2, the author compares the error rate given by DTW and Segmental DTW with different numbers of segments (from 2 to 32). DTW can be seen as a special case of Segmental DTW where the number of segments is set to 1. For each error tolerance, the conclusions are identical: a small number of segments gives an error close to DTW whereas a large number of segments gives a larger error rate.

Figure 2: Error rate of DTW and Segmental DTW for different numbers of segments, evaluated under different tolerances.

Segmental DTW adds additional steps to compute the alignment path. Even if steps 1 and 4 are parallelizable, an interesting question would be to know, if all the steps require more runtimes if they are computed on a single thread. Figure 3 shows that, for all cost matrices from size 1k (i.e. 1000x1000) to 50k (i.e. 50kx50k), DTW and SegmentalDTW require the same computation time. Thus, each parallelizable step is directly saving time.

Figure 3: Error rate of DTW and Segmental DTW for different numbers of segments, evaluated under different tolerances.

From the previous experiment, the next question is to know what is the percentage of runtime that can be parallelized. Figure 4 shows that, for cost matrices bigger than 5k and in using 32 segments, more than 99% of the runtime is parallelizable. This experiment shows that the proposed algorithm shorten drastically the alignment computation time.

Figure 4: Breakdown of runtime by component. The steps Seg DP and Seg Back are not parallelizable.

Conclusion and insights

The author shows through his work a possible alternative to DTW that can parallelize almost all of its runtime without losing too much accuracy. The method is particularly interesting because it can also answer different problems that are well-known in the Music Information Retrieval community. Last year, the strategy to segment one of the sequences was already used to tackle audio-to-score alignments where repeats and jumps appear at unknown locations. Presented at ISMIR 2020 under the name Hierarchical DTW, the algorithm considers each score line as a segment, searches for the most probable subsequence in the audio recording for each score line, and finally connects the segments to output the optimal alignment path.

The idea of applying Subsequence DTW between an audio segment and a whole recording inspired my research in real-time Opera tracking. The task is challenging because the tracker has to be robust to deviations from the score, acting, or unknown skip and repeat sections that tend to interfere with the tracking process. Computing Subsequence DTW between segments received in real-time and the whole score permits the tracker to be sensitive to sudden score jumps or to redirect the tracker if it gets lost.

I will be looking forward to the next contributions of the author!

Lyrics Transcription & Alignment

2020-12-11T00:00:00+00:00

A few weeks ago, I attended the International Society for Music Information Retrieval (ISMIR) 2020 Conference, which was held online in the format of a virtual conference. In this blog post, I write about few papers from the conference and give a brief summary of what I found interesting related to my research interests: lyrics transcription and alignment, vocal extraction, and singing voice.

Multilingual Lyrics-to-audio Alignment

Andrea Vaglio, Romain Hennequin, Manuel Moussallam, Gael Richard, Florence d’Alché-Buc

Summary:

This paper presents a novel scheme for audio-to-lyrics alignment in a multilingual setting using BiLSTMs trained on CTC loss and phonemes as the intermediate representation for lyrics.

Pros:

One of the first and most successful multilingual lyrics alignment works.
Evaluation and training on public datasets.
New subsets of the DALI dataset are released.
The system works on languages with limited resources.

Cons:

The authors evaluate on a set with automatically aligned lyrics. Even though the authors mentioned in Q&A sessions that they think these annotations are of good quality by inspection, the word boundaries should be manually annotated for evaluation, as the precision of the currently best alignment systems is around 100ms, it would be difficult to judge annotations by inspection.

A Deep Learning Based Analysis-synthesis Framework for Unison Singing

Pritish Chandna, Helena Cuesta, Emilia Gomez

Summary:

This study applies an extensive computational analysis on choir recordings from the Soprano-Tenor-Alto-Bass (SATB) approach. Moreover, the authors use a novel approach for the retrieval of separate vocal tracks from a mixture of a choir through singing voice synthesis. The authors use singer identity and singer independent linguistic embeddings encoded via neural networks and use this representation for the synthesis. The quality of synthesis is reported through subjective listening tests. Some audio examples are also provided at the following link.

Pros:

One of the first and few studies on a research topic that has been rarely tackled upon within the MIR community, which is choir singing.
Reproducibility; open-source datasets for analysis and evaluation, online demo, and code are made publicly available. Something we are used (and appreciate) to see from the works of MTG.
The online audio examples are convincing by inspection. I anticipate that they would function well for further singing analysis.

Cons:

I guess objective evaluation metrics are necessary at some point for a more scientific comparison with already existing or future systems.

Content Based Singing Voice Source Separation via Strong Conditioning Using Aligned Phonemes

Gabriel Meseguer Brocal, Geoffroy Peeters

Summary:

Here is another paper that applies text informed music source separation. Instead of following the naive approach of feature concatenation, the authors have tried multiple ways to incorporate the phonetic information from input lyrics. The words in lyrics are decomposed into their phonetic representations via a pronunciation dictionary. For each word aligned with the audio signal, a global phoneme activation matrix is created on the word-level (see figure). For conditioning the network, this phoneme matrix is embedded in the “Feature-wise Linear Modulation” layers in the C-U-Net architecture the authors employ. Even though the improvement is marginal compared to the unconditioned network, the idea is novel and promising to be investigated more extensively.

Pros:

A new multitrack dataset for music source separation with aligned lyrics. We always love new annotated data!
A novel approach to condition text for source separation. There is definitely room for improvement.
High quality of the paper presentation

Cons:

Even though the authors report statistical significance, the improvement is still marginal.
There should be better ways to incorporate phonetic information. In this paper’s approach, the phonemes are not extracted at the frame-level, but at the word-level. Further improvement might be possible if phonemes were aligned with the audio signal on frame-level.
Also, the phonetic information is represented as word-level global binary activation matrices. Perhaps it would lead to better results if phonetic information was encoded as a probability distribution. This would circumvent the need for lyrics as the input for separation.

Exploring Aligned Lyrics-informed Singing Voice Separation

Chang-Bin Jeon, Hyeong-Seok Choi, Kyogu Lee

Summary:

This paper proposes to use phonetic information for source separation and investigates if it does improve the performance. It has lots of similarities with the works by Kilian Schulze-Forster. The phonetic features are extracted from lyrics text only.

Pros:

The authors propose two methods to incorporate phonetic information into vocal source separation: Local conditioning and feature concatenation.
I do think that incorporating phonetic information for source separation should work and is a nice idea.

Cons:

Private dataset! I know it is extremely difficult to find well-curated public datasets for MIR tasks, however, there are some datasets for source separation that people have been using. I would expect to see at least one public dataset in the evaluation. (Notice to peer-reviewers!)
The work is not extremely different comparing to Kilian’s work. Even though consistent improvement is observed, it is not significant. Perhaps there are other ways to leverage phonetic information.

Automatic Rank-ordering of Singing Vocals with Twin-neural Network

Chitralekha Gupta, Lin Huang, Haizhou Li

Summary:

In this paper, the authors present a purely engineered way of measuring the singing quality using music-informed features and a Siamese network, which learns to choose the better singer from two performances.

Pros:

A nice Deep Learning-based method for evaluating the singing quality.
The authors use public datasets.

Cons:

The authors tackle a problem that has been widely discussed by musicologists: “What are the objective measures for singing quality?”. Though there has not been a common consensus among singing voice researchers, educators, and performers, the authors claim that there is, referring to only two recent MIR papers. I think some of their claims are too bold to be scientific. ISMIR reviewers should also think musicologically as much as they consider the details from an engineering perspective.
For the ground truth labels of their first dataset, the authors use crowd-sourced singing quality rankings of the users of a karaoke app (Smule). However, we do not necessarily know that the people involved in voting have enough expertise in singing to be able to judge other performers. I guess the term “ground truth” should be revised.

A Chorus-section Detection Method for Lyrics Text

Kento Watanabe, Masataka Goto

Summary:

This study proposes a sequence labeling method to be used to detect the chorus sections of lyrics of songs using only the text as input. The authors train a BiLSTM network on a large but private dataset with semi-automatically generated chorus-line annotations.

Pros:

This is the first work on an important computational problem within MIR. Such a robust, generalizable, and scalable model would have a range of applications.
It shows an example case of an NLP application in MIR.
There are several open-source tools revealed in the paper.
The list of features used for this task makes almost complete sense and combining them for sequence labeling is novel.

Cons:

Both training and evaluation data are private and not accessible by other researchers. The scientific approach of the 21st century requires open science for fairness, transparency, and reproducibility of published results.
According to the authors, BiLSTMs against MLPs or heuristic-based models are beneficial for sequence prediction, which they state as a contribution of the paper. Well, we knew this already which was shown in several dozens of peer-review papers.
Among the features used for the task, the only thing that I did not get is the motivation of using “lyrics line syllable count”. Also, the choice of some features could be better justified.

(Notice: The figures in this post are taken from the related papers including their captions to make them more explanatory. Please don’t pay attention to the figure number - this is not in order)

Score following in sheet images

2020-12-02T00:00:00+00:00

This year saw multiple international conferences go virtual, smitten by the COVID-bug. ISMIR 2020 was no exception to this trend, and while we were disappointed to be unable to meet physically in Montréal, Canada; the organizers did a great job with the virtual format. I attended ISMIR 2020, the 21^st International Society for Music Information Retrieval Conference, from the 11^th to the 16^th of October, 2020. I was delighted to find some fascinating papers related to my current research. This blog post summarizes one such paper that piqued my interest. The paper is titled “Learning to read and follow music in complete score sheet images”; and the authors are Florian Henkel, Rainer Kelz, and Gerhard Widmer, from JKU Linz, Austria.

Score following in raw sheet images

The work presented in this paper aims at score following in sheet music that is given as unprocessed images. The general goal of score following is to determine the current position in the score for an incoming music performance. The authors build upon their previous work presented in ISMIR 2019¹ and extend the capabilities of the score following model. The novelty of this work lies in the fact that they deal directly with complete raw images of sheet music (although this is done one page at a time). Existing methods for score following either rely on computer-readable representations of the score (such as MIDI or MusicXML), or work with small snippets of the sheet images. While the former representation is not readily available and/or depends on robust OMR, the latter is unable to capture context and disambiguate between different possible matching positions. It also has another constraint that the sheet image excerpt has to correspond partially to the incoming audio excerpt, in the absence of which the tracker is lost and subsequent predictions are incorrect. Additionally, the methods that work with snippets of audio and sheet image need an “unrolled” score representation, i.e. the staves have to be cut out from the sheet image and stitched together in a sequence.

The authors attempt to overcome this limitation by working with entire pages of sheet music images. They extend the work initially proposed in ¹ to be able to handle the temporal aspect of score following, i.e. disambiguate multiple positions in the sheet image that match with the audio excerpt. Another recent work has shown the effectiveness frame similarity learning for audio-to-score alignment ², a task related to score following.

A brief detour: Referring image segmentation

The basic idea is motivated by “referring image segmentation” ³, a task in computer vision that aims to identify a region in an image, given a language expression as the query. To better understand this task, take a look at this figure:

Figure from the paper "Referring Image Segmentation by Generative Adversarial Learning"

As the figure suggests, given the input images (left) and the corresponding referring expressions (below), the goal of referring image segmentation is to segment out the regions in the image referred to by the expression (right).

Back to score following

The idea discussed above is carried over to the score following task by modelling it as the following question: “Given the incoming audio, which position in the sheet image does it correspond to?”. In other words, the authors treat the incoming audio as the language expression, to extract the region from the sheet image that acts as the entity to reason about.

Specifically, they predict a binary segmentation mask that gives for each pixel in the sheet image page, the probability of it corresponding to the current audio. The authors employ a U-net based architecture for this task, illustrated in the figure below:

The U-net architecture employed in the paper

A feature-wise linear modulation layer ⁴ is also applied before the last activation function, to directly interfere with the learned representation of the sheet image by modulating its feature maps. This helps the CNN to focus on the parts required for segmentation. The FiLM layer is illustrated below:

The FiLM layer used before the last activation

The results are reported on the MSMD data set, a synthetic dataset containing polyphonic piano music. The authors report that their method outperforms previous methods that are based on snippets of sheet music matched to audio excerpts, as well as an OMR based DTW method. A noteworthy point is that pieces containing several pages of sheet music are split into multiple pieces to aid training. Perhaps future work could deal with entire pieces in one go. All in all, this is a promising paper furthering the research on score following. Some limitations of their method are the inability to handle structural differences like repeats, the implications of the size of the segmentation mask on the granularity of alignment, and generalizing to real recordings.

References

Henkel, Florian, Rainer Kelz, and Gerhard Widmer. “Audio-Conditioned U-Net for Position Estimation in Full Sheet Images.” arXiv preprint arXiv:1910.07254 (2019). ↩ ↩²
Agrawal, Ruchit, and Simon Dixon. “Learning Frame Similarity using Siamese networks for Audio-to-Score Alignment.” 28th European Signal Processing Conference (EUSIPCO 2020) ↩
Liu, Chenxi, et al. “Recurrent multimodal interaction for referring image segmentation.” Proceedings of the IEEE International Conference on Computer Vision. 2017. ↩
Perez, Ethan, et al. “FiLM: Visual Reasoning with a General Conditioning Layer.” AAAI. 2018. ↩

Music Source Separation and Transcription. Can they be done at the same time?

2020-11-11T00:00:00+00:00

From 4^th to 8^th of May 2020, I attended the ICASSP 2020, short for International Conference on Acoustics, Speech and Signal Processing. Even though it was planned to be held in Barcelona, Spain, the conference moved to a fully virtual format due to the recent COVID-19 induced lockdowns around the world. While this reduced the networking aspect of the event, the fully online conference allowed a higher number of participants, and the free registration for non-authors also facilitated the participation of machine learning enthusiasts worldwide.

Apart from the fantastic tutorials and great keynote talks given by academic and industry researchers from all over the world, the scientific programme included many interesting papers. One of them immediately caught my attention due to the novelty and relation to my own PhD topic. The work I am talking about is Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instruments, by Ethan Manilow, Prem Seetharaman and Bryan Pardo, where a multi-task system capable of performing music source separation and transcription simultaneously is proposed.

In this post I would like to talk a little bit more about this paper, summarizing its leading ideas and explaining in my own (and hopefully easier) words the contributions and achievements made by the authors.

Music Source Separation and Transcription

As most of you already know, Music Information Retrieval (MIR) researchers call Music Source Separation the task of separating an audio signal into its constituting sources. The definition of sources, though, vary in the specialised literature, allowing us to divide source separation problems into multiple categories depending on the researcher’s interest. Some examples are source separation algorithms to separate the signal into each constituent isolated single-instrument recording, methods interested in separating only one of the instruments or the vocal track from the “rest”, or even methods to separate harmonic instruments from percussive ones. In this paper, the authors propose a system of the first category. More specifically, their system can separate mixtures with up to 5 instruments into its constituent single-instrument recordings. The instruments they use are piano, guitar, drums, bass and strings. Recently, music source separation has significantly advanced, with many recent deep neural network architectures achieving impressive results for the task. Furthermore, music source separation is a crucial MIR task because it enables other downstream tasks, such as analysis of each separated track, remixing of audio recordings, musical rhythm analysis or beat-tracking, among many others.

Similarly, the task of Automatic Music Transcription challenges MIR researchers to generate any type of notation for the music signal automatically. The type of target notation also varies in the literature, it can be a MIDI piano roll, the score of a particular instrument in the audio signal, or even the full score for the whole piece. In this project, the authors estimate MIDI piano rolls for each source used in the separation task.

Both the tasks have been historically considered the holy grails in the MIR community due to the high level of difficulty involved. In the presence of polyphonic instruments, where multiple notes can be played simultaneously by each instrument, the challenges are even more complicated due to the overlap of the harmonics and of the timbre features in the time-frequency representation of the music signal.

The Cerberus Network

The main contribution of this work is the fact that instead of performing the separation and the transcription independently or performing the transcription as a downstream task after source separation, the authors proposed to jointly perform both tasks using a single neural network in a multi-task scenario. The authors are able to leverage the similarities between those two tasks and use a shared latent representation, so both of them can be performed simultaneously.

The paper proposes a deep neural network architecture built upon the Chimera Network¹, which was previously proposed exclusively for source separation. Originally, in the Chimera network, a mixture magnitude spectrogram of size $T\times F$ is used as input to a stack of Bidirectional Long Short-Term Memories (LSTMs) and 2 outputs are estimated. The first is the typical output of mask-based source separation algorithms: a mask with the same dimension as the input, whose element-wise multiplication by the original mixture magnitude spectrogram recovers the magnitude spectrogram of the target source. The other output is an embedding space of dimension $TF\times 20$, where every row represents a projection of a time-frequency bin of the mixture spectrogram into a 20-dimensional space. The projection is such that bins that primarily contain energy from the same source are projected closer to each other while bins belonging to different sources are located far apart. This embedding technique is called Deep Clustering² and can be seen as a regularisation technique for the source separation in the Chimera architecture.

In this work, the authors extend this architecture by adding a new output to it. They named it Cerberus Network in an allusion to the three-headed guardian dog from the Greek mythology. In the Cerberus network, the new output is the direct piano roll of each of the sources of the mixture. The piano roll is trained using $L2$-distance from the original piano roll of each instrument using a total of 88 possible MIDI notes. In summary, the final loss of the proposed architecture is a linear combination of the Deep Clustering (DC), Mask Inference (MI) and Transcription (TR) losses such as:

$$\mathcal{L} = \alpha\mathcal{L}_{\mathrm{DC}} + \beta\mathcal{L}_{\mathrm{MI}} + \gamma\mathcal{L}_{\mathrm{TR}}$$

The figure below shows an overview of the proposed system. Note the 3 heads (outputs) in the network. Two are responsible for the two tasks mentioned above, and the other is treated as a regularisation to improve the network’s performance.

Overview of the Cerberus multi-task system³.

Experiments and Results

The authors trained their system using the Slack2100 dataset⁴ by creating 4 sets of synthesized mixtures. The instrument combinations for the four sets were piano + guitar (set 1), piano + guitar + bass (set 2), and piano + guitar + bass + drums (set 3), and piano + guitar + bass + drums + strings (set 4). Moreover, they also implemented a version of a (piano + guitar) model trained and evaluated on real-world recordings audio of isolated notes from those two instruments by creating incoherent (random) mixtures of data from MAPS⁵ and GuitarSet⁶ datasets. The authors compared the Cerberus network with all the combinations of models using just one or two of the three-loss terms from Eq 1. They show that the Cerberus network is able to maintain the separation performance of the Chimera network and improve the transcription F1-measure by up to 4% if compared to a model trained only with the transcription output using $\mathcal{L}_{\mathrm{TR}}$.

In the project’s official website you can find not only the original paper but also listen to audio samples of estimated by their system and check the complete evaluation for the music source separation and transcription.

Conclusion

The authors showed that it is possible to jointly learn how to separate and transcribe up to five instruments with a single network. Their multi-task system was able to leverage a shared latent representation by exploiting the fact that separation and transcription are complementary related tasks.

From my point of view, I believe that their approach is interesting and shows the power of multi-task systems and how each task can help in solving other related tasks. This paper is definitely worth a full read. Keep posted for the next blog post!

References

Y. Luo, Z. Chen, J. Hershey, J. Le Roux and N. Mesgarani “Deep Clustering and Conventional Networks for Music Separation: Stronger Together” In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE (2017) New Orleans, USA https://doi.org/10.1109/ICASSP.2017.7952118 ↩
J. Hershey, Z. Chen, J. Le Roux and S. Watanabe “Deep Clustering: Discriminative Embeddings for Segmentation and Separation” In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE (2016) Shangai, China https://doi.org/10.1109/ICASSP.2016.7471631 ↩
E. Manilow, P. Seetharaman and B. Pardo “Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instruments” In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE, (2020)Barcelona, Spain https://doi.org/10.1109/ICASSP40776.2020.9054340 ↩
E. Manilow, G. Wichern, P. Seetharaman and J. Le Roux “Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity” In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), (2019) New Paltz, USA https://doi.org/10.1109/WASPAA.2019.8937170 ↩
V. Emiya, R. Badeau and B. David, “Multipitch Estimation of Piano Sounds Using a New Probabilistic Spectral Smoothness Principle” In IEEE Transactions on Audio, Speech, and Language Processing vol. 18 no. 6, (2009) pp. 1643–1654 https://doi.org/10.1109/TASL.2009.2038819 ↩
Q. Xi, R. Bittner, J. Pauwels, X. Ye and J. P. Bello “Guitarset: A Dataset for Guitar Transcription” In International Society for Music Information Retrieval (ISMIR) (2018) Paris, France https://doi.org/10.5281/zenodo.1492449 ↩

AI Carbon Footprint

2020-11-10T00:00:00+00:00

The topic of this post is the carbon footprint of Artificial Intelligence (AI). I thought this was a relevant topic to discuss as we, as researchers, are part of the emission process, even if we are mostly unaware of it. I believe it is ethically essential to be aware of the consequences that can derive from training our large deep learning models and discuss guidelines and best practices for our research community.

Climate Change

First, let us recall the consequences of climate change, even if they are under our eyes every day. The United Nations declared climate change a “defining crisis of our time”¹, and most climate scientists agree that human activity is its main driver². However, the effects of climate change are not only the rise of temperatures and the melting of glaciers, but also:

food and water emergency due to soil degradation and rise of temperatures;
extreme meteorological events, such as, heatwaves, hurricanes, which are becoming more frequent and intense. 90% of them are classified as weather- and climate-related.

This creates competition for land, food, and water, leading to socio-economic tensions and mass displacement¹. More than 140 million people, mostly from the south of the world, will be forced to migrate by 2050¹. According to a Stanford University research³, the economic gap between the world’s richest and poorest countries is 25% larger today than it would have been without global warming.

Urgent actions are needed to avoid an environmental catastrophe, including reducing emissions to zero by the middle of the twenty-first century and limiting the average global warming to 1.5 °C¹. Technology has, of course, a central role in fighting this crisis.

The Role of AI

In this light, it has become an urgent matter to consider the dual role of the AI technology in the crisis. On one hand, it can help reduce climate change effects by, for example, modelling and developing solutions to avoid it. On the other hand, the AI is itself a significant emitter of carbon.

Below you can see a visual map extracted from an essay of 2018 entitled Anatomy of an AI system⁴ that shows the impact of an AI device on a global scale. The authors took as an example Amazon Echo and followed its impact from manufacturing to usage. They analysed its impact in terms of social work, data, and resources required during its lifespan.

Map extracted from Anatomy of an AI system⁴

In the whole map, the location where we, as researchers, come into the game and play an active role is the small section below, related to training an AI system and eventually preparing and labelling data.

Details of the map extracted from Anatomy of an AI system⁴

Red vs Green AI

In 2019, a team from MIT analysed some Natural Language Processing (NLP) models available online⁵. The researchers estimated the energy consumption (in kilowatts) required to train them and converted the numbers into approximate carbon emissions and electricity costs. They estimated that the carbon footprint of training a single big language model is equal to around 300,000 kg of carbon dioxide emissions, which is orders of magnitude higher than other familiar consumptions. Furthermore, the authors also quantified the computational cost of Research and Development (R&D) for a new NLP model. This cost is the one we, as researchers and engineers, should primarily consider, as it reflects the actual carbon footprint of a project. In the figure below, you can see the details of the estimated CO$_2$ emissions from training standard NLP models compared to everyday consumption.

Tables of carbon dioxide emission. Extracted from ⁵

In a paper from 2019, Roy Schwartz and collaborators observed that a linear gain in performance requires an exponentially larger model leading to substantial carbon emissions⁶. They called this trend ‘red AI’, that is, ‘buying’ better results using massive computing. Their study enumerated three factors making AI research red:

the cost of running the model on a single example;
the training dataset size, which controls the number of times the model is run;
the number of hyperparameters, which controls how many computations are necessary to train the model.

They analysed papers from top conferences and observed that the majority prioritised accuracy over efficiency (90% from the Computational Linguistics Conference 2018, 80% from NeurIPS 2018, and 75% from the Conference on Computer Vision and Pattern Recognition 2019). In the same paper, the authors also defined the term ‘green AI’ as “AI research that yields novel results without increasing or, ideally reducing, computational cost”. Green AI considers efficiency as a critical evaluation criterion and is considered an opposite term to red AI. Ideally, this type of research would level the possibilities of academia versus big tech companies, whose research is often facilitated by impressive computational resources.

A virtuous example in audio research is SuDoRM-RF⁷, a novel deep architecture for efficient universal sound source separation. It extracts multi-resolution temporal features through successive depth-wise convolutional down-sampling and aggregates them using a nonparametric interpolation scheme. This way, the authors can significantly reduce the required number of layers while still effectively capturing long-term temporal dependencies. The proposed model performs similarly or even better than state-of-the-art models while requiring significantly less computational resources in terms of the number of trainable parameters, number of floating-point operations, memory allocation, and time.

Separation performances alongside their computational requirements for performing inference on CPU (I) and a backward update step on GPU (B) for one-second long audio input or the equivalent of 8000 samples⁷.

Even if this is a virtuous example, there is no estimate of carbon emissions and energy consumption. Mainly, the problem is the absence of a standard of measurement and the intrinsic difficulties in measuring it.

The emissions are related to the training server’s location and the energy grid it uses, the training procedure’s duration and the training’s hardware. Thus, it is difficult to measure and requires specific knowledge. Therefore, we can take advantage of libraries and tools that do the work for us, for example:

Quantify the Energy Accounting

The consumed energy consists of the amount of energy needed to power the computational system and it is measured in Joules (J) or Watt-hours (Wh). It is given mostly by the cooling of the system and by the server/storage power consumption⁹:

cooling (50%)
lighting (3%)
power conversion (11%)
network hardware (10%)
server/storage (26%)

We can further break down the server and storage component into DRAM, CPUs, and GPUs’ contributions. Accurate accounting for all these components requires complex modelling and varies depending on workload. Most carbon/energy trackers consider DRAM/CPUs/GPUs consumption and account for the other components through the PUE (Power Usage Effectiveness) factor⁹. This factor rescales the power metrics by an average projected overhead of different elements. This way, one can evaluate the impact of her/his experiment only without considering background processes.

Carbon emissions are typically measured in CO$_{2}$eq, which is the amount of carbon dioxide released into the atmosphere due to the project⁹. Sometimes (especially in regulations), it is considered the financial impacts through carbon’s social cost (SC-CO$_2$), which measures the long-term damage done by CO$_2$ ⁹. To measure it, one should use the per-country social cost of carbon, which accounts for the risk profiles of different countries.

We can estimate carbon emissions by understanding the local energy grid’s carbon intensity and the system’s energy consumption⁹. The carbon intensity corresponds to the grams of CO$_{2}$eq emitted per kWh of energy used and is determined by the energy sources supplying the grid:

coal power: 820 gCO$_{2}$eq / kWh
hydro-electricity: 24 gCO$_{2}$eq / kWh

Thus, running our job in countries where the energy supply is green can be crucial. It is not necessary to eliminate computation-heavy models, as shifting training resources to low carbon regions can immediately reduce carbon emissions with little impact on us⁹. (e.g., training the same model in Quebec rather than in Lettonia can reduce CO$_{2}$eq by 30 times!)

Guidelines

In conclusion, we, as researchers, should think about:

report training time, computational resources, and sensitivity to hyperparameters;
make a cost-benefit (accuracy) analysis of our models;
prioritise computationally efficient hardware and algorithms;
quantify energy consumption and carbon emission;
move jobs to low carbon regions;

Check out these excellent initiatives:

Looking forward to seeing something similar in the MIR community!

References:

The United Nations (UN), “The Climate Crisis – A Race We Can Win” In https://www.un.org/en/un75/climate-crisis-race-we-can-win ↩ ↩² ↩³ ↩⁴
“The Climate Reality Project” In https://www.climaterealityproject.org/ ↩
J. Garthwaite, “Climate change has worsened global economic inequality” In Stanford Earth (2019) https://earth.stanford.edu/news/climate-change-has-worsened-global-economic-inequality#gs.g3u2y5 ↩
K. Crawford and V. Joler, “Anatomy of an AI System: The Amazon Echo As An Anatomical Map of Human Labor, Data and Planetary Resources,” In AI Now Institute and Share Lab (2018) https://anatomyof.ai/ ↩ ↩² ↩³
E. Strubell, A. Ganesh and A. McCallum, “Energy and Policy Considerations for Deep Learning in NLP” In ArXiv Preprint (2019) https://arxiv.org/abs/1906.02243 ↩ ↩²
R. Schwartz et al. “Green AI.” In ArXiv Preprint (2019) https://arxiv.org/pdf/1907.10597.pdf ↩
Tzinis, Efthymios, Zhepei Wang, and Paris Smaragdis. “SuDo RM-RF: Efficient Networks for Universal Audio Source Separation.” In IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP) IEEE, (2020) https://arxiv.org/pdf/2007.06833.pdf ↩ ↩²
A. Lacoste et al. “Quantifying the carbon emissions of machine learning” In ArXiv Preprint arXiv:1910.09700 (2019) https://arxiv.org/pdf/1910.09700.pdf ↩
P. Henderson et al. “Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning.” In ArXiv Preprint (2020) https://arxiv.org/pdf/2002.05651.pdf ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
L. F. W. Anthony, B. Kanding, and R. Selvan “Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models.” In ArXiv Preprint (2020) https://arxiv.org/pdf/2007.03051.pdf ↩

Groove Toolbox

2020-11-06T00:00:00+00:00

Here we take a look at a lovely paper by Fred Bruford et al. that presents and evaluates a set of engineered features for drum pattern analysis in the symbolic domain¹.

It gathers both new and pre-existing features in literature, grouped in rhythm features, similarity metrics, and microtiming features. The final toolbox, written in Python, is evaluated on drum loop similarity perception experiments, where the authors are able to find significant correlations between feature values and similarity scores.

Toolbox

The Groove Toolbox is a comprehensive framework for modelling perceptual qualities of drum loops in the symbolic domain. It builds on previous research on drum loop analysis and, while not studying data-driven approaches yet, it provides a cohesive and interpretable set of perceptually motivated features.

The authors decided to implement those pre-existing algorithms that were proven to be perceptually relevant and that could handle different onset velocities. They also introduced four microtiming features and two similarity models. These similarity models are the fuzzy hamming distance, a variant of the hamming distance that accounts for minimal onset deviations, and a new structural similarity metric based on a transformation model² that removes ghost notes and ornamentations to facilitate modelling.

List of features and similarity measures currently implemented in the GrooveToolbox. New features are in bold.

Evaluation

The evaluation was carried out using a study derived from a listener study on drum loop similarity³. That study used a set of drum loop templates from FXpansion’s BFD3, whose raw (unquantised) information allows the use of microtiming features. It collected similarity scores from 21 participants on 80 pairs of drum loops.

After the correlation study, the authors found that drum loop similarity models built upon similarity metrics (baseline) performed best when combined with rhythm and microtiming features. Also, the new structural similarity metric alone (r=0.65, p=6.1e-11) performs better than the standard Hamming distance (r=0.59, p=9.7e-9). The fuzzy hamming distance, on the other hand, did not perform better than the standard one (r=0.56, p=6.1e-8).

Model performance as R-squared value for rhythm R feature set, microtiming MT feature set, structural similarity feature SS and all three combined for each participant. The best performance is found when the three are combined.

Relevance

Extracting features from drum loops is an important yet relatively underexplored topic in MIR. Today, practically all commercial drum libraries include loop templates ready for the user to use and edit. There also exist datasets with a large number of these templates like the Groove MIDI Dataset⁴, ready to be imported to virtual instruments.

The features in this toolbox provide a quick and easy way to query drum loops that are perceptually similar to a target loop without one needing to explore the dataset manually. And, of course, this target loop could also be an automatic transcription of a drum performance or even a vocal imitation (beatboxing) performance. Such systems would allow music producers to quickly explore alternative drum loops for their compositions in an efficient and reliable way.

References

Bruford, F., Lartillot, O., McDonald, S., & Sandler, M. “Multidimensional similarity modelling of complex drum loops using the GrooveToolbox”, In Proceedings of the 21st Conference of the International Society for Music Information Retrieval (ISMIR), 2020. ↩
Sioros, G., Davies, M. E., & Guedes, C. “A generative model for the characterization of musical rhythms.”, Journal of New Music Research, 2018 ↩
Bruford, F., Barthet, M., McDonald, S., & Sandler, M. “Modelling Musical Similarity for Drum Patterns: A Perceptual Evaluation.” In Proceedings of the 14th International Audio Mostly Conference: A Journey in Sound, 2019. ↩
Gillick, J., Roberts, A., Engel, J., Eck, D., & Bamman, D. “Learning to groove with inverse sequence transformations.”, arXiv preprint arXiv:1905.06118, 2019. ↩

MIP-Frontiers

Future steps in version identification

1. Strategies for short-term gains

1.1 Incorporating lyrics into the workflow

1.2 Using automatically generated tags/metadata

1.3 Scalability-related issues

1.4 Handling short queries

1.5 Postprocessing operations

2. Issues to address for medium-term benefits

2.1 Definition of musical versions and musical work

2.2 Evaluation metrics

2.3 Genre distributions

MIP-Frontiers Final Workshop

Nature and Purpose of the Workshop

Thursday 14th October 2021

Friday 15th October 2021

Keynote Speakers

Augusto Sarti: Applications of Machine Intelligence in Computational Acoustics

Rachel Bittner: Source Separation Metrics: What are they really capturing?

Kazuyoshi Yoshii: Generative Modeling Meets Deep Learning: A Modern Statistical Approach to Music Signal

Fellows Presentations

Furkan Yesiler: Data-Driven Musical Version Identification: Accuracy, Scalability, and Bias Perspectives

Carlos Lordelo: Deep Learning Methods for Musical Instrument Separation and Recognition

Kilian Schulze-Forster: Informed Audio Source Separation with Deep Learning in Limited Data Settings

Emir Demirel: Deep Neural Networks for Automatic Lyrics Transcription

Alejandro Delgado: Data-Driven Approaches for Query by Vocal Percussion

Ruchit Agrawal: Towards Neural Context-Aware Performance-Score Synchronization

Charles Brazier: Autonomous and Robust Live Tracking of Complete Opera Performances

Luís Carvalho: Large-Scale Multi-Modal Music Search and Retrieval without Symbolic Representation

Vinod Subramanian: Investigating the Behaviour of Audio Classification Models through Adversarial Attacks and Gradient based Interpretability Methods

Philip Tovstogan: Exploration of Music Collections with Audio Embeddings

Karim Ibrahim: Audio Auto-tagging as Proxy for Contextual Music Recommendation

Ondřej Cífka: Deep Learning Methods for Music Style Transfer

Javier Nistal: Exploring Generative Adversarial Networks for Controllable Musical Audio Synthesis

Giorgia Cantisani: Neuro-Steered Music Source Separation

António Ramires: Automatic Characterization and Generation of Music Loops and Instrument Samples for Electronic Music Production

How Multidisciplinary Researchers Come Together to Solve a Problem

Crossroad of domains

Conclusion

References

The Science Behind Music Technology: an animated story

Science dissemination

Videos about science

Making-of

Special Thanks

Segmental DTW

The algorithm

Accuracy and speed

Conclusion and insights

Lyrics Transcription & Alignment

Multilingual Lyrics-to-audio Alignment

Summary:

Pros:

Cons:

A Deep Learning Based Analysis-synthesis Framework for Unison Singing

Summary:

Pros:

Cons:

Content Based Singing Voice Source Separation via Strong Conditioning Using Aligned Phonemes

Summary:

Pros:

Cons:

Exploring Aligned Lyrics-informed Singing Voice Separation

Summary:

Pros:

Cons:

Automatic Rank-ordering of Singing Vocals with Twin-neural Network

Summary:

Pros:

Cons:

A Chorus-section Detection Method for Lyrics Text

Summary:

Pros:

Cons:

Score following in sheet images

Score following in raw sheet images

A brief detour: Referring image segmentation

Back to score following

References

Music Source Separation and Transcription. Can they be done at the same time?