Experiment Reproducibility of A Holistic Approach To Polyphonic
Music Transcription With Neural Networks
Alec Burge Isaac Hong
Music Technology Music Technology
Georgia Inst. of Tech. Georgia Inst. of Tech.
Atlanta, GA, USA Atlanta, GA, USA
aburge6@email.com ihong34@gatech.edu
ABSTRACT
Through partial recreation of the Audio-to-Score (A2S) Task as
described in A Holistic Approach To Polyphonic Music
Transcription With Neural Networks by Miguel A. Román,
Antonio Pertusa, and Jorge Calvo-Zaragoza [1], we have
determined that its experiment is reproducible, though with
difficulty. Our recreation of the experiment included modifying
source code and outputting a neural network model based on
training and test data. With our issues with installation, setup,
time and resource restraints, and execution of code, we were
only able to run six epochs. While we obtained results that
aligned with the original paper, we were not able to evaluate
the full reproducibility of the experiment, which ran one
hundred epochs. We can conclude through our findings that,
although inconclusive, our results are nontrivial in replicating
an end-to-end polyphonic transcription.
1 Introduction
Reproducibility is essential to science because it proves the
validity of an experiment, allowing for more thorough research
while confirming its results. The result of an experiment is
considered a building block for other studies only after it is
proven valid. Therefore, it is important for us to determine
reproducibility of an experiment. Specifically, we would like to
reproduce this paper due to its model’s potential in automatic
music transcription.
To assess the reproducibility of the paper, we have adopted the
definition of reproducibility from the National Academies of
Sciences, Engineering, and Medicine. Reproducibility is
obtaining consistent results using the same input data;
computational steps, methods, and code; and conditions of
analysis [10].
The source code and data from A Holistic Approach To Polyphonic Music
Transcription With Neural Networks are available
https://github.com/mangelroman/audio2score.
Joanna Lin Heesuk Son
Music Technology Music Technology
Georgia Inst. of Tech. Georgia Inst. of Tech.
Atlanta, GA, USA Atlanta, GA, USA
jlin442@gatech.edu hson40@gatech.edu
1.1 Reproducibility
In determining reproducibility, we have used the following set
of questions, which are also adopted from the National
Academies of Sciences, Engineering, and Medicine:
● Are the data and analysis laid out with sufficient
transparency and clarity that the results can be
checked?
● If checked, do the data and analysis offered in support
of the result in fact support that result?
● If the data and analysis are shown to support the
original result, can the result reported be found again
in the specific study context investigated?
We will explore this by comparing our execution of the
experiment with that of the original paper.
2 Background
2.1 Background From the Paper
The main purpose of the original experiment, as presented by
Román, Pertusa, and Calvo-Zaragoza, was to perform automatic
notation-level music transcription, a task evolved from frame-,
note-, and stream-level transcription [1]. The input data, as
described in the paper, was the time-domain spectral
information derived from the Hamming window STFT of the
audio found in the Quartets and Chorales dataset of the
humdrum-data repository [4]. Their methodology was to
implement a single-step convolutional recurrent neural
network (CRNN) that would output **kern files, which could
then be translated into a western music score format. Aspects
of scores not included in the creation of the model were clefs,
time signatures, key signatures, and several music notation
symbols due the potential for misleading the training of the
model.
at
In the original experiment, preprocessing done on the data
included manually editing **kern files for correctness, adding
metronome markings, including a random scaling factor to
account for tempo variability, and using symbolic notation to
condense the number of characters. After this processing, the
method for the A2S task included training and decoding a CRNN
as well as its outputs. The CRNN training was accomplished
using a Connectionist Temporal Classification (CTC) loss
function. To reduce overfitting, 10 batch normalization layers
and 20 dropout layers were added after all the recurrent Figure 2: A chart showing the error per epoch trained. Taken
convolutional layers. However, due to the smaller size of the from Figure 4 in [1].
Chorales dataset, there was a greater risk of overfitting
compared to the Quartets dataset. The architecture of the Overall, the original experiment was an expansion of multiple
CRNN was as shown below: previous automatic music transcription attempts. It improved
upon previous attempts by not requiring a frame-alignment of
the ground truth, including an end-to-end approach that keeps
errors within one stage, and using **kern-format output files,
which allows for easy translation to the western music score
format.

2.1 Background From the Repository

Figure 1: A model of the original experiment’s CRNN
architecture. LSTM is the acronym for Long Short-Term
Memory. Taken from Figure 3 in [1].
The code, as presented in the original GitHub [3], was based
off of DeepSpeech2, an end-to-end English and Mandarin
speech recognition experiment run by Sean Naren [5]. The
base system requirement for the original experiment’s
installations was an Ubuntu 18.04 operating system with a
miniconda package and environment manager, and the
python version used in the original experiment was Python
3.7. Below is a figure detailing the flowchart of required
installations:

The evaluation metrics, called Word Error Rate (WER) and

Character Error Rate (CER), were adopted from an ASR task
[2]. These metrics are “defined as the number of elementary
editing operations (insertion, deletion, or substitution)
needed to convert the predicted sequences into the ground-
truth sequences, at the word and character level respectively”
[1]. These were calculated after each of the 100 epochs that
the model trained, and the model with the lowest WER was
used as the test model. The batch sizes of the training data
was as follows: “The Chorales dataset, whose samples are full-
length chorales, is trained with a batch size of 4. The Quartets
dataset, whose samples are small excerpts extracted from the
full-length quartets, is trained with a batch size of 16” [1].
Figure 3: A flowchart of the installations necessary to run the
The best model after training was determined to be one with A2S task in python.
“WER of 30.96% and CER of 18.10% for Chorales, and WER of
21.02% and CER of 13.53% for Quartets” [1]. The most errors The purpose of the first step, the miniconda environment, was
arose from differentiating between voices and rhythms, to activate a virtual environment that could run the
leading to syntax errors in the **kern sequences. Figure 2 experiment. Pytorch Audio, which allows for GPU acceleration
shows the correlation between error percentage and number and running deep neural networks [6], includes functions for
of epochs run for both the Chorales and Quartets datasets. input/output, signal processing, dataset, and forming the
neural network model. This is an important second step

2
because of its necessity in creating and running the CRNN
mentioned in the paper. The third step, NVIDIA Apex, is an
extension for easy mixed precision and distributed training in
Pytorch [7], and the fourth step, Fluidsynth, is a software
synthesizer that generates audio from MIDI data or plays MIDI
files [8]. FFmpeg allows for the processing of audio [9]. The last
step, the humdrum path, installs the tools necessary to process
humdrum files.
After the necessary installations were completed, the
downloaded Chorales and Quartets datasets were then stored
in a folder. In the same folder was a preparation script, which
created a training file, a validation file, a testing file, and an
output label file. These represented the CRNN stored into csv
files.
After the data preparation was completed, the model was
trained, starting with a shell script that trained the model using
default parameters. This script’s parameters could be modified
to be multi-GPU, mixed precision, or a combination of the two.
The testing of the model was also done through shell, and after
both testing and training were completed, the model as well as
its metrics were outputted using python. As stated in the
Section 2.1, the evaluation metrics were WER and CER, and the
output model was the one with the lowest WER. This marked
the end of the original experiment code, from which they
derived a WER of 30.96% and CER of 18.10% for Chorales, and
WER of 21.02% and CER of 13.53% for Quartets.
2 Implementation
Many technical issues arose when attempting to replicate the
paper. A major hurdle in this project was the fact that no model
was provided. This meant that we needed to complete an entire
installation of the repository in order to train a model to
validate the results. Therefore, the scope of our project early on
was to create a model to see if we could get similar results to
the paper. If we did, we could also change parameters in the
training to create interesting results. For instance, given a
dataset of music of a different era, we could see how a model
trained on another genre of music fares with the automated
transcription algorithm.
Early in the process, it became apparent that the program was
not going to be smoothly compatible with Windows or MacOS,
so workarounds had to be found in order to get the installation
running smoothly. Our first solution was to run a virtual
machine with the appropriate Ubuntu environment. This led to
easier installation of the required packages, some of which
were not available on non-Linux systems.
Once installation was complete, training the model became the
next step. First, the dataset had to be prepared. This included
trimming the audio files, removing ties from the **kern files,
and other manipulations that were performed using the
humdrum tools. This preparation also split the training data
from the testing data. Before training began, however, a major
flaw in the environment was realized, which was that a virtual
environment could not access GPU hardware. This was a huge
oversight that required us to change how our environment was
set up. Assuming that our goal was to replicate a model as
shown in the paper, training a model with 100 epochs with the
limited resources partitioned to the virtual machine would take
far too long to train. Additionally, due to the usage of NVIDIA
Apex AMP, a pytorch extension that enabled automated mixed
precision, the training would have to be done on a GPU or
require significant intervention within the code to get a model
trained solely using a CPU.
The next solution was to import the entire system into Google
Colaboratory (Colab) and use their resources to train our
model. Trivial issues with package imports and version
conflicts arose, but when those were resolved, training was
finally able to begin. However, the switch to Colab introduced
an unfortunate issue to the scope of our project. Because Colab
only allowed for limited-time GPU resource usage, we had to
limit the total number of epochs our model ran. Due to time and
Colab constraints, the model was only trained and validated
against 6 epochs. Due to the model taking around 30 minutes
per epoch to train, we eventually ran out of GPU resources from
Colab while attempting to train the 20 epochs that we had
reduced our scope to. This complication was further
exacerbated by a failure on our part to properly document our
changes in a way that Colab could easily replicate the fixes.
Many of the fixes were changed during the runtime of the Colab
and not onto a forked repo. This meant that each time the
model had to be changed, it would be changed just for that
instance of the Colab and lines of code would have to be
changed manually. Ideally, we would be able to share the model
once our Colab timed out and rotate the training of the model,
or we would create different Google accounts to train the model
at all times.
Regardless of these issues, we were able to get a trained model
and evaluate it on a testing set. With our model trained with the
quartet dataset, we were able to get a WER of 34.93% and a
CER of 26.03%.
3

Figure 4: Ground truth **kern (right) and the model’s
hypothesis **kern (left). According to the algorithm in the
paper, the WER is 12.30% and the CER is 8.45%.
The final task was to see if the model could produce a
transcription from audio. A script was provided that allows for
a .wav file to be converted to a **kern file using the model,
which could then be converted to musicXML. While the **kern
file was able to be created, the conversion between **kern and
XML proved difficult, as the exported file required significant
manual cleaning and a proper understanding of the **kern
format.
3 Conclusion
Throughout the experiment, we questioned the reproducibility
of the paper. Trivial issues, like debugging and trouble with the
initial installation, along with more pertinent problems, like
finding the best operating system and problems within the
source code, reduced the chances of reproducing the results.
Through all the challenges we faced, we found that the original
experiment is reproducible, but with many caveats. Syntax
errors, invalid parameters, missing packages, and other issues
prevented us from reproducing the experiment fully. However,
through scanning and studying the source code, it was
transparent enough for us to be able to diagnose and change
lines of code to get the experiment running. In modifying the
source code, important parameters had to be manipulated in
order to get the training to begin. Despite these manipulations
to the experiment, we were able to produce nearly identical
WER and CER when comparing our model and the paper’s
model at 6 epochs. This discovery is non trivial, and one could
argue that it proves a certain degree of reproducibility; there is
only a small difference in WER and CER between 6 epochs and
80 epochs in the paper. However, this argument is also abstract
and inconclusive because: 1) Changes were still present in WER
and CER in the paper up to 80 epochs; and 2) The experiment
we ran was inconclusive. To replicate the paper’s results, we
would have needed at least 80 epochs to validate our results
accurately. We also acknowledge that we were limited in time
and resources, so our evaluation of this experiment’s
reproducibility may be incomplete. However, the issues in
setting up the experiment existed despite our limitations and
contributed to the difficulty of its reproducibility.
To make this experiment easier to reproduce, we recommend
updating code versions, checking for syntax correctness, and
providing details for all source code. This would allow us to
smoothly execute all code without errors due to misspelling or
version/package/system differences. We would also like to
offer one potential extension on the original experiment that
we would have explored if we were able to reproduce this
experiment fully—that is, to train the model on datasets with
genres of music other than classical. The training of this model
on wider ranges of music would expand its application to
include different instruments and rhythms, and we predict that
this new training would aid the model in the differentiation of
different voices. Overall, because of the model’s potential in the
realm of automatic music transcription, we believe that this
experiment is worth revisiting fully and editing so that it
becomes an easily reproducible task.
REFERENCES
[1] Miguel A. Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. 2019. A
holistic approach to polyphonic music transcription with neural networks.
arXiv. DOI: https://doi.org/10.48550/ARXIV.1910.12086.
[2] Miguel A. Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. An End-to-End
Framework for Audio-toScore Music Transcription on Monophonic
Excerpts. In Proc. of the 19th International Society for Music Information
Retrieval Conference, ISMIR 2018, Paris, France, 2018.
[3] https://github.com/mangelroman/audio2score.
[4] https://github.com/humdrum-tools/humdrum-data.git.
[5] https://github.com/SeanNaren/deepspeech.pytorch.
[6] https://github.com/pytorch/pytorch#installation.
[7] https://github.com/NVIDIA/apex.
[8] https://github.com/FluidSynth/fluidsynth.
[9] https://github.com/FFmpeg/FFmpeg.
[10] National Academies of Sciences, Engineering, and Medicine; Policy and
Global Affairs; Committee on Science, Engineering, Medicine, and Public
Policy; Board on Research Data and Information; Division on Engineering
and Physical Sciences; Committee on Applied and Theoretical Statistics;
Board on Mathematical Sciences and Analytics; Division on Earth and Life
Studies; Nuclear and Radiation Studies Board; Division of Behavioral and
Social Sciences and Education; Committee on National Statistics; Board on
Behavioral, Cognitive, and Sensory Sciences; Committee on Reproducibility
and Replicability in Science. Reproducibility and Replicability in Science.
Washington (DC): National Academies Press (US); 2019 May 7. 3,
Understanding Reproducibility and Replicability. Available from:
https://www.ncbi.nlm.nih.gov/books/NBK547546/.