Arabian Journal for Science and Engineering (2022) 47:3447–3458

https://doi.org/10.1007/s13369-021-06297-w

RESEARCH ARTICLE-ELECTRICAL ENGINEERING

A Deep Learning Framework for Audio Deepfake Detection

Janavi Khochare1 · Chaitali Joshi1 · Bakul Yenarkar1 · Shraddha Suratkar1 · Faruk Kazi1

Received: 6 June 2021 / Accepted: 3 October 2021 / Published online: 8 November 2021
© King Fahd University of Petroleum & Minerals 2021

Abstract
Audio deepfakes have been increasingly emerging as a potential source of deceit, with the development of avant-garde methods
of synthetic speech generation. Hence, differentiating fake audio from the real one is becoming even more difficult owing to
the increasing accuracy of text-to-speech models, posing a serious threat to speaker verification systems. Within the domain
of audio deepfake detection, a majority of experiments have been based on the ASVSpoof or the AVSpoof dataset using
various machine learning and deep learning approaches. In this work, experiments were performed on a more recent dataset,
the Fake or Real (FoR) dataset which contains data generated using some of the best text to speech models. Two approaches
have been adopted to the solve problem: feature-based approach and image-based approach. The feature-based approach
involves converting audio data into a dataset consisting of various spectral features of the audio samples, which are fed to the
machine learning algorithms for the classification of audio as fake or real. While in the image-based approach audio samples
are converted into melspectrograms which are input into deep learning algorithms, namely Temporal Convolutional Network
(TCN) and Spatial Transformer Network (STN). TCN has been implemented because it is a sequential model and has been
shown to give good results on sequential data. A comparison between the performances of both the approaches has been
made, and it is observed that deep learning algorithms, particularly TCN, outperforms the machine learning algorithms by
a significant margin, with a 92 percent test accuracy. This solution presents a model for audio deepfake classification which
has an accuracy comparable to the traditional CNN models like VGG16, XceptionNet, etc.

Keywords Audio deepfakes · Feature-based classification · Image-based classification · Temporal convolutional networks ·
Spatial transformer networks

1 Introduction human lives but when misused can be the source of great
damage is deepfakes. A technology that was conceived as an
A steep increase in the number of social media platforms has innovative research area now threatens to pose as a tool to
facilitated the rapid spread of information, bringing with it spread misinformation.
several advantages by connecting people across the globe. Deepfake—an amalgamation of the words deep learning
But these media are also being used as weapons of mass and fake, refers to the synthetic data generated using deep
misinformation and causing widespread unrest. One of the learning techniques such as Generated Adversarial Networks
technologies which possesses great potential to transform (GAN) and Recurrent Neural Networks (RNN). Deepfake
has been used as a tool for committing forgeries, allowing
B Shraddha Suratkar a person to impersonate someone else, the development of
sssuratkar@ce.vjti.ac.in
which has been bolstered by an increasing influx of infor-
Janavi Khochare mation over the internet making it possible to train the deep
jskhochare_b17@ee.vjti.ac.in
learning network on a large set of data making the forged
Chaitali Joshi material indistinguishable from the real ones for humans.
cpjoshi_b17@ee.vjti.ac.in
Earlier studies into audio deepfake detection tasks have
Bakul Yenarkar primarily used the AVSpoof or ASVSpoof datasets. How-
byyenarkar_b17@ee.vjti.ac.in
ever, one major disadvantage of using these datasets is that
Faruk Kazi they do not include audio created by the most up-to-date text-
fskazi@el.vjti.ac.in
to-speech algorithms, which sound more like human speech
1 Veermata Jijabai Technological Institute, Mumbai, India

123
3448
and may be indistinguishable to the human ear. When classi-
fying such audio from actual human-generated audio, a more
complicated task may arise, necessitating the development of
robust solutions.
The experiments as described in this work have been based
on the Fake or Real (FoR) dataset [1]. This dataset has been
chosen to train the models as described in this work because
it contains samples of audio generated by the latest text-to-
speech models as well as natural utterances, allowing the
development of such models that can distinguish between
fake and real audio more robustly.
This work proposes a framework for the task of audio
deepfake detection. To classify between real and fake audios,
two approaches have been adopted: a feature-based approach
using machine learning algorithms and an image-based
approach using deep learning algorithms. In the feature-
based classification approach, the audio file is converted into
a feature-based dataset consisting of various spectral features
of the samples, these features are further input to the machine
learning algorithms to differentiate between audio samples
as real or fake. The machine learning algorithms that have
been utilized for this purpose are Support Vector Machines
(SVM) [2], Light Gradient Boosting Machines (LGBM)
[3], Extreme Gradient Boosting (XGBoost) [4], K-Nearest
Neighbors (KNN) [5], and Random Forest (RF) ) [6]. The
image-based classification approach has been undertaken
to represent the audio sample in the form of melspectro-
grams which are further fed as an input to the deep learning
algorithms that classify whether the given audio instance is
real or fake. The deep learning architectures that have been
used include Convolutional Neural Networks (CNN), Spatial
Transformer Network (STN), and Temporal Convolutional
Network (TCN). TCN being a sequential network is better
able to capture the features in data which is also sequen-
tial in nature and hence, presents a model which is more
suited to the task of audio deepfake detection. The deep
learning architectures have been used to classify melspectro-
grams generated from the incoming audio, and this presents
a case similar to image classification. Thus, the major con-
tributions of this work include adopting two approaches, viz,
image-based approach and feature-based approach for audio
deepfake detection on FoR dataset, comparing the perfor-
mance of the models implemented under each approach and
proposing a comparatively simpler deep learning model for
the task of audio deepfake detection which has an accuracy
which is comparable to that of the traditional deep learning
algorithms like VGG19, XceptionNet, etc. The rest of the
paper is organized as follows: Sect. 2 describes the works
which have been done previously in the domain of audio pro-
cessing and audio deepfake detection. Section 3 introduces
and explains the most important concepts and terminologies
that have been implemented in this work. Section 4 explains
the exact methodology and experiments performed to obtain
123
Arabian Journal for Science and Engineering (2022) 47:3447–3458
the results as described in Sect. 5 which also presents a
comparison between the various models implemented in this
works as well as the methods employed in a previously pub-
lished work. Section 6 mentions a closing remark of the paper
and highlights the future directions of the work.
2 Related Work
Audio signals form an unregulated natural world with a
variety of content, such as music, speech, and ambient
sounds. Most research involving audio as an input over
the last two decades using methods that differ primarily
in the types of audio features used or classification tech-
niques employed to various scenarios to classify audio
[7–11]. With the advancements in text-to-speech technolo-
gies, cutting-edge systems can now generate good-quality,
near realistic-sounding speech from a limited quantity of
speech data from target speakers [12]. In contrast, it also
offers a substantial threat to automatic speaker verification
(ASV) systems, since the faked speech generated by such
techniques can readily attack the system [13,14]. The protec-
tion of the ASV system, therefore, calls for deepfake speech
detection.
A majority of the experiments in the domain of audio deep-
fakes detection have been based on the ASVS-poof dataset.
These works provide an early insight into the methods and
models used for detecting forged speech. Both machine
learning and deep learning models have been implemented
for the said task. [15–17] propose the implementation of Deep
Neural Networks (DNNs) for the task of synthetic speech
detection and involve the use of variants of Convolutional
Neural Networks to study the suitability of these models
and suggest that approaches which are end-to-end are more
suited for detecting fake audio. [18] proposes to use a Rel-
ative Phase Shift to detect synthetic speech and Gaussian
Mixture Model (GMM) and Support Vector Machine(SVM)
which are shown to reduce the vulnerability of speaker ver-
ification systems. A comparative study between DNN and
Hidden Markov Model (HMM) based on their capability to
identify spoofed speech has been made in [19] which shows
that DNN outperforms HMM, implying that DNN is better
able to capture the patterns that can be identified as being
characteristic of spoofed audio. [20] demonstrates the usage
of spectrograms as a form of image input to CNN in audio
processing scenarios and thus, forms the basis for image-
based audio processing techniques. The paper [21] presents
a DNN classifier for detection, also highlighting that Human
Log-Likelihoods (HLL) as a scoring metric is more appro-
priate as compared to the traditional log-likelihood ratios
(LLR). The paper also implements different cepstral coeffi-
cients to train classifiers. The research paper [22] examines
various robust audio features like Mel Frequency Cepstral
Arabian Journal for Science and Engineering (2022) 47:3447–3458
Coefficient (MFCC), spectrogram, etc., and their effect on the
accuracy of GMM-UBM as the classifier model and shows
that when a combination of a various audio features is used
gives the best result in terms of Equal Error Rate (EER).
[23,24] propose the use of CNN as classifiers for sound
classification scenarios. An exhaustive description and com-
parison of the various deep learning models used for audio
deepfake detection has been made in [25] and shows that the
CNN-RNN model achieves the best result among all the other
models used. [24,26] describes how short-term spectral fea-
tures can be effectively used for synthetic speech detection,
further showing that the MFCCs outperform other spectral
features when used as feature inputs to the model. Finally,
[27] discusses the various limitations of the spoofing detec-
tion mechanisms.
CNNs are considered to be robust and sturdy models; how-
ever, they are restricted to be spatially invariant to the data
which is given as an input in a parameter effective and compu-
tational manner. The paper [28] introduces a state-of-the-art
architecture called STN which permits the spatial manipula-
tion of data present in the network. STN engender models that
learn invariance to scale, rotation, translation, and generic
wrapping, providing more efficient results
TCN [29] has been shown in experiments to outperform
traditional RNNs and LSTMs across a wide range of tasks.
[24] serves as a starting point for understanding the concept
of TCN and its building blocks, as well as outlining the var-
ious applications to which this model can be applied. TCN
has already been used in language processing [30], activity
detection [31,32], and event detection tasks. [33] describes
the use of TCN for audio spoof detection. This paper com-
pares TCN and Multilayer Perceptron (MLP) performance
and has shown that TCN is better than MLP. [34] describes
a time series forecast TCN architecture and outlines how
the network can efficiently identify temporal dependencies
in data. It can be concluded from the experiments described
in these abovementioned papers that TCN captures temporal
dependence in the data effectively.
With the synthetic speech generation making advance-
ments each day, the requirement of detection of synthetic
speech increases and so does the requirement for an up-
to-date dataset for natural and computer-generated speech.
The paper “FoR: A Dataset for Synthetic Speech Detec-
tion” [1] introduces a new dataset called FoR dataset which
has around 198,000 utterances including both real and syn-
thetic speech generated using the latest algorithms. It also
focuses on analyzing how synthetic speech is generated and
the performance of various deep learning models that classify
this synthetic speech. It also presents various experiments
that demonstrate the versatility and usefulness of the newly
introduced FoR dataset to improve the quality of synthetic
audios generated using various methods and also in the detec-
tion of computer-generated speech as it is possible to train
3449
deep learning-based classifiers using this dataset. This paper
shows that the VGG16 model performs the best among all
the other models used. A major drawback of these works is
that TCN as a model and its suitability for sequential input
data has not been considered for the task of audio deepfake
detection which has been successfully addressed in this work.
3 Preliminaries
1. TCN
Temporal Convolution Networks or TCN (Fig. 1) [24,29]
is a family of CNNs with two distinctive features:
(a) The output is identical in terms of length to the input.
(b) Data are not leaked to the past from the future.
To satisfy the first condition, a 1D Fully-Convolutional
Network (FCN) is used wherein the length of all hidden layers
is kept equal to the length of the input layer along with a
zero padding which has a length of kernel size is equal to 1.
Further to meet the second condition, causal convolutions are
used, meaning that only such convolutions where the output
at certain time is convolved only with components from that
and earlier times which are present in the previous layer are
used.
Hence, TCN can be seen as a combination of 1D-FCN
and causal convolutions. This type of architecture is useful
to catch long-term dependencies using dilated convolutions.
Dilated convolutions allow for the TCN to have a wider recep-
tive field, meaning that a wider section of the input data can
be used to obtain the output. Here, for a 1D sequence input
is given as y ∈ Rn and filter as f : {0, . . . , m − 1} → R
and the dilated convolution operation F on element u can be
defined as:

m−1
F(u) = (y ∗b f )(u) = f ( j)yu−b j (1)
j=0
here in Eq. (1), b represents dilation factor, m is the filter
size, and u − b j is the direction of the past.
The receptive field size can be changed, by changing the
kernel size or the dilation rate. Furthermore, skip connec-
tions and residual blocks within the architecture allow the
gradients to pass through easily.
o = Activation(y + F(y)) (2)
As given in Eq. (2), the output of series of transformations
F is added to input y of the block.
2. STN
CNNs are considered to be extremely powerful models
123
3450 Arabian Journal for Science and Engineering (2022) 47:3447–3458

Equation (3) represents affine transformation matrix Θ

[35].
⎡ t⎤
 s xi
xi
s = Θ ⎣ yit ⎦ (4)
yi
1

Here, (xit , yit ) = coordinates in the target (output) feature

Fig. 1 TCN map and (xis , yis ) = coordinates in the source (input) feature
map.
Source column vector and target column vector with Θ
being the affine transformation matrix as shown in Eq. (4).
⎛ t⎞ ⎛ ⎞
xi   xit
xis θ θ θ
= Tθ (G i ) = Aθ ⎝ yit ⎠ = 11 12 13 ⎝yt ⎠ (5)
Fig. 2 A spatial transformer yis θ21 θ22 θ23 i
1 1

Calculation of affine transformation matrix is carried out

for image-based tasks, but are hindered because they
are spatially variant to the information that has been
fed as an input in a computational and parameter effec-
tive manner. So, a spatial transformer was introduced
which notably allows spatial manipulation of the data. A
dynamic mechanism that can perform a spatial transfor-
mation on an image of a feature map is a spatial transform
module. A generalization of differentiable attention to
a spatial transformation is called an STN [28] (Fig. 2).
STN applies an affine transformation which is followed
by bilinear interpolation that, in turn, helps to remove
spatial invariance from the images.
The three components that make up the spatial trans-
former mechanism are localization network, grid generator,
and sampler. The input feature maps are passed through sev-
eral hidden layers by the localization network to produce
output parameter theta for the spatial transformer. The size
of theta varies depending on the type of transformation that
is parameterized. Localization network can be defined as
– Input Input feature map U of shape (H, W, C)
– Output A transformation matrix Θ of shape (6,)
– Structure Either a ConvNet or FCN
The grid generator creates a sampling grid by using pre-
dicted transformation parameters. The task allocated to the
grid generator is to calculate the coordinates in the source
feature map of every target pixel. This can be performed by
an affine transformer [35].
 
θ11 θ12 θ13
Θ= (3)
θ21 θ22 θ23
using Eq. (5) [35].
Feature map and sampling grid are given as an input to
the sampler, and it generates an output map which is sampled
from input at the grid points. Generation of pixel intensity of
every target coordinate by bilinear interpolation is performed
by the sampler.
 H W
Vic = c
Unm max
n m
×(0, 1 − |xis − m|)max(0, 1 − |yis − n|) (6)
Pixel intensities via Bilinear Interpolation is calculated
using Eq. (6) [35].
3. Melspectrograms
The spectrogram is a graphic representation of signal
strength in terms of the signal’s intensity, which shows
frequency variations of acoustic signals with respect to
time. It is generated by performing Fast Fourier Trans-
form (FFT) on overlapping windowed sections of the
signal. Often, the frequency is stated on the y-axis and
the time on the x-axis. The color of the plot exposes the
intensity. Melspectrogram [21,28] is a type of spectro-
gram where the frequency is mapped and presented in
the Mel scale, i.e., a scale of pitches. Fig. 4 represents a
melspectrogram file of an audio instance in FoR dataset.
Steps performed are:
(i) First step in generating melspectrogram involves
framing and windowing of input audio signal. The
goal is to divide the signal into a series of short
overlapping frames in order to ensure that they are
stationary. A stationary signal represents the genuine
statistical and temporal properties. Windowing is fre-
quently done with rectangular windows, such as the

123
Arabian Journal for Science and Engineering (2022) 47:3447–3458
Fig. 3 Conversion of acoustic signal to melpectrogram
Fig. 4 Melspectrogram
Hamming window, which hides the possibility of dis-
torted segments at the edges.
(ii) The signals are converted from the time domain to
the frequency domain using the Fourier transform,
which allows them to be represented statistically and
spectrally.
(iii) The spectrogram envelope is generated using the Mel
filter banks [36].
4 Methodology
1. Dataset Description
(a) Fake or Real (FoR) Dataset The FoR dataset [1] is
a set of around 195,000 utterances from real humans
and computer generated speech. The computer gen-
erated samples are created using latest methodologies
in speech synthesis. The real utterances have samples
recorded using different microphones that include dif-
ferent samples from variety of accents and from all
genders. Classifiers can be trained using this dataset
to identify synthetic speech. There are four versions
of this dataset: for-orig, for-norm, for-2sec, and for-
rerec:
(i) For-original It consists of a collection of original
files from various speakers. This dataset has an
unequal number of samples from different gender
3451
and class. It has around 195,541 samples from
various sources, and it was published so that the
raw data can be used in various pre-processing
techniques.
(ii) For-norm It has similar files to that of for-
original, and it has an equal number of samples
from different gender and class. It is normalized
in terms of volume, sample rate and several chan-
nels. The for-norm version consists of around
69,400 samples.
(iii) For-2sec This dataset consists of files trim-med at
2 s. It has equal number of samples from different
gender and class just like the normalized version
(for-norm) and has a total of 17,870 samples from
these sources.
(iv) For-rerec This version of dataset includes rere-
corded files from for-2second dataset.
This is done to simulate a situation in case an
audio is sent through a phone call or voice mes-
sage or similar voice channel by any attacker.
(b) Generated audios This dataset consists of a collection
of audios generated using real-time voice cloning [37].
The input given to this model consists of an audio sam-
ple of a person extracted from YouTube and sample
text. The output is audio generated using real-time
voice cloning and includes audios from speakers of
all genders, age and accents. This collection of audios
is included in the dataset to know how well can the
model be generalized.
2. Preprocessing
The dataset consists of data in the form of audio which
cannot be directly given as an input to the classifica-
tion models. These data are needed to be converted to
a suitable format before starting the actual classification
task and so preprocessing of data was necessary. In this
project, two classification-based approaches have been
adopted, namely feature-based classification and image-
based classification. Process flow diagram of deepfakes
audio detection is as shown in Fig. 5
(a) Feature-based classification:
In the feature-based classification approach, the audio
file is converted into a feature-based dataset consist-
ing of various features of the audio samples [22] as
123
3452
Fig. 5 Process flow diagram for
deepfake audio detection
Fig. 6 Feature extraction
shown in Fig. 6. First, the individual audio file is
taken as an input and loaded using librosa library. This
outputs a time series and the sampling rate which rep-
resent the digital form of the audio file. Using librosa
library functions, features, as mentioned below, are
extracted. This gives another array and to have a sin-
gle value for each feature, the mean of this array is
calculated which gives the final value of the feature
for that particular audio file. These features are fur-
ther fed to the machine learning algorithms for the
classification of audio samples as real or fake. Each
audio sample in the dataset has been represented as a
vector of 37 features of audio. This was done using a
sliding window technique, and thus, the features are
time-varying sequences. The features that have been
extracted include:
(i) Mean Square Energy Given a signal x(n), the
Root Mean Square Energy (RMSE) is given by
123
Arabian Journal for Science and Engineering (2022) 47:3447–3458
Eq. (7):

1 2
xrms = (x + x22 + · · · + xn2 ) (7)
n 1
where n = number of samples and xi = ith sam-
ple.
For a general signal, the RMSE is calculated
as a single value from all the values present in
the signal. However, due to the non-stationary
nature of audio signals, it is preferable to com-
pute the energy over short intervals to capture
the variation over time. Different types of speech
may have different energy content, making it an
important feature to consider for audio classifi-
cation.
(ii) Chroma Features The chroma features represent
the frequency spectrum as 12 pitch classes. The
entire frequency is divided into 12 bins denoting
the 12 chromas present in the musical octave.
Arabian Journal for Science and Engineering (2022) 47:3447–3458
The chromagram of a sample audio signal is pre-
sented in Fig. 7.
(iii) Spectral Centroid The spectral centroid is indica-
tive of the nature of the frequency prevalent
in the signal. For instance, a higher spectral
centroid value indicates a concentration in the
high-frequency region in the spectrum. It is com-
puted for the ith audio frame as shown in Eq. (8):
k=N
k=1 f (k). f (k)
μ= n=N
k=1 m(k)
where m(k) is the magnitude at the kth frequency
bin, f (k) is the center frequency at the kth fre-
quency bin. The spectral centroid plot of the
audio sample is shown in Fig. 8.
(iv) Spectral Bandwidth The range of frequencies in
the audio signal spectrum which corresponds to
one-half of the peak magnitude of the spectrum
is known as the spectral bandwidth of the audio
signal. The pth order spectral bandwidth is given
by Eq. (9):
 1
 p
m(k)( f (k) − μ) p
k
where m(k) is the magnitude at the kth frequency
bin, f (k) is the center frequency at the kth fre-
quency bin and μ is the spectral centroid. The
spectral bandwidth plot of the sample audio sig-
nal is shown in Fig. 9.
(v) Spectral Rolloff The spectral rolloff is the point
in the frequency below which 85 percent of the
energy present in the spectrum resides. This pro-
vides the general shape of the spectra by ignoring
the outlier higher frequencies and focusing on the
portions of the spectra where most of the energy
is present. It is given by Eq. (10):

fr 
N
arg max m(k) ≥ 0.85
f r {1,...,N }
k=1 k=1
where f r is the rolloff frequency, and m(k) is the
magnitude at the kth frequency bin. The spectral
rolloff plot of sample audio signal is shown in
Fig. 10.
(vi) Zero Crossing Rate The rate of sign-changes
in a signal is referred to as the zero-crossing
rate. Lower frequency signals have a lower zero-
crossing rate due to lesser oscillations per second
as compared to higher frequency signals. It is
3453
given by Eq. (11):
1 
WL
|sgn[x(n)] − sgn[x(n − 1)]| (11)
WL
n=1
where x(n) is the audio signal, WL is the length
of the window, and sgn is the signum function.
The zero crossing rate plot of the audio sample
is presented in Fig. 11.
(8) (vii) Mel Frequency Cepstral Coefficients (MFCCs)
MFCCs reflect the enclosed envelope of the
power spectrum that depicts the characteristics of
the vocal tract and human voice. Mel-frequency
cepstrum, formed by these coefficients, is used to
identify periodic components of a time domain
signal as peaks in a new domain called the “que-
frency” domain [38]. The MFCCs are obtained
by transforming the signal to frequency domain
from time domain, and then to the quefrency
domain from frequency domain, using a series of
mathematical transformations. The process for
generating MFCC is similar to that for generating
a melspectrogram, with the following additions
[39]:
(9)
A. The magnitude of powers from Mel filters is
converted to a logarithmic scale.
B. Then, discrete cosine transform is applied
which outputs cepstral coefficients.
C. Applying the Discrete Cosine Transform
(DCT) on the results produced in the pre-
ceding stage, which results in cepstral coef-
ficients. The MFCC plot of the sample audio
signal is shown in Fig. 12.
In this work, 20 MFCCs have been considered.
(b) Image-based classification
The image-based classification approach has been
undertaken to represent the audio sample in the form
of melspectrograms which are further fed as an input
for deep learning algorithms that classify whether the
given audio instance is real or fake. Each audio sample
m(k) (10) in the dataset has been represented as a melspectro-
gram of the audio data. Melspectrograms have been
generated by using the librosa library. The mel level
for representing the frequency axis was selected as
256. The human ear recognizes frequencies that are
in the range of 20 Hz and 20 kHz. As the lowest
threshold frequency perceptible to the human ear is
approximately 20 Hz, frequencies below the value of
20 Hz are not shown in the spectrogram. The sam-
pling rate of all audio files is 16 kHz, so according
to the Nyquist sampling theorem, the audio samples
should contain frequencies only up to 8 kHz. So the
123
3454 Arabian Journal for Science and Engineering (2022) 47:3447–3458

Fig. 7 Chromagram of sample audio signal

frequency range of the spectrogram is between 20

Hz and 8 kHz. The Hann window function is imple-
mented, and the window is set as 1024. The hop length Fig. 8 Spectral centroid plot of sample audio signal
of the window is taken as 256. The melspectrogram
generated as an image of the audio sample has a height
of 256 due to its number of mel levels and width being
conditional on the length of the audio. As most of the
deep learning algorithms implemented for the clas-
sification task required the dataset to be uniform in
terms of the size of the melspectrogram image, the
width threshold of the image has been set to be 256.
For this task, the For-2-sec dataset has been used so
each spectrogram corresponds to 2 s of audio samples.
3. Implementation
(a) Feature-based approach
The feature-based classification task involves classi-
fying the given audio instance as real or fake. The data
Fig. 9 Spectral bandwidth plot of sample audio signal
obtained by preprocessing of the audio sample, i.e.,
the vector of 37 audio features was fed as an input
to the machine learning algorithms. For this task,
five machine learning algorithms namely, SVM [2],
LGBM [3], XGBoost [4], KNN [5], and RF [6] have
been implemented and the performance of each of
these has been compared. The hyperparameters were
optimized using GridSearchCV. The output of these
models is whether the given instance of audio is real
or fake.
(b) Image-based Approach:
In the image-based classification task, melspectro-
grams obtained after preprocessing of the audio
samples are given as an input to the models. In this
approach, two deep learning architectures have been
Fig. 10 Spectral rolloff plot of sample audio signal
used as audio classifiers and those are STN [28] and
TCN [29]. The main purpose of using these net-
works is that the data presented in a melspectrogram The detailed explanation of the architectures of the models
is sequential in nature and a sequential network per- implemented is as follows:
forms well on sequential input data. The entire dataset
is segregated into training, testing and evaluation set (i) STN The architecture consists of a 2D-FCN. The opti-
according to the ratio 60:20:20. The output of the mizer used for this model is Adam with the loss
models is whether the given sample is real or fake. function being binary cross-entropy. The model is
implemented according to the architecture explained

123
Arabian Journal for Science and Engineering (2022) 47:3447–3458 3455

in the preliminaries and also in (Fig. 2). The archi-

tecture includes the localization network, the grid
generator, and the sampler. The model was trained on
Nvidia-DGX 1 for 15 epochs and the training and eval-
uation process took around half an hour to complete.
To avoid overfitting, early stopping has been imple-
mented. The model is trained on both the classes real
and fake.
(ii) TCN The architecture consists of a 1D-FCN where
the convolutions are causal. This is accomplished by
setting the “padding” parameter as causal. The overall
architecture is as shown in Fig. 13a. The choice of the
optimizer is Adam optimizer with the loss function
Fig. 11 Zero crossing rate plot of sample audio signal
being binary cross-entropy. The entire model can be
described as follows: The basic building block of TCN
is made up of fully connected 1D convolution layers
with a spatial dropout layer with the necessary dilation
factor. A residual connection connects the input to
output to allow for the flow of gradients. This structure
is repeated several times. The input is given to three
TCN architectures with varying kernel sizes, and the
output from the three blocks is concatenated followed
by batch normalization and dropout layer to finally
give the desired output. The overall architecture is
as shown in Fig. 13b. The TCN model was trained
and tested on Nvidia-DGX 1 for 25 epochs with the
use of early stopping to prevent overfitting. The entire
process of training and testing took around half an
Fig. 12 MFCC plot of sample audio signal hour to complete.
For both STN and TCN, after training, the model’s perfor-
mance has been validated using a validation dataset and
finally, they are applied on a test dataset which includes

Fig. 13 TCN

123
3456 Arabian Journal for Science and Engineering (2022) 47:3447–3458

Table 1 Feature-based audio

Model Validation accuracy Test accuracy Precision Recall F-score
classification results
SVM 0.85 0.67 0.69 0.67 0.65
Random Forest 0.80 0.62 0.60 0.59 0.57
KNN 0.75 0.62 0.61 0.61 0.61
XGBoost 0.70 0.59 0.59 0.58 0.57
LGBM 0.75 0.60 0.60 0.59 0.58

Table 2 Image-based classification results machine learning models have almost similar performance
Model Validation accuracy Test accuracy and saturate after a certain value. Accuracies of the models
were in the range of 0.60–0.70. The SVM model performed
TCN 0.98 0.92 the best when compared to other machine learning models.
STN 0.89 0.80 Table 2 shows the results for image-based classification after
using the deep learning models. It can be observed that the
TCN performed the best by giving a test accuracy of 92 per-
Table 3 Test accuracy values
cent. STN performed better than the machine learning models
Reference Algorithm Test by giving a test accuracy of 80 percent. Overall, the image-
accuracy
based classification provided better results than feature-based
Algorithms 4-layer fully 0.46 classification on this audio deepfake detection task. This can
implemented in [1] connected be attributed to the fact that TCN being a sequential net-
2-Layer CNN(+2FC) 0.41 work is able to capture the features of sequential data more
3-Layer CNN(+2FC) 0.38 effectively as compared to the other models that have been
VGG16 0.95 implemented. Hence, it can be observed that employing an
VGG19 0.96 image-based approach with TCN as the classifier can give
InceptionV3 0.79 better results as compared to the Machine Learning models
ResNet 0.78 with a feature-based approach.
MobileNet 0.94 Moreover, experiments have been held earlier on the same
XceptionNet 0.74 dataset as described in [1]. The results obtained in those
Proposed system SVM 0.67 experiments are shown in Table 3. It can be observed that the
Random Forest 0.62 best-performing model has an accuracy of about 96%. How-
KNN 0.62 ever, these models are computationally expensive, whereas
XGBoost 0.59
the model implemented in this paper i.e., TCN is compara-
LGBM 0.60
tively less complex and achieves an accuracy that does not
lag behind by a significant amount. Hence, implementing
TCN 0.92
TCN can be computationally efficient as well as provides
STN 0.80
equivalent results.

data similar to the one encountered in the training dataset

as well as that from the custom-built dataset. The overall
flow of the experiments carried out is shown in Fig. 5.
5 Result
This section describes the results obtained from the exper-
iments conducted as described in the previous sections for
audio deepfake detection. The metrics used for evaluating
the performance of various models are accuracy, precision,
and recall. Table 1 represents the results as observed from the
machine learning models for feature-based audio classifica-
tion tasks in this experiment. It can be observed that all the
6 Conclusion
Deepfakes have been a major cause of concern in recent
times. This material poses a danger to the whole world in
the form of manipulated audio-visual content, as it can be
the source of mass misinformation or even harassment. It is,
therefore, necessary to use more rigorous methods to combat
the destructive uses of deepfakes. This work focuses mainly
on the task of deepfake audio detection. Two approaches
have been adopted to classify between real and fake audio:
a feature-based approach using machine learning algorithms
and an image-based approach using deep learning algorithms
and later a comparative analysis on both these approaches has

123
Arabian Journal for Science and Engineering (2022) 47:3447–3458
been made. From the results mentioned above, the deep learn-
ing algorithms surpassed the machine learning algorithms.
The TCN model gave highest 92 percent test accuracy. The
reason behind TCN performing well may be attributed to
the fact that the data input is sequential and TCN being a
sequential network performs well on such data. Also, the
architecture in the proposed system is simpler as compared
to the models used in paper [1] and achieves similar accuracy
as those models.
This work can be expanded in the future by introducing
new features to the feature-based methodology that will help
in improving the results of machine learning models. Further,
amplitude-based classification can be implemented by using
deep learning models. One limitation of this work is that in
the image-based approach, STFT and MFCC have not been
used as input. The image-based approach can be extended to
using STFT and MFCC as input to the deep learning models
along with melspectrogram and a comparative study can be
performed to see which feature gives the best result with the
given classifier. Also, variants of TCN can be implemented
as classifiers and a comparison can be drawn between their
performances to ascertain which variant performs the best.
According to Wavenet [40], it is feasible to provide raw audio
as input into neural networks without converting it to spec-
trograms. Raw audio was not much utilized as a classifier for
synthetic speech recognition. This might improve classifica-
tion accuracy while also cutting down on pre-processing time
as spectrograms need not be generated. Hence, the develop-
ment of raw audio classifiers can be studied.
Acknowledgements Authors acknowledge Centre of Excellence in
Complex and Nonlinear Dynamical Systems (CoE-CNDS) laboratory
for providing support and platform for research.
Declarations
Conflict of interest The authors declare that they no conflict of interest.
References
1. Reimao, R.; Tzerpos, V.: For: a dataset for synthetic speech detec-
tion. In: 2019 International Conference on Speech Technology and
Human–Computer Dialogue (SpeD), pp. 1–10. IEEE (2019)
2. Evgeniou, T.; Pontil, M.: Support vector machines: theory and
applications. In: Advanced Course on Artificial Intelligence, pp.
249–257. Springer, Berlin (1999)
3. Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.;
Liu, T.-Y.: Lightgbm: a highly efficient gradient boosting decision
tree. Adv. Neural. Inf. Process. Syst. 30, 3146–3154 (2017)
4. Chen, T.; Guestrin, C.: Xgboost: a scalable tree boosting system. In:
Proceedings of the 22nd ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining, pp. 785–794 (2016)
5. Guo, G.; Wang, H.; Bell, D.; Bi, Y.; Greer, K.: KNN model-based
approach in classification. In: OTM Confederated International
3457
Conferences on the Move to Meaningful Internet Systems, pp. 986–
996. Springer, Berlin (2003)
6. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
7. Wold, E.; Blum, T.; Keislar, D.; Wheaten, J.: Content-based clas-
sification, search, and retrieval of audio. IEEE Multimedia 3(3),
27–36 (1996)
8. Nanni, L.; Costa, Y.M.G.; Lucio, D.R.; Silla, C.N., Jr.; Brahnam,
S.: Combining visual and acoustic features for audio classification
tasks. Pattern Recogn. Lett. 88, 49–56 (2017)
9. Lie, L.; Zhang, H.-J.; Jiang, H.: Content analysis for audio clas-
sification and segmentation. IEEE Trans. Speech Audio Process.
10(7), 504–516 (2002)
10. Zhao, J.; Mao, X.; Chen, L.: Speech emotion recognition using
deep 1D & 2D CNN LSTM networks. Biomed. Signal Process.
Control 47, 312–323 (2019)
11. Carey, M.J.; Parris, E.S.; Lloyd-Thomas, H.: A comparison of fea-
tures for speech, music discrimination. In: 1999 IEEE International
Conference on Acoustics, Speech, and Signal Processing. Proceed-
ings. ICASSP99 (Cat. No. 99CH36258), vol. 1, pp. 149–152. IEEE,
London (1999)
12. Stylianou, Y.: Voice transformation: a survey. In: 2009 IEEE Inter-
national Conference on Acoustics, Speech and Signal Processing,
pp. 3585–3588. IEEE, New York (2009)
13. Wu, Z.; Evans, N.; Kinnunen, T.; Yamagishi, J.; Alegre, F.; Li, H.:
Spoofing and countermeasures for speaker verification: a survey.
Speech Commun. 66, 130–153 (2015)
14. Wu, Z.; De Leon, P.L.; Demiroglu, C.; Khodabakhsh, A.; King,
S.; Ling, Z.-H.; Saito, D.; Stewart, B.; Toda, T.; Wester, M.;
et al.: Anti-spoofing for text-independent speaker verification: an
initial database, comparison of countermeasures, and human per-
formance. IEEE/ACM Trans. Audio Speech Lang. Process. 24(4),
768–783 (2016)
15. Reimao, R.A.M.: Synthetic speech detection using deep neural net-
works. Thesis, York University, Toronto, Ontario (2019)
16. Muckenhirn, H.; Magimai-Doss, M.; Marcel, S.: End-to-end con-
volutional neural network-based voice presentation attack detec-
tion. In: 2017 IEEE International Joint Conference on Biometrics
(IJCB), pp. 335–341. IEEE, New York (2017)
17. Dinkel, H.; Qian, Y.; Kai, Yu.: Investigating raw wave deep neural
networks for end-to-end speaker spoofing detection. IEEE/ ACM
Trans. Audio Speech Lang. Process. 26(11), 2002–2014 (2018)
18. De Leon, P.L.; Hernaez, I.; Saratxaga, I.; Pucher, M.; Yamagishi,
J.: Detection of synthetic speech for the problem of imposture. In:
2011 IEEE International Conference on Acoustics, Speech and Sig-
nal Processing (ICASSP), pp. 4844–4847. IEEE, New York (2011)
19. Ze, H.; Senior, A.; Schuster, M.: Statistical parametric speech syn-
thesis using deep neural networks. In: 2013 IEEE International
Conference on Acoustics, Speech and Signal Processing, pp. 7962–
7966. IEEE, London (2013)
20. Dörfler, M.; Bammer, R.; Grill, T.: Inside the spectrogram: Convo-
lutional neural networks in audio processing. In: 2017 International
Conference on Sampling Theory and Applications (SampTA), pp.
152–155. IEEE, New York (2017)
21. Hong, Yu.; Tan, Z.-H.; Ma, Z.; Martin, R.; Guo, J.: Spoofing
detection in automatic speaker verification systems using DNN
classifiers and dynamic acoustic features. IEEE Trans. Neural
Netw. Learn. Syst. 29(10), 4633–4644 (2017)
22. Balamurali, B.T.; Lin, K.E.; Lui, S.; Chen, J.-M.; Herremans, D.:
Toward robust audio spoofing detection: a detailed comparison
of traditional and learned features. IEEE Access 7, 84229–84241
(2019)
23. Maccagno, A.; Mastropietro, A.; Mazziotta, U.; Scarpiniti, M.; Lee,
Y.-C.; Uncini, A.: A CNN approach for audio classification in con-
struction sites. In: Progresses in Artificial Intelligence and Neural
Systems, pp. 371–381. Springer, Berlin (2019)
123
3458
24. Bai, S.; Kolter, J.Z.; Koltun, V.: An empirical evaluation of
generic convolutional and recurrent networks for sequence model-
ing (2018). arXiv preprint arXiv:1803.01271
25. Zhang, C.; Yu, C.; Hansen, J.H.L.: An investigation of deep-
learning frameworks for speaker verification antispoofing. IEEE
J. Select. Top. Signal Process. 11(4), 684–694 (2017)
26. Paul, D.; Pal, M.; Saha, G.: Spectral features for synthetic speech
detection. IEEE J. Select. Top. Signal Process. 11(4), 605–617
(2017)
27. Kinnunen, T.; Sahidullah, M.; Delgado, H.; Todisco, M.; Evans, N.;
Yamagishi, J.; Lee, K.A.: The ASVspoof 2017 challenge: assessing
the limits of replay spoofing attack detection (2017)
28. Jaderberg, M.; Simonyan, K.; Zisserman, A.; Kavukcuoglu, K.:
Spatial transformer networks. Adv. Neural Inform. Process. Syst.
28, 2017–2025 (2015)
29. Lea, C.; Vidal, R.; Reiter, A.; Hager, G.D.: Temporal convolutional
networks: A unified approach to action segmentation. In: Euro-
pean Conference on Computer Vision, pp. 47–54. Springer, Berlin
(2016)
30. Alqahtani, S.; Mishra, A.; Diab, M.: Efficient convolutional
neural networks for diacritic restoration (2019). arXiv preprint
arXiv:1912.06900
31. Lea, C.; Flynn, M.D.; Vidal, R.; Reiter, A.; Hager, G.D.: Tempo-
ral convolutional networks for action segmentation and detection.
In: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pp. 156–165 (2017)
32. Farha, Y.A.; Gall, J.: MS-TCN: multi-stage temporal convolu-
tional network for action segmentation. In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recog-
nition, pp. 3575–3584 (2019)
123
Arabian Journal for Science and Engineering (2022) 47:3447–3458
33. Tian, X.; Xiao, X.; Chng, E.S.; Li, H.: Spoofing speech detection
using temporal convolutional neural network. In: 2016 Asia-Pacific
Signal and Information Processing Association Annual Summit
and Conference (APSIPA), pp. 1–6. IEEE, London (2016)
34. Chen, Y.; Kang, Y.; Chen, Y.; Wang, Z.: Probabilistic forecasting
with temporal convolutional neural network. Neurocomputing 399,
491–501 (2020)
35. Danilyuk, K.: Convnets series. Spatial transformer networks-
towards data science. Towards Data Sci. (2017)
36. Nagarajan, S.; Nettimi, S.S.S.; Kumar, L.S.; Nath, M.K.; Kanhe,
A.: Speech emotion recognition using cepstral features extracted
with novel triangular filter banks based on bark and ERB frequency
scales. Digit. Signal Process. 104, 102763 (2020)
37. Jia, Y.; Zhang, Y.; Weiss, R.J.; Wang, Q.; Shen, J.; Ren, F.; Chen,
Z.; Nguyen, P.; Pang, R.; Moreno, I.L.; et al.: Transfer learning
from speaker verification to multispeaker text-to-speech synthesis
(2018). arXiv preprint arXiv:1806.04558
38. Dash, T.K.; Mishra, S.; Panda, G.; Satapathy, S.C.: Detection of
COVID-19 from speech signal using bio-inspired based cepstral
features. Pattern Recogn. 117, 107999 (2021)
39. Zheng, F.; Zhang, G.; Song, Z.: Comparison of different imple-
mentations of MFCC. J. Comput. Sci. Technol. 16(6), 582–589
(2001)
40. van den Oord, A.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals,
O.; Graves, A.; Kalchbrenner, N.; Senior, A.; Kavukcuoglu, K.:
Wavenet: a generative model for raw audio (2016). arXiv preprint
arXiv:1609.03499