A A C H - D R: Lternative Pproaches For Omputing Ighest Ensity Egions

A LTERNATIVE A PPROACHES FOR C OMPUTING
H IGHEST-D ENSITY R EGIONS
A P REPRINT
Nina Deliu Brunero Liseo

MEMOTEF, Sapienza Università di Roma (IT) MEMOTEF, Sapienza Università di Roma (IT)
arXiv:2401.00245v1 [stat.ME] 30 Dec 2023
MRC – Biostatistics Unit, University of Cambridge (UK) brunero.liseo@uniroma1.it

nina.deliu@uniroma1.it
A BSTRACT
Many statistical problems require estimating a density function, say f , from data samples. In this
work, for example, we are interested in highest-density regions (HDRs), i.e., minimum volume sets
that contain a given probability. HDRs are typically computed using a density quantile approach,
which, in the case of unknown densities, involves their estimation. This task turns out to be far from
trivial, especially over increased dimensions and when data are sparse and exhibit complex structures
(e.g., multimodalities or particular dependencies). We address this challenge by exploring alternative
approaches to build HDRs that overcome direct (multivariate) density estimation. First, we generalize
the density quantile method–currently implementable on the basis of a consistent estimator of the
density–to neighbourhood measures, i.e., measures that preserve the order induced in the sample
by f . Second, we discuss a number of suitable probabilistic- and distance-based measures such as
the k-nearest neighbourhood Euclidean distance. Third, motivated by the ubiquitous role of copula
modeling in modern statistics, we explore its use in the context of probabilistic-based measures.
An extensive comparison among the introduced measures is provided, and their implications for
computing HDRs in real-world problems are discussed.
Keywords Statistical intervals · Density estimation · Copula models · k-nearest neighborhood
1 Introduction
A ubiquitous problem in statistics is to derive statistical intervals or regions–especially in the multivariate setting–for
population parameters or other unknown quantities. Their role is to provide a way to quantify and describe the
uncertainty about a quantity of interest, or simply a way to summarize the information contained in a distribution.
Statistical regions may address different problems. For example, a confidence interval (CI) describes the uncertainty
surrounding an estimate for an unknown parameter, while a prediction interval provides bounds for one or more
future observations. Alternatively, a tolerance interval is the interval expected to contain a specified proportion of
the sampled population. In a Bayesian setting, highest posterior density or credible regions provide, in a natural way,
set estimates for a specific parameter (Box and Tiao, 1992; Turkkan and Pham-Gia, 1993). We refer to Meeker et al.
(2017) and Krishnamoorthy and Mathew (2009) for an overview. Furthermore, even when the focus is on a specific
type of interval, e.g., a two-sided 95% prediction interval, questions on the way this should be defined may still arise.
Should we use the interval symmetric about the mean, the interval of shortest length, or the interval that minimizes
the probability of covering a given set? Although each of these intervals has 95% coverage, they may all be different.
Consider, for example, the case of nonsymmetric or/and multimodal distributions, with an illustration given in Figure 1.
An additional layer of complexity arises in multivariate settings, where there are no unique agreed definitions, and
generalizations include the concept of simultaneous CIs (Guilbaud, 2008), multivariate CIs (Korpela et al., 2017), or
different definitions for multivariate quantiles (Figalli, 2018; Cai, 2010; Coblenz et al., 2018), among others.
The multivariate setting will be the target of this work, with a focus on bivariate distributions. In particular, our interest
is devoted to statistical regions for summarizing probability distributions in the form of highest-density regions (HDRs;
Hyndman, 1996). Statistical regions other than HDRs are beyond the scope of the present work. As the name suggests,
Alternative Approaches for Computing HDRs A P REPRINT
an HDR specifies the set of points of highest density: The density for points inside the region must be higher than that for
points outside it. More specifically, considering a d-dimensional continuous variable of interest X ∈ Rd with probability
density function f , the concrete problem is to estimate minimum volume sets of the form C(fα ) = {x : f (x) ≥ fα },
such that P (X ∈ C(fα )) ≥ 1 − α, where 1 − α, with α ∈ (0, 1), represents a prespecified coverage probability.
Although they share substantial similarities with multivariate quantiles, estimating an HDR differs from estimating
level sets in that one is interested in specifying a probability content rather than the level directly. This complicates the
problem, and we refer to Doss and Weng (2018) for more details.
The scope of an HDR can be wide and diverse; the following are possible applications.
Forecasting To obtain a prediction or forecast region for a set of observable variables in order to inform the most likely
future realizations and convey in a simple way the accuracy of a forecast (for illustrative examples, see e.g.,
Hyndman, 1996; Kim et al., 2011).
Anomaly detection To detect abnormal observations from a sample: if a data point does not belong to a region of
“normal” or concentrated data (the HDR), then it is regarded anomalous (see e.g., Steinwart et al., 2005, and
references therein).
Unsupervised or semi-supervised classification To identify areas or clusters with a relatively high concentration of a
given phenomenon; see, e.g., the work of Saavedra-Nieves (2022), aimed at finding areas of high incidence of
coronavirus.
Due to their flexibility “to convey both multimodality and asymmetry”, HDRs are argued to be a more effective summary
of the distribution (Hyndman, 1995). In the case of unimodal symmetric distributions, such as the normal distribution,
an HDR coincides with the usual probability region symmetric about the mean, spanning the α/2 and 1 − α/2 quantiles.
However, in the case of a multimodal distribution, it may consist of several disjoint subregions, each containing a
local mode. This provides useful information that is “masked” by other probability regions. An illustrative example is
provided in Figure 1, where an HDR is compared to an equal-tailed interval.
Statistical interval
0.25
100(1 − α)% HDR

[qα 2, q1−α 2]
0.20
Coverage = 100(1 - α)% = 95%

0.15
0.10
0.05
fα
0.00
-5 0 5 10
Figure 1: Comparison between two 100(1 − α)% probability intervals for a normal mixture density: an HDR and an
equal-tailed interval. The coverage parameter α is set to 0.05.
In principle, knowledge of an accurate estimator of the underlying probability density function f of a variable X is
sufficient to compute its HDRs. However, existing methods for estimating f , such as the kernel density estimator (KDE;
Parzen, 1962) or the local likelihood approach (Hjort and Jones, 1996), work very well for unidimensional problems,
but may be inefficient for multidimensional problems (Liu et al., 2007). In fact, over increased dimensions, KDE
suffers from the difficulty of finding optimal kernel functions and the corresponding bandwidths (i.e., the smoothing
parameters). In particular, bandwidth selection in KDE is recognized as the most crucial and difficult step (see e.g.,
Chapter 2 in Wand and Jones, 1994a), with no definite and broadly accepted solution. As an illustrative example,
in Figure 2 we estimate a 95% HDR for a bivariate Gaussian distribution using KDE and three different bandwidth
specifications. These are given in the works of Silverman (1996)–in the form of rule of thumb, in Wand and Jones
(1994b)–currently employed in the ks and hdrcde packages in R, and in Chacón et al. (2011)–under optimal asymptotic
2
considerations. As depicted in Figure 2, the three different values result in three different HDRs, with an open question
on the most appropriate one. In particular, if the task would be to identify abnormal or anomalous values, contradicting
95% HDR with KDE

h = (0.42, 0.42); Wand & Jones, 1994 h = (0.96, 0.96); Silverman, 1986 h = (1.12, 1.12); Chacon et al., 2011
6
6
4
4
2
2
X2
X2
X2
0
0
-2
-2
-2
-4
-4
-4
-6
-6
-6
-5 0 5 -5 0 5 -5 0 5
X1 X1 X1
Figure 2: Estimated 95% HDR on a sample of n = 200 units using KDE with three bandwidth specifications. Cadet-blue
points are the HDR points, while purple x-marks and red triangles denote the points outside the HDR. The blue contour
line represents the true level set (i.e., the one delimiting a 95% density region) of a bivariate Gaussian distribution (see
Scenario S2 in Section 4).
conclusions would arise from the three different HDRs (see e.g., the red triangle points in Figure 2). More concretely,
our analysis is directly motivated by, and sees our involvement in, the World Anti-Doping Agency (WADA) mission of
doping detection.
Furthermore, high-dimensional data also pose challenges from an algorithmic/computational perspective. Taken
together, these aspects hamper the ability to derive an appropriate HDR. Motivated by these existing density-estimation
issues, our research question is the following: “Can we relax the multivariate density estimation problem, either by
using alternative ways (to direct density estimation, e.g., using distance-based proxies) that will robustly inform the
density around a given point or by reducing the multidimensional estimation problem to a lower dimension?”. We
address this question by exploring, evaluating, and proposing some novel approaches to building HDRs.
The contribution of the present work is fourfold. First, we generalize the density-quantile approach for computing
HDRs–currently implementable on the basis of a consistent estimator of the density–to neighbourhood measures (Munoz
and Moguerza, 2006), i.e., measures that preserve the order induced in the sample by the density function (Section 2).
Second, we elaborate on and evaluate a number of suitable probabilistic- and distance-based measures, including a
variation of the measure adopted by the k-nearest neighbors algorithm (Section 3). Third, motivated by the ubiquitous
role of copula modeling (Nelsen, 2006) in modern statistics, among probabilistic-based measures, we explore the use of
copulae in an HDR estimation context. This is done in Section 3.4. Specifically, copulae make it possible to relax the
estimation of multivariate random vectors, by separately estimating the marginals and their dependence structure, i.e.,
the copula model. In addition, by placing a strong focus on the dependence model, a copula approach has the advantage
of better capturing data specificities, especially when these exhibit complex relationships such as asymmetric and/or tail
dependencies. Finally, in Section 4, we provide a comprehensive comparison between the introduced measures and
standard density estimators in all simulated scenarios, as well as in the MAGIC data set, which classifies high-energy
Gamma particles in the atmosphere (Bock et al., 2004). Empirical studies are focused on bivariate scenarios that vary
according to the complexity of the data (marginal and dependence structure, multimodality, etc) and the sample size. A
distribution on a compact support, namely, the Dirichlet distribution, is also considered. We conclude by summarizing
the main findings and discussing their implications for computing HDRs in real-world problems.
2 HDR: Formulation and Computation

Let {X1 , . . . , Xn } be a sample of n independent and identically distributed (iid) replications of a random variable X
defined on Rd , with d ≥ 1. In our problem, we assume to have access to a sample sn = {x1 , . . . , xn } ∈ Sn of their
actual realizations, which we wish to use to build an HDR, that is, a statistical region containing those sample values of
relatively high density (see Definition 1). We denote by Xij (xij ) the j-th component of Xi (xi ), j = 1, . . . , d and
i = 1, . . . , n. We restrict our discussion to continuous random variables X and, unless otherwise stated, we denote by
f their probability density function (PDF) and by F their cumulative density function (CDF).
3
Definition 1 (Highest-density region; Hyndman (1996)) Denote by f the PDF of a continuous, possibly multivariate,
random variable X ∈ Rd ; the 100(1 − α)% HDR is defined as the subset C(fα ) of the sample space of X such that:
C(fα ) = {x : f (x) ≥ fα },
where fα is the largest constant such that P (X ∈ C(fα )) ≥ 1 − α, with α ∈ (0, 1).
One of the most distinctive properties of HDRs is that, among all regions of probability coverage 100(1 − α)%, the
HDR has the smallest possible volume. The notion of “smallest” is to be understood with respect to some measure such
as the usual Lebesgue measure; in the continuous one-dimensional case that would lead to the shortest-length set, while
in two dimensions that would be the smallest-area set. It also follows from the definition that the boundary of an HDR
consists of those values of the sample space with equal density. Hence a plot of a bivariate HDR has as the boundary a
contour plot (see Figure 2).
2.1 Density quantile approach
The study of HDRs has been largely enhanced by Hyndman, who proposed the density quantile approach (outlined in
Proposition 1) to compute multivariate HDRs (Hyndman, 1996). Today, this still represents the typical strategy and
involves estimating the density.
Proposition 1 (Hyndman (1996)) Let {f (x1 ), . . . , f (xm )} be a sample of independent observations of size m of
the random variable Y = f (X), with f a bounded and continuous function in x. Consider the ordered sample
{f(1) , . . . , f(m) } with f(j) the j-th largest among the f (xi )’s so that f(j) is the (j/m) sample quantile of Y . Then,
given a constant α ∈ [0, 1], and denoted with ⌊j⌋ the greatest integer less than or equal to j, we have that, in probability,
.
fˆα = f⌊αm⌋ → fα as m → ∞,
.
Cm (fˆα ) = C(x : f (x) > fˆα ) → C(fα ) as m → ∞.
Basically, the HDR is derived based on the sample quantile of the density f . However, the density function itself is
.
often unknown and we have to estimate it based on a set of available iid observations sn = {x1 , . . . , xn }. In this case,
the 100(1 − α)% HDR can be estimated as
.
Ĉn (fˆα ) = C(x : fn (x) > f⌊αn⌋ ), (1)
with fn being a possibly consistent estimator of f . Note that for small n, it may not be possible to get a reasonable
density estimate. Also, with few observations and no prior knowledge on the underlying density function, there seems
to be little point in attempting to summarize the density.
2.2 One-class neighbor machines (OCNM) approach
An alternative approach to HDR estimation, inspired by the theory of support vector machines (Schölkopf et al., 2001)
and the notion of neighborhood measures, has been introduced in the machine learning literature by Munoz and
Moguerza (2006). The procedure is outlined in Proposition 2, after introducing neighborhood measures in Definition 2.
Definition 2 (Neighbourhood Measure; Munoz and Moguerza (2006)) Let X be a random variable with density
function f defined on Rd . Denoted by Sn the set of random iid samples sn = {x1 , . . . , xn } of size n (drawn from f ),
the real-valued function g : Rd × Sn → R is a neighbourhood measure if one of the following holds:
(a) f (x) < f (y) −→ lim P (g(x, sn ) > g(y, sn )) = 1 x, y ∈ sn , ∀sn ∈ Sn ,
n→∞
(b) f (x) < f (y) −→ lim P (g(x, sn ) < g(y, sn )) = 1 x, y ∈ sn , ∀sn ∈ Sn .
n→∞
The function g is called either a (a) sparsity or a (b) concentration measure.
Proposition 2 (Munoz and Moguerza (2006)) Consider an iid sample sn = {x1 , . . . , xn } and a sparsity measure
g. Define ρ∗ = g(x(νn) , sn ), with x(νn) being the (νn)-th sample in the order induced in sn by g, provided that
νn ∈ N; otherwise, the least integer greater than νn, denoted by ⌈νn⌉, is taken. Then, the binary decision function
h(x) = sign(ρ∗ − g(x, sn )) is such that:
n
!
1X
P I(h(x) ̸= −1) = ν → 1 as n → ∞.
n i=1
4
where I denotes the indicator function, and

. .
CnOCNM = {x : h(x) ≥ 0} → C(fν ) = {x : f (x) ≥ fν } as n → ∞,
where C(fν ) is the minimum-volume set such that P(C(fν )) ≥ ν.
Notably, the OCNM algorithm relaxes the HDR estimation problem in the following sense: Instead of estimating
and evaluating the density f , a more general and potentially simpler measure g that asymptotically preserves the
order induced by the density, can be considered. It is, however, important to remark that the quality of the estimation
procedure heavily depends on using a neighborhood measure. If the measure used is neither a concentration nor a
sparsity measure, there is no reason why the method should work.
2.3 Neighborhood-quantile approach
We now discuss a hybrid approach that can be viewed as: i) a generalization of the density quantile approach and; ii)
a restatement of the OCNM method directly in terms of its solution ρ∗ . Compared to the former, it ensures a wider
applicability allowing for a more general set of functions, including the density f as a special case; with respect to the
latter, it offers a more direct, interpretable, and computationally efficient method.
Theorem 1 Let X be a continuous random variable defined on Rd with density function f , and consider a set
sn = {x1 , . . . , xn } ∈ Sn of size n (drawn from f ). Assume g : Sn × Rd → R is a neighbourhood measure. Then an
estimate of the 100(1 − α)% HDR can be obtained as:
Cn = {x : g(x, sn ) ≤ g(x⌈(1−α)n⌉ , sn )} if g is a sparsity measure,
Cn = {x : g(x, sn ) ≥ g(x⌊αn⌋ , sn )} if g is a concentration measure,
where g(x⌊αn⌋ , sn ) is the α-quantile of the sample {g(x1 , sn ), . . . , g(xn , sn )}, and ⌊j⌋ denotes the greatest integer
less than or equal to j.
The proof is straightforward when considering the relationship between the region CnOCNM as defined in Proposition 2
and the region Cn as defined in Theorem 1. In fact, without loss of generality, taken g to be a sparsity measure, and
noticing that ν plays the role of 1 − α, we can observe that:
CnOCNM = {x : h(x) = sign(ρ∗ − g(x, sn )) ≥ 0}
= {x : sign(g(x⌈νn⌉ , sn ) − g(x, sn )) ≥ 0}
= {x : g(x, sn ) ≤ g(x⌈(1−α)n⌉ , sn )}
= Cn .
Remark If g is chosen to be a concentration measure, then to ensure a 100(1 − α)% coverage (notice that in this case
P(C(fν )) = 1 − ν), the decision value ρ∗ induced by the concentration measure is given by ρ∗ = g(x⌊αn⌋ , sn ).
Noticing that the density represents a concentration measure, the resemblance with the density quantile approach in
Proposition 1 should now be clear. In fact, from the perspective of the density quantile approach, we can view Theorem 1
as a generalization of Proposition 1, where f is replaced by any function g that satisfies the criteria of neighborhood
measures. Intuitively, provided that the density function f is replaced by a function that preserves the order induced in
the sample by f , the estimated HDR is asymptotically valid. Neighborhood measures ensure this ranking. To see it,
without loss of generality, consider a sparsity measure g and a sample sn = {x1 , . . . , x(1−α)n , . . . , xn } ordered so that
f (x1 ) < · · · < f (x(1−α)n ) < · · · < f (xn ),
where, for simplicity, we suppose (1 − α)n ∈ N and f (xj ) ̸= f (xj ) for all i ̸= j. From Definition 2 (a),
for each pair (xi , xj ), with i < j, it holds that P (g(x i , sn ) > g(xj , sn )) → 1. Hence, given ϵ ∈ (0, 1), there
exists nij ∈ N such that P g(xi , snij ) > g(xj , snij ) > 1 − ϵ. Taking n ≥ max{nij }, it is guaranteed that
P g(x1 , sn ) > · · · > g(x(1−α)n , sn ) > · · · > g(xn , sn ) > 1 − ϵ. Therefore, as n → ∞

P g(x1 , sn ) > · · · > g(x(1−α)n , sn ) > · · · > g(xn , sn ) → 1.
Remark Proposition 1 represents a special case of Theorem 1 with g(x, sn ) being either: (a) a concentration measure
g(x, sn ) ∝ fˆ(x, sn ) or; (b) a sparsity measure g(x, sn ) ∝ fˆ(x,s
1
, where fˆ can be any consistent density estimator.
n)
Among the plethora of density estimators, in this work we focus on KDE (Parzen, 1962), and use it as a benchmark
5
.
measure for the proposed comparators in Section 3. Specifically, given a set of iid observations sn = {x1 , . . . , xn }
drawn from an unknown target density f , the KDE measure M0 (x, sn , h) at the location x is defined as
n n
1X 1 X ∥x − xi ∥
M0 (x, sn , h) = Kh (∥x − xi ∥) = K , (2)
n i=1 nhd i=1 h
where K : Rd → R denotes the kernel function, satisfying K(x)dx = 1 and K(x) ≥ 0, ∀x, and h > 0 the bandwidth
R
hyperparameter. Details on the chosen kernel and bandwidth value will be given in Section 4.
3 Alternative Measures for Building HDRs

We now discuss a number of neighborhood measures that could be used to estimate HDRs. We repurpose some measures
that have been successfully employed in areas such as classification or clustering, and introduce some novel ideas,
included a copula-based approach.
3.1 kNN-Euclidean distance
This corresponds to the sum of all Euclidean distances of a given point x from its k-nearest neighbors. The concept of
closeness (“nearest”) is defined according to the Euclidean metric or L2 -norm denoted by ∥·∥2 .
Definition 3 (kNN-Euclidean distance) Given a data point x ∈ Rd of a sample set sn of size n, and an integer
k ∈ [1, n], we define the k-nearest neighborhood Euclidean distance of point x from sample sn as:
k
. X
M1 (x, sn , k) = ∥x − x(i) ∥2 ,
i=1
where x(i) denotes the i-th observation in the reordered data such that ∥x − x(1) ∥2 ≤ · · · ≤ ∥x − x(i) ∥2 ≤
. . . ∥x − x(n) ∥2 .
M1 (x, sn , k) is entirely based on a metric distance, that is, the Euclidean metric, and represents a sparsity measure.
The property follows from the convergence in probability of the k-nearest neighbor density estimator (Silverman, 1996).
Similar distances have been used in the literature for nonparametric classification and regression, within the so-called
k-nearest neighbors (kNN) algorithm. Notably, the distance to the nearest neighbors can also be seen as a local density
estimate (Loftsgaarden and Quesenberry, 1965; Silverman, 1996) and as a special case of a variable-bandwidth KDE
with a uniform kernel (Terrell and Scott, 1992). A large kNN distance indicates that the density is usually small and
vice versa. Furthermore, by ranking each data point according to the distance from its k-nearest neighbors, this measure
can also be used as an outlier score in anomaly detection (Ramaswamy et al., 2000).
Compared to other methods, the kNN approach has several advantages such as: (i) being purely nonparametric, hence
able to flexibly adapt to any continuous distribution; (ii) having a reasonable time complexity; (iii) depending on a
unique hyperparameter k, whose tuning is relatively simple. The choice of k should be made based on the sample
data. Generally, higher values of k reduce the effect of noise; however, they underfit the model–making boundaries
between classes less distinct–and
√ are computationally more expensive, especially for large d. A general rule of thumb
in classification is k = ⌊ n⌋, with n the number of samples in the dataset. Further investigation of the role of k
in different settings and how it relates to the missclassification error is given in the Supplementary Material B and
in (Meeker et al., 2017).
3.2 kNN-CDF distance
Although the Euclidean distance and other general notions of measures (including concepts like area or volume) can
certainly be useful for capturing relevant insights on the topology of a given set, they may be equipped with a probability
measure to better resemble the notion of density. Let P be a probability measure and denote by F its associated CDF
and by Fi the CDF of the i-th marginal.
Given two data points x1 = (x11 , . . . , x1d ) ∈ Rd and x2 = (x21 , . . . , x2d ) ∈ Rd from the sample set sn , a preliminary
version of the CDF distance between the two data points, denoted by dP (x1 , x2 ), has been defined in Venturini (2015)
in the context of clustering problems as:
v
u d
uX 2
dP (x1 , x2 ) = t (Fj (x1j ) − Fj (x2j )) . (3)
j=1
6
1
Pn
In the typical case of unknown marginal CDFs, their empirical counterpart Fn,j (t) = n i=1 I(xij ≤ t), j = 1, . . . , d,
could be considered.
The distance dP can be interpreted as the composition of the (nonlinear) transformation F : Rd → [0, 1] and the
computation of the ordinary Euclidean distance. It fulfills all the properties of being a proper metric. Furthermore, it
has the nice property that the distance between two points is proportional to the probability contained between the
two points. However, as defined in Eq. (3), it presents some key limitations for inferring the underlying density and
estimating HDRs. Consider for simplicity the univariate Gaussian mixture model illustrated in Figure 3 and the set
of points x1 = −2, x2 = −1.6, x3 = −0.72, x4 = 0.5. When evaluating the CDF distance between x2 and all other
points, we have that dP (x2 , x1 ) ≈ dP (x2 , x3 ), but there is little to say about the density around x1 and x3 , apart from
realizing that they are clearly different. We could gain more information if we were to compute the CDF distance from
the global mode x1 , assuming we have this information, noticing that the higher dP (x1 , xi ), for all i, the lower the
density of xi . However, this is no longer verified for multimodal densities. In fact, we have dP (x1 , x4 ) > dP (x1 , x3 ),
but f (x4 ) > f (x3 ), contrasting with the definition of a neighborhood measure.
0.4
0.3
PDF
0.2
0.1
0.0
-4 -3 x1 = -2 x2 = -1.6 x3 = -0.72 x4 = 0.5 1 2 3 4
Figure 3: Illustration of the CDF-distances computed on the set of points x1 = −2, x2 = −1.6, x3 = −0.72, x4 = 0.5
based on a Gaussian mixture model.
Motivated by these limitations and inspired by the idea of the kNN distance in Definition 3, we propose a new variant
consisting of the sum of CDF-distances between a point x and its k-nearest neighbors. Here, “nearest” should again be
understood according to the Euclidean distance. The rationale is simple: the higher the CDF distances between a point
and its neighbors, the higher we expect the density to be for that point. Since the k neighbors of each data point will be
at different (Euclidean) distances compared to the k neighbors of the other data points, we want to properly scale or
weight the CDF distances according to this information. The proposed measure is reported in Definition 4.
Definition 4 (kNN-CDF distance) Given Fi , the i-th marginal of a CDF F , for i = 1, . . . , d, and an integer k ∈ [1, n],
we define the k-nearest neighborhood CDF distance as:
 qP
k k d 2
dP (x,x(i) ) j=1 (Fj (xj )−Fj (x(i)j ))

P P
= k>1
M2 (x, sn , k) = i=2 ∥x−x(i) ∥2
q Pd 2
i=2 j=1 (xj −x(i)j )

0 k = 1,
where x = (x1 , . . . , xd ) ∈ Rd , and x(i) = (x(i)1 , . . . , x(i)d ) is the i-th observation in the sample sn such that
∥x − x(1) ∥2 ≤ · · · ≤ ∥x − x(i) ∥2 ≤ . . . ∥x − x(n) ∥2 .
Property 1 (Semimetric) Consider the M2 measure as defined in Definition 4. It can be easily verified that, for any
k ∈ [1, n], M2 (x, x(k) , k) : Rd × Rd → R is a semimetric. In fact, for any set x, x(k) ∈ Rd , we have that:
• If x = x(k) , M2 (x, x(k) , 1) = 0 by definition.

• If x ̸= x(k) , M2 (x, x(k) , k) > 0. In fact, (xj − x(k)j )2 > 0 and (Fj (xj ) − Fj (x(k)j ))2 > 0 for all j and all
k, when x ̸= x(k) and X is continuous.
• M2 (x, x(k) , k) = M2 (x(k) , x, k), as (xj −x(k)j )2 = (x(k)j −xj )2 and (Fj (xj )−Fj (x(k)j ))2 = (Fj (x(k)j )−
Fj (xj ))2 for all j and all k.
7
The triangular inequality, stating that M2 (x, x(k) , 1) ≤ M2 (x, x(k′ ) , 1)+M2 (x(k′ ) , x(k) , 1), for all x, x(k) , x(k′ ) ∈ Rd ,
does not hold. It is enough to consider a simple counterexample such as: d = 1, F the CDF of a standard normal
distribution, and three points x = 1 < x(k) = 2 < x(k′ ) = 3.
3.3 ϵ-neighborhood multivariate CDF distance
The measure M2 (x, sn , k) introduced in Section 3.2 has several advantages; for example: (i) it takes into account the
probabilistic information in the data, (ii) it is based on marginal CDFs which may be easy to estimate, and (iii) it has
convenient computational times. However, using only the marginal CDFs may compromise their validity whenever the
marginal components of a multivariate random variable X ∈ Rd show a dependence structure. In that case, the use of a
multivariate CDF may be more appropriate. If the multivariate CDF is unknown, we could use its empirical counterpart.
Notice, however, that this estimation part may compromise computational efficiency, especially in high dimensions and
for large values of k. We therefore focus on an ϵ-neighborhood version, which reduces the number of operations from k
to 1, once ϵ is determined.
Definition 5 (ϵ-neighborhood multivariate CDF distance) Given the CDF F of a multivariate variable X ∈ Rd , we
define the ϵ-neighborhood multivariate CDF distance of a point x = (x1 , . . . , xd ) to be the probability of that point
belonging to a hyperrectangle of dimension d defined on vertexes [x1 − ϵ1 , x1 + ϵ1 ] × · · · × [xd − ϵd , xd + ϵd ] scaled
by hyperrectangle’s d-volume:
n(ν)
P
P(X ∈ [x − ϵ, x + ϵ]) ν∈V (−1) F (ν) X
M3 (x, ϵ) = Qd = Qd ∝ (−1)n(ν) F (ν), (4)
j=1 2ϵj j=1 2ϵj ν∈V
Pd
where ν = (ν1 , . . . , νd ), with νj ∈ {xj − ϵj , xj + ϵj }, for j ∈ 1, . . . , d, and n(ν) = j=1 I(νj = xj − ϵj ). The sum
is computed over the 2d vectors of the set V.
As in the case of the density f itself, when the CDF F is unknown, a consistent estimator, say Fn , may be used leading
to:
(−1)n(ν) Fn (ν)
P
X
M3 (x, sn , ϵ) = ν∈VQd ∝ (−1)n(ν) Fn (ν).
j=1 2ϵj ν∈V
Relationship in Eq. (4) can be derived by recursion starting from small d values. For d = 2, for example, it is easy to
verify that
M3 (x, ϵ) ∝ P(X ∈ [x − ϵ, x + ϵ])
= P(X1 ∈ [x1 − ϵ1 , x1 + ϵ1 ], X2 ∈ [x2 − ϵ2 , x2 + ϵ2 ])
= P(X1 ∈ [x1 − ϵ1 , x1 + ϵ1 ], X2 ≤ x2 + ϵ2 ) − P(X1 ∈ [x1 − ϵ1 , x1 + ϵ1 ], X2 < x2 − ϵ2 )
= P(X1 ≤ x1 + ϵ1 , X2 ≤ x2 + ϵ2 ) − P(X1 < x1 − ϵ1 , X2 ≤ x2 + ϵ2 )
− P(X1 ≤ x1 + ϵ1 , X2 ≤ x2 − ϵ2 ) − P(X1 < x1 − ϵ1 , X2 < x2 − ϵ2 )
= F (x1 + ϵ1 , x2 + ϵ2 ) − F (x1 − ϵ1 , x2 + ϵ2 ) − F (x1 + ϵ1 , x2 − ϵ2 ) + F (x1 − ϵ1 , x2 − ϵ2 )
X
= (−1)n(ν) F (ν),
ν∈V
with V = {(x1 + ϵ1 , x2 + ϵ2 ), (x1 − ϵ1 , x2 + ϵ2 ), (x1 + ϵ1 , x2 − ϵ2 ), (x1 − ϵ1 , x2 − ϵ2 )}.

We now state the following result for this measure.
Property 2 (Density equivalence) Consider the M3 measure as defined in Definition 5 and assume that F is differen-
tiable at any point x of its support. Then, as ϵ → 0, M3 (x, ϵ) → f (x), for any x ∈ Rd .
The proof follows from basic probability and calculus theory.
3.4 Copula-based measures
A d-dimensional copula C : [0, 1]d → [0, 1] is a CDF with uniform marginal distribution functions (Nelsen, 2006). If
we consider a random vector X = (X1 , . . . , Xd ) with joint CDF F and marginals F1 , . . . , Fd , then the copula of X is
8
represented by the joint distribution of F1 (X1 ), . . . , Fd (Xd ) and is derived as:

C(u1 , . . . , ud ) = P(F1 (X1 ) ≤ u1 , . . . , Fd (Xd ) ≤ ud )
= P(X1 ≤ F1−1 (u1 ), . . . , Xd ≤ Fd−1 (ud ))
= F (F1−1 (u1 ), . . . , Fd−1 (ud )).
.
Letting uj = Fj (xj ), this yields the following well-known result due to Sklar (1959):
F (x1 , . . . , xd ) = C(F1 (x1 ), . . . , Fd (xd )), for all x = (x1 , . . . , xd ) ∈ Rd . (5)
Furthermore, when the random vector X is continuous with density f , we also have that

f (x1 , . . . , xd ) = c F1 (x1 ), . . . , Fd (xd ) × f1 (x1 ) × · · · × fd (xd ), (6)

where c is the density of the random vector F1 (x1 ), . . . , Fd (xd ) ∈ [0, 1]d .
In summary, we can decompose every d-dimensional CDF F or PDF f into a composition of their marginal distribution
functions and a d-copula. This allows us to redefine both M0 in Eq. (2) and M3 in Definition 5 in terms of their copula
representation. For example, in the case of the M3 measure, we may construct its copula-based alternative given in
Definition 6.
Definition 6 (ϵ-neighborhood copula-based CDF distance) Given the CDF F of a random vector X =
(X1 , . . . , Xd ), we define the ϵ-neighborhood copula-based CDF distance of point x = (x1 , . . . , xd ) to be
(−1)n(ν) C(F1 (ν1 ), . . . , Fd (νd ))
P
Cop
X
M3 (x, ϵ) = ν∈V Qd ∝ (−1)n(ν) C(F1 (ν1 ), . . . , Fd (νd )), (7)
j=1 2ϵ j ν∈V
Pd
where ν = (ν1 , . . . , νd ), with νj ∈ {xj − ϵj , xj + ϵj }, for j ∈ 1, . . . , d, and n(ν) = j=1 I(νj = xj − ϵj ).
For d = 2, it is straightforward to see that

M3Cop (x, ϵ) ∝ C (F1 (x1 + ϵ1 ), F2 (x2 + ϵ2 )) − C(F1 (x1 − ϵ1 ), F2 (x2 + ϵ2 ))
− C(F1 (x1 + ϵ1 ), F2 (x2 − ϵ2 )) + C(F1 (x1 − ϵ1 ), F2 (x2 − ϵ2 )).
The main advantage of this representation over the one involving the joint CDF is that the estimation of the multivariate
distribution–when unknown–is performed through the estimation of the (univariate) marginals, evading thus the curse
of dimensionality (see e.g., Nagler and Czado, 2016). Furthermore, copulae offer a flexible framework that captures
complex dependency structures while offering direct control over the marginals.
4 Empirical Evaluation
4.1 Simulation setup
We now evaluate and compare the proposed measures in an extensive number of simulated bivariate settings that vary
according to the complexity of the data, as defined below.
(i) We consider different copula models for the dependence structure among marginals, all sharing the same
degree of dependence, expressed via Kendall’s τ , and set equal to 0.5. The list of different copulae follows.
Gauss
Cθ=0.7 Gaussian family (Elliptical class) with zero tail dependence and radial symmetry.
t
Cθ=0.7,ν=6 Student-t family (Elliptical class) with radial symmetry.
Frank
Cθ=5.75 Frank family (Archimedian class) with radial symmetry.
Clay
Cθ=2 Clayton family (Archimedian class) not restricted to radial symmetry.
(ii) Each of the above copulae is then combined with different simulation schemes for the marginals, according to
the following details. Fixing σ 2 = 2 and w1 = 1 − w2 = 0.5, and taking µ11 = 0, µ12 = 9, µ21 = 1, µ22 = 8,
we consider the following.
Unimodal–heavy tails Student-t model Xi ∼ tν=2 , i = 1, 2.
Unimodal Gaussian model Xi ∼ N (µi1 , σ 2 ), i = 1, 2.
9
P2
Bimodal Gaussian X1 ∼ N (µ11 , σ12 ) & Gaussian mixture X2 ∼ k=1 wk N (µ2k + 5I(k = 2), σ 2 ).
P2
Quadrimodal Gaussian mixture Xi ∼ k=1 wk N (µik , σ 2 ), i = 1, 2.
(iii) We also consider a distribution on a compact supports, that is, the Dirichlet distribution defined on the (K − 1)-
simplex, with K = 3 and with parameters α = (1, 1, 2). In this case, the dependence structure is uniquely
determined and its closed-form copula expression is given in Section 4.4.1.
(iv) We also vary the sample size, with n ∈ {50, 100, 500, 1000}.
In total, 17 different scenarios are considered, named S1 to S17. Their graphical representation, in the form of contour
plots, is provided in Figure 4 (S1-S16) and Figure 5 (S17).
Unimodal: Unimodal: Bimodal: Quadrimodal:

Student-t marginals Gaussian marginals Gaussian & GaussianMixture marginals GaussianMixture marginals
15
6
6
Gaussian copula (θ = 0.7)
10
4
10
2
5
0
5
-2
-2
S1 S2 S3 S4
0
-4
-4
-5
-6
-6
-6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6 -5 0 5 10 15
Student-t copula (θ = 0.7; ν = 6)
15
6
10
4
10
2
5
0
5
-2
-2
S5 S6 S7 0 S8
-4
-4
-5
-6
-6
-6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6 -5 0 5 10 15
6
6
Frank copula (θ = 5.75)
10
10
4
4
2
5
0
0
-2
-2
S9 S10 S11 S12

-4
-4
-5
-5
-6
-6
-6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6 -5 0 5 10 15
6
6
Clayton copula (θ = 2)
10
10
4
4
2
5
0
0
-2
-2
S13 S14 S15 S16

-4
-4
-5
-5
-6
-6
-6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 -5 0 5 10 15
Figure 4: Evaluated scenarios with their contour plots at different levels, included the one delimiting the 95% HDR
(blue color).
10
4.2 Evaluated measures
In all settings, we employ the proposed neighborhood-quantile method (see Section 2.3) on data samples of different
sizes generated according to the aforementioned scenarios. For each scenario, we evaluate the following eight methods,
with full details on their hyperparameter tuning reported in the Supplementary Material B.
M0 :KDE Direct estimation of the bivariate density. We use KDE, with Gaussian kernel and bandwidth selection
based on the asympotically optimal solution proposed in Chacón et al. (2011), where its adequacy is shown in
general settings, including Gaussian mixture models.
M0NPCop :DE Nonparametric indirect density estimation with copula. We use standard KDE with the same optimal
bandwidth of Chacón et al. (2011) for the univariate marginals, and KDE with the transformation local
likelihood estimator and nearest-neighbor bandwidth for the copula density. We refer to Nagler and Czado
(2016) for details.
M0PCop :DE Parametric indirect density estimation with copula. We adopt a fully parametric approach (with maximum
likelihood fitting) to estimate both marginals and the copula. For the copula model, we select the best model
using the AIC criterion; no misspecification is introduced for the marginals.
M1 :kNN-Eucl Cumulative Euclidean distances from the kNNs. We sum the Euclidean distances between each point
and its k neighbors defined according to the Euclidean metric. The choice of the hyperparameter k is based on
an extensive
hp icross-validation procedure, leading to a general rule of thumb aligned with the existing literature:
k= n/2 , with [x] the integer closest to x.
M2 :kNN-CDF Cumulative CDF distances from the kNNs. We sum the CDF distances between each point and its k
neighbors, which are defined according to the Euclidean metric. The measure follows a univariate approach
as detailed in Section 3.2. The choice of the hyperparameter k is based on an extensive cross-validation
procedure, suggesting a uniform choice across different scenarios and sample sizes: k = 30.
M3 :ϵ-CDF This represents the ϵ-neighborhood multivariate CDF distance introduced in Section 3.3. The empirical
CDF is used as the CDF estimator. The optimal choice of the hyperparameter ϵ is based on an extensive
cross-validation procedure, leading to the following heuristic for S1-S16 as a result of an exponential decay
model fit with respect to the sample size: ϵ = exp (2.13 − 0.3 log(n)).
M3NPCop :ϵ-CDF Fully nonparametric indirect estimation of the ϵ-CDF measure with copula. We use the empirical CDF
for estimating the univariate marginals, and KDE with the transformation local likelihood estimator and nearest-
neighbor bandwidth for the copula density. The optimal choice of the hyperparameter ϵ follows the same
strategy as the previous measure, leading to the following heuristic for S1-S16: ϵ = exp (1.74 − 0.26 log(n)).
M3PCop :ϵ-CDF Parametric indirect CDF estimation with copula. We adopt a fully parametric approach (with maximum
using the AIC criterion; no misspecification is introduced for the marginals. The optimal choice of the
hyperparameter ϵ follows the heuristic ϵ = exp (1.60 − 0.41 log(n)) for S1-S16.
Exception made for the measures depending on ϵ (with considerations deferred to Section 4.4.1), the same hyperparam-
eter choices are adopted in S17.
4.3 Performance metrics
The measurement of the performance of an HDR estimator can be closely related to the specific problem of interest.
In this work, for example, our main interest is in detecting abnormal values that, under an HDR framework, would
correspond to those points that fall outside the region. This motivates certain metrics of common use in one-class
classification problems, which we also employ in this work. However, we emphasize that the derivation of an HDR can
in principle have a wider scope compared to a classification goal. Indeed, it would define the region of highest-density
points (e.g., normal values), regardless of whether these points have been observed or not. Thus, it would not only allow
us to classify an observed point as normal or abnormal, but it would also provide the entire region of normal values,
useful, for example, in a prediction setting prior to observing a point.
With that being said, let FP, TP, FN, and TN be the number of false positive, true positive, false negative, and true
negative points, respectively, where positive refer to those points that should be outside the true (1 − α)% HDR and
11
negative the others:

X X
TN = I(xi ∈ C(fα )), FN = I(xi ∈ Ĉn (fˆα ) | xi ∈
/ C(fα )),
i∈sn i∈sn
X X
TP = I(xi ∈
/ C(fα )), FP = / Ĉn (fˆα ) | xi ∈ C(fα )).
I(xi ∈
i∈sn i∈sn
Well-established measures of inefficiency are false negative/positive rates (FNR/FPR), and the total error rate (ERR),
that is, the one-complement of accuracy:
FN FP FN + FP
FNR = , FPR = , ERR = = 1 − Accuracy.
FN + TP FP + TN FN + FP + TN + TP
To account for the potentially high imbalance between positives and negatives, we also evaluate the two-sided F1
score, and the Matthews correlation coefficient (MCC; Matthews, 1975) alternatively known in statistics as the
ϕ-coefficient (see pg. 282 in Cramér, 1946):
2T P 2T N
F1 = + ,
2T P + F P + F N 2T N + F P + F N
TP × TN − FP × FN
MCC = p .
(T P + F P ) × (T P + F N ) × (T N + F P ) × (T N + F N )
In particular, MCC has been shown to produce good scores only if the classification is adequate in all four elements
of interest (true positives, false negatives, true negatives, and false positives), overcoming the overoptimistic inflated
results, especially on imbalanced datasets, of other popular classification measures (Chicco and Jurman, 2020). All
evaluations are based on α = 0.05, that is, a coverage probability of 95%, or, alternatively stated, 5% and 95% of
positives and negatives, respectively.
4.4 Simulation results
Comparative performance results are reported in Table 1. Due to space limitation, we focus on four scenarios that are
representative of the different copula-induced dependencies and multimodalities, specifically those on the diagonal of
Figure 4 (S1, S6, S11, S16), deferring to the Supplementary Material C all the scenarios and sample sizes considered.
The Dirichlet case is covered in Section 4.4.1.
In general, copula-based approaches result in the most performing measures, with classification errors (ERR, FPR,
FNR) uniformly smaller–or at most equal–than those of the other measures and at no cost in terms of variability. Their
advantage is particularly interesting in more complex scenarios, such as the quadrimodal case, where the parametric
copula approach (both M0PCop :DE and M3PCop :ϵ-CDF) practically halves the total error rate (ERR) of a standard
M0 :KDE. Although this distance may be considered relatively negligible when looking at the ERR (maximum distance
of 0.015 in S16) and the FPR (maximum distance of 0.008 in S16), it plays a significant role for the FNR, with an error
difference of 0.157. In practice, the probability of a true positive being missed by the “test” or measure decreases from
around 28% (M0 :KDE; S16) to around 13% (M0PCop :DE, M3PCop :ϵ-CDF; S16). This aspect is well-captured by the
alternative performance metrics of F1 and MCC, which offer a more reliable global measure of efficiency compared to
the ERR or the Accuracy. As depicted in Table 1–S16, when comparing M0 :KDE to M0PCop :DE or M3PCop :ϵ-CDF, the
MCC, for example, increases from 0.693 to more than 0.85. Note that MCC varies from −1 (worst value) to 1 (best
value).
When comparing the measures without the copula, no substantial differences can be noticed between M0 :KDE,
M1 :kNN-Eucl and M3 :ϵ-CDF, with a slightly improved performance of the second. The worst behavior is shown by
M2 :kNN-CDF (with FNR being as high as 72% in S16). Note that this was expected as M2 :kNN-CDF follows a
univariate rationale, motivating therefore the multivariate M3 :ϵ-CDF proposal (see Section 3).
4.4.1 Simplex scenario: Dirichlet

A particular interest is devoted to the simplex scenario. We consider a Dirichlet random vector X ∼ D(α1 , α2 , α3 )
defined on the 2-dimensional simplex, where αj > 0, j = 1, 2, 3; we refer to Chapter XI in Devroye (1986) for a
detailed exposition. Up to a normalizing constant, the density of X is given by
1 −1 α2 −1
f (x1 , x2 ; α1 , α2 , α3 ) ∝ xα
1 x2 (1 − x1 − x2 )α3 −1 , x1 , x2 ∈ [0, 1]; x1 + x2 ≤ 1.
12
Table 1: Performance results in terms of mean (standard deviation) of the compared methods in selected scenarios for
sample size n = 500.
Measures w/o copula Measures w copula
Metrics M0 :KDE M1 :kNN-Eucl M2 :kNN-CDF M3 :ϵ-CDF M0NPCop :DE M0PCop :DE M3NPCop :ϵ-CDF M3PCop :ϵ-CDF
Scenario S1 (Unimodal): Gaussian copula – Student-t marginals

ERR 0.015 (0.006) 0.013 (0.006) 0.018 (0.007) 0.017 (0.006) 0.015 (0.007) 0.009 (0.006) 0.014 (0.006) 0.009 (0.006)
FPR 0.008 (0.005) 0.007 (0.006) 0.009 (0.007) 0.010 (0.006) 0.008 (0.006) 0.004 (0.005) 0.007 (0.006) 0.005 (0.006)
FNR 0.135 (0.092) 0.117 (0.079) 0.173 (0.075) 0.132 (0.089) 0.139 (0.088) 0.073 (0.087) 0.125 (0.086) 0.075 (0.086)
Accuracy 0.985 (0.006) 0.987 (0.006) 0.982 (0.007) 0.983 (0.006) 0.985 (0.007) 0.991 (0.006) 0.986 (0.006) 0.991 (0.006)
F1 1.842 (0.061) 1.863 (0.063) 1.807 (0.080) 1.826 (0.066) 1.840 (0.075) 1.908 (0.061) 1.854 (0.062) 1.906 (0.061)
MCC 0.846 (0.058) 0.867 (0.059) 0.811 (0.077) 0.830 (0.063) 0.844 (0.072) 0.912 (0.055) 0.858 (0.058) 0.910 (0.055)
Scenario S6 (Unimodal): Student-t copula – Gaussian marginals

ERR 0.015 (0.006) 0.017 (0.006) 0.060 (0.009) 0.019 (0.006) 0.014 (0.006) 0.011 (0.005) 0.017 (0.006) 0.011 (0.005)
FPR 0.008 (0.005) 0.009 (0.005) 0.032 (0.006) 0.010 (0.006) 0.007 (0.005) 0.006 (0.005) 0.009 (0.005) 0.006 (0.005)
FNR 0.140 (0.089) 0.161 (0.094) 0.601 (0.090) 0.165 (0.092) 0.128 (0.090) 0.096 (0.085) 0.157 (0.093) 0.095 (0.085)
Accuracy 0.985 (0.006) 0.983 (0.006) 0.940 (0.009) 0.981 (0.006) 0.986 (0.006) 0.989 (0.005) 0.983 (0.006) 0.989 (0.005)
F1 1.838 (0.062) 1.816 (0.065) 1.363 (0.096) 1.804 (0.068) 1.851 (0.059) 1.884 (0.058) 1.820 (0.061) 1.886 (0.058)
MCC 0.843 (0.059) 0.820 (0.062) 0.365 (0.096) 0.808 (0.066) 0.855 (0.055) 0.888 (0.053) 0.825 (0.059) 0.890 (0.053)
Scenario S11 (Bimodal): Frank copula – Gaussian & Gaussian mixture marginals
ERR 0.019 (0.007) 0.016 (0.006) 0.061 (0.009) 0.018 (0.006) 0.015 (0.006) 0.011 (0.006) 0.015 (0.006) 0.011 (0.006)
FPR 0.010 (0.005) 0.008 (0.005) 0.032 (0.006) 0.011 (0.006) 0.008 (0.005) 0.006 (0.005) 0.008 (0.005) 0.006 (0.005)
FNR 0.175 (0.095) 0.144 (0.093) 0.608 (0.092) 0.150 (0.093) 0.139 (0.093) 0.096 (0.089) 0.136 (0.093) 0.096 (0.089)
Accuracy 0.981 (0.007) 0.984 (0.006) 0.939 (0.009) 0.982 (0.006) 0.985 (0.006) 0.989 (0.006) 0.985 (0.006) 0.989 (0.006)
F1 1.800 (0.068) 1.832 (0.064) 1.355 (0.097) 1.810 (0.065) 1.837 (0.061) 1.882 (0.059) 1.841 (0.061) 1.883 (0.059)
MCC 0.804 (0.066) 0.836 (0.061) 0.357 (0.097) 0.815 (0.063) 0.842 (0.058) 0.887 (0.055) 0.845 (0.058) 0.887 (0.054)
Scenario S16 (Quadrimodal): Clayton copula – Gaussian mixture marginals

ERR 0.029 (0.007) 0.027 (0.008) 0.072 (0.010) 0.032 (0.008) 0.021 (0.007) 0.014 (0.006) 0.024 (0.007) 0.014 (0.006)
FPR 0.015 (0.006) 0.014 (0.006) 0.038 (0.005) 0.019 (0.007) 0.011 (0.006) 0.007 (0.006) 0.013 (0.006) 0.008 (0.006)
FNR 0.283 (0.092) 0.256 (0.093) 0.720 (0.088) 0.276 (0.095) 0.197 (0.089) 0.126 (0.089) 0.226 (0.088) 0.129 (0.089)
Accuracy 0.971 (0.007) 0.973 (0.008) 0.928 (0.010) 0.968 (0.008) 0.979 (0.007) 0.986 (0.006) 0.976 (0.007) 0.986 (0.006)
F1 1.689 (0.080) 1.716 (0.081) 1.238 (0.089) 1.670 (0.083) 1.777 (0.074) 1.850 (0.067) 1.748 (0.077) 1.848 (0.069)
MCC 0.693 (0.079) 0.720 (0.080) 0.239 (0.090) 0.675 (0.082) 0.782 (0.072) 0.855 (0.064) 0.752 (0.075) 0.852 (0.066)
In the 2-dimensional simplex, when α1 = 1, α2 = 1, α3 = a, the associated copula density has the following expression
up to a normalizing constant:
h 1 1
ia−1
(1 − u1 ) a+1 + (1 − u2 ) a+1 − 1
c (u1 , u2 ; α1 = 1, α2 = 1, α3 = a) ∝ a ,
[(1 − u1 )(1 − u2 )] a+1
1 1
with u1 , u2 such that u1 , u2 ∈ [0, 1] and (1 − u1 ) a+1 + (1 − u2 ) a+1 ≥ 1. The analytical derivation is given in
Supplementary Material A, while its graphical representation–in terms of a set of random draws and different level
sets–can be depicted in Figure 5.
To evaluate the proposed measures, we first need to check or perform an accurate tuning of their hyperparameters. As
shown in the Supplementary Material B, the optimal choice of k–which represents the optimal number of neighbors to
take hinto account
i for computing both M1 :kNN-Eucl and M2 :kNN-CDF–remains the same as for the other scenarios:
p
k= n/2 . However, for measures depending on ϵ–which defines the length, area, or volume of the neighborhood–
the optimal choice depends on the support of the underlying variable. Being defined on a simplex, in a Dirichlet
scenario, the optimal values of ϵ have a smaller magnitude compared to noncompact or less restrictive scenarios such as
S1-S16. In this case, the following choices are made for ϵ, guided by simulation studies reported in the Supplementary
Material B:
M3 :ϵ-CDF Performances are robust to sample size, with optimal ϵ = 0.10.

M3NPCop :ϵ-CDF Performances depend on the sample size, according to a heuristic given by: ϵ =
exp (−1.22 − 0.23 log(n)). This relationship is obtained by fitting a nonlinear regression (exponential de-
cay model), with the empirical optimal ϵ and the sample size n as dependent and independent variables,
respectively.
M3PCop :ϵ-CDF Performances are robust to sample size, with optimal ϵ = 0.02.
13
S17: Dirichlet (1,1,2) Dirichlet (1,1,2) Copula
1.0
1.0
0.8
0.8
0.6
0.6
u2
x2
0.4
0.4
0.2
0.2
0.0
0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
x1 u1
Figure 5: Contours of a Dirichlet(1,1,2) density (left) and of its copula density (right). The 95% HDR is depicted in
blue color.
In terms of results, as suggested in Table 1, the more complex the scenario, the more difficult we expect it to be for
identifying the highest-density points (or true negatives) vs. the true positives. In this particular case, as shown in
Figure 6, the discrepancy between the different measures is even more evident with high and persistently high FNR
values, even with increasing sample sizes, for M0 :KDE and M2 :kNN-CDF. All other measures improve with the
sample size, achieving results comparable to the other scenarios.
4.5 MAGIC data
We now apply the proposed measures to construct an HDR for the joint distribution of two selected variables from
the MAGIC dataset (Bock et al., 2004). These data simulate the registration of high-energy gamma particles in a
ground-based atmospheric Cherenkov gamma telescope, and have been studied in classification problems (Dvořák and
Savický, 2007), as well as to analyze the dependency structure of some of the variables (see e.g., Nagler and Czado,
2016; Grazian et al., 2022).
In this evaluation, we focus on gamma-ray observations (overall n = 12, 332) and consider the two variables “fConc1”
and “fM3Long”, after scaling them. We refer to Bock et al. (2004) for a full description of the dataset. In this case (as
deduced from the complex structure of the data; see Figure 7), the parametric approach is inappropriate for both the
estimation of the marginal distribution and, more importantly, the copula model. Thus, in Figure 7, we illustrate the
derived 95% HDR using the nonparametric measures only. Although in the absence of the underlying truth it is not
possible to perform a reliable evaluation, it seems that the two nonparametric copula-based approaches (M0NPCop :DE,
M3NPCop :ϵ-CDF), as well as the distance-based M1 :kNN-Eucl and M3 :ϵ-CDF, more sensibly exclude tail data points
(which may be expected to have a lower density) from the HDR.
To deal with estimation and data uncertainty, we also explore the potential of a “measure averaging” in a similar,
but simplified, fashion to model averaging (see e.g., Hoeting et al., 1999; Hjort and Claeskens, 2003). By forming a
consensus between the different available measures, we expect the resulting HDR to be more robust than the individual
estimates. Specifically, in Figure 8, we illustrate a 95% HDR formed by averaging across the different nonparametric
measures: the estimated HDR is defined by the set of points identified as highest-density points by more than half of
the evaluated metrics. Notably, when comparing the resulted HDR in Figure 8 and those obtained by the individual
measures (Figure 7), we identify M3 :ϵ-CDF and M3NPCop :ϵ-CDF as the closest regions (both with < 1% classification
difference). Therefore, these may result in a possible superior ability to detect anomalous values.
5 Discussion and Conclusion
In this work, we have discussed some generalizations of the standard density-based approach to compute HDRs by
using neighborhood measures. Various measures, with distinct properties, have been introduced and compared to the
14
S17: Dirichlet(1,1,2)
Accuracy Matthews Correlation Coefficient (MCC)
0.975 0.75
0.950 0.50
0.925
0.25
0.900
0.00
50 100 250 500 1000 50 100 250 500 1000

Sample Size (n) Sample Size (n)
False Positive Rate (FPR) False Negative Rate (FNR)

1.00
0.06 0.75
0.50
0.04
0.25
0.02
0.00
50 100 250 500 1000 50 100 250 500 1000

Sample Size (n) Sample Size (n)
NPCop NPCop
M0:KDE M2:kNN-CDF M0 :DE M3 :ε-CDF
PCop PCop
M1:kNN-Eucl M3:ε-CDF M0 :DE M3 :ε-CDF
Figure 6: Performance results in terms of mean and error bounds of the compared methods in the Dirichlet scenario for
varying sample sizes.
classical kernel density estimator, across different scenarios and sample sizes. Furthermore, the use of copula models
for representing a multivariate distribution through a combination of its marginals and their dependence structure
has been investigated. Our results suggest that such a generalized approach may provide great advantages to HDR
estimation when considering several alternative measures to KDE, in particular those based on copulae (both under a
parametric and nonparametric approach). In fact, compared to traditional KDE, the copula-based HDRs resulted in
greater accuracy and lower FPR and FNR in a number of simulation scenarios and possibly in real data (as suggested
by the MAGIC data application). Such performances are particularly important when the interest is in balancing
different types of errors, minimizing both false positives and false negatives, and maximizing therefore the probability
of detecting outliers or anomalous values. Among the different evaluated measures, those based on the ϵ-neighborhood
multivariate CDF distance (M3 :ϵ-CDF, M3PCop :ϵ-CDF, M3NPCop :ϵ-CDF) show the most promising results, especially
in the nonparametric setting and when the copula is involved. Since the main characteristic of these measures is
that they are based on the CDF rather than the PDF, it may suggest that focusing on estimating the former may be
advantageous. Similar findings are reported by Magdon-Ismail and Atiya (2002), who acknowledge that, compared
to directly estimating the PDF with e.g., KDE, approximating the CDF is less sensitive to statistical fluctuations and
its convergence rate is faster than the convergence rate of KDE methods. Although heuristic considerations for each
specific measure are made in terms of their hyperparameters, e.g., the choice of k in the kNN-based approaches, further
work may investigate the existence of optimal theoretical values in a similar fashion to the plethora of asymptotic works
on the optimal bandwidth choice in KDE (see e.g., Wand and Jones, 1994b; Chacón et al., 2011).
In this work, we focused on estimating HDR for continuous distributions that are dominated by the Lebesgue measure.
Furthermore, despite providing a general multidimensional framework for building HDRs, we evaluated the proposed
15
95% HDR on MAGIC Data

M0:KDE M1:kNN-Eucl M2:kNN-CDF
6
6
4
4
2
2
0
0
-2
-2
-2
-4
-4
-4
-6
-6
-6
-2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4
NPCop NPCop
M3:ε-CDF M0 :DE M3 :ε-CDF
6
6
4
4
2
2
0
0
-2
-2
-2
-4
-4
-4
-6
-6
-2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 -6 -2 -1 0 1 2 3 4
Figure 7: Estimated 95% HDR of two scaled variables (“fConc1” on x-axis and “fM3Long” on y-axis) of the MAGIC
dataset. Only nonparametric measures are evaluated. Cadet-blue points define the estimated 95% HDR; in contrast,
purple points are those lying outside the HDR.
approach in a bivariate context. Future lines of research may examine the problem in the underappreciated setting
of discrete probability distributions and in higher dimensions. In the first case, possible connections could be made
with the work of O’Neill (2022), which established some theory and an algorithm for HDRs in the space of discrete
distributions. In the second case, and more specifically with reference to copula-based measures, our aim is to explore
the use of vine copulae (Nagler and Czado, 2016) to construct flexible dependence models for an arbitrary number of
variables using only bivariate building blocks. We expect, in fact, to see remarkable advantages in using copulae over
an increased number of variables, as the extension of the common KDE to high dimensions has proven challenging in
terms of both computational efficiency and statistical inference.
References
Bock, R. K., Chilingarian, A., Gaug, M., Hakl, F., Hengstebeck, T., Jirina, M., Klaschka, J., Kotrc, E., Savicky, P.,
Towers, S., Vaiciulis, A., and Wittek, W. (2004). Methods for multidimensional event classification: a case study
using images from a Cherenkov gamma-ray telescope. Nuclear Instruments and Methods in Physics Research Section
A: Accelerators, Spectrometers, Detectors and Associated Equipment, 516(2):511–528.
Box, G. E. P. and Tiao, G. C. (1992). Bayesian inference in statistical analysis. Wiley, New York.
Cai, Y. (2010). Multivariate Quantile Function Models. Statistica Sinica, 20(2):481–496.
Chacón, J. E., Duong, T., and Wand, M. P. (2011). Asymptotics for General Multivariate Kernel Density Derivative
Estimators. Statistica Sinica, 21(2):807–840.
Chicco, D. and Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and
accuracy in binary classification evaluation. BMC Genomics, 21(1):6.
16
95% HDR on MAGIC Data (Measure Averaging)
6
4
2
0
x2
-2
-4
-6
-2 -1 0 1 2 3 4
x1
Figure 8: Estimated 95% HDR based on a “measure averaging” approach of two scaled variables (“fConc1” on x-axis
and “fM3Long” on y-axis) of the MAGIC dataset. Cadet-blue points define the estimated 95% HDR; in contrast, purple
points are those lying outside the HDR.
Coblenz, M., Dyckerhoff, R., and Grothe, O. (2018). Nonparametric estimation of multivariate quantiles. Environmetrics,
29(2):e2488.
Cramér, H. (1946). Mathematical methods of statistics. Mathematical methods of statistics. Princeton University Press,
Princeton, NJ, US.
Devroye, L. (1986). Non-Uniform Random Variate Generation. Springer New York, New York, NY.
Doss, C. R. and Weng, G. (2018). Bandwidth selection for kernel density estimators of multivariate level sets and
highest density regions. Electronic Journal of Statistics, 12(2):4313–4376.
Dvořák, J. and Savický, P. (2007). Softening Splits in Decision Trees Using Simulated Annealing. In Beliczynski, B.,
Dzielinski, A., Iwanowski, M., and Ribeiro, B., editors, Adaptive and Natural Computing Algorithms, Lecture Notes
in Computer Science, pages 721–729, Berlin, Heidelberg. Springer.
Figalli, A. (2018). On the continuity of center-outward distribution and quantile functions. Nonlinear Analysis,
177:413–421.
Grazian, C., Dalla Valle, L., and Liseo, B. (2022). Approximate Bayesian conditional copulas. Computational Statistics
& Data Analysis, 169:107417.
Guilbaud, O. (2008). Simultaneous confidence regions corresponding to Holm’s step-down procedure and other
closed-testing procedures. Biometrical Journal. Biometrische Zeitschrift, 50(5):678–692.
Hjort, N. L. and Claeskens, G. (2003). Frequentist Model Average Estimators. Journal of the American Statistical
Association, 98(464):879–899.
17
Hjort, N. L. and Jones, M. C. (1996). Locally parametric nonparametric density estimation. The Annals of Statistics,
24(4):1619–1647.
Hoeting, J. A., Madigan, D., Raftery, A. E., and Volinsky, C. T. (1999). Bayesian Model Averaging: A Tutorial.
Statistical Science, 14(4):382–401.
Hyndman, R. J. (1995). Highest density forecast regions for nonlinear and non-normal time series models. Journal of
Forecasting, 14(5):431–441.
Hyndman, R. J. (1996). Computing and Graphing Highest Density Regions. The American Statistician, 50(2):120–126.
Kim, J. H., Fraser, I., and Hyndman, R. J. (2011). Improved interval estimation of long run response from a dynamic
linear model: A highest density region approach. Computational Statistics & Data Analysis, 55(8):2477–2489.
Korpela, J., Oikarinen, E., Puolamaki, K., and Ukkonen, A. (2017). Multivariate Confidence Intervals. In Proceedings
of the 2017 SIAM International Conference on Data Mining, pages 696–704, Houston, Texas, USA. Society for
Industrial and Applied Mathematics.
Krishnamoorthy, K. and Mathew, T. (2009). Statistical tolerance regions: theory, applications, and computation. Wiley
series in probability and statistics. Wiley, Hoboken, N.J.
Liu, H., Lafferty, J., and Wasserman, L. (2007). Sparse Nonparametric Density Estimation in High Dimensions Using
the Rodeo. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, pages
283–290. PMLR.
Loftsgaarden, D. O. and Quesenberry, C. P. (1965). A Nonparametric Estimate of a Multivariate Density Function. The
Annals of Mathematical Statistics, 36(3):1049–1051.
Magdon-Ismail, M. and Atiya, A. (2002). Density estimation and random variate generation using multilayer networks.
IEEE Transactions on Neural Networks, 13(3):497–520.
Matthews, B. W. (1975). Comparison of the predicted and observed secondary structure of T4 phage lysozyme.
Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2):442–451.
Meeker, W. Q., Hahn, G. J., and Escobar, L. A. (2017). Statistical intervals: a guide for practitioners and researchers.
Wiley series in probability and statistics. Wiley, Hoboken, New Jersey, second edition edition.
Munoz, A. and Moguerza, J. (2006). Estimation of high-density regions using one-class neighbor machines. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 28(3):476–480.
Nagler, T. and Czado, C. (2016). Evading the curse of dimensionality in nonparametric density estimation with
simplified vine copulas. Journal of Multivariate Analysis, 151:69–89.
Nelsen, R. B. (2006). An introduction to copulas. Springer series in statistics. Springer, New York, 2nd ed edition.
O’Neill, B. (2022). Smallest covering regions and highest density regions for discrete distributions. Computational
Statistics, 37(3):1229–1254.
Parzen, E. (1962). On Estimation of a Probability Density Function and Mode. The Annals of Mathematical Statistics,
33(3):1065–1076.
Ramaswamy, S., Rastogi, R., and Shim, K. (2000). Efficient algorithms for mining outliers from large data sets. ACM
SIGMOD Record, 29(2):427–438.
Saavedra-Nieves, P. (2022). Nonparametric estimation of highest density regions for COVID-19. Journal of Nonpara-
metric Statistics, 34(3):663–682.
Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., and Williamson, R. C. (2001). Estimating the Support of a
High-Dimensional Distribution. Neural Computation, 13(7):1443–1471.
Silverman, B. W. (1996). Density estimation for statistics and data analysis. Number 26 in Monographs on statistics
and applied probability. Chapman & Hall/CRC, Boca Raton.
Sklar, M. (1959). Fonctions de répartition à N dimensions et leurs marges. Annales de l’ISUP, VIII(3):229–231.
Steinwart, I., Hush, D., and Scovel, C. (2005). A Classification Framework for Anomaly Detection. Journal of Machine
Learning Research, 6(8):211–232.
Terrell, G. R. and Scott, D. W. (1992). Variable Kernel Density Estimation. The Annals of Statistics, 20(3):1236–1265.
Turkkan, N. and Pham-Gia, T. (1993). Computation of the highest posterior density interval in Bayesian analysis.
Journal of Statistical Computation and Simulation, 44(3-4):243–250.
Venturini, M. (2015). Statistical distances and probability metrics for multivariate data, ensembles and probability
distributions. PhD Thesis, Universidad Carlos III de Madrid.
18
Wand, M. P. and Jones, M. C. (1994a). Kernel Smoothing. Chapman and Hall/CRC.

Wand, M. P. and Jones, M. C. (1994b). Multivariate plug-in bandwidth selection. Computational Statistics, 9(2):97–116.
Supporting information
Additional supporting information is provided in the Supplementary material, made available at the end of this document.
The implementation of the algorithms and R codes to reproduce all the results and Figures/Tables of the main manuscript
are provided at https://github.com/nina-DL/HDR_neigh_measures.
19
Supplementary Material
A Derivation of the Dirichlet copula

In this section, we provide the analytical derivation of the Dirichlet copula in the specific case of a K − 1-dimensional
simplex distribution, with K = 3. Let X be a Dirichlet random vector, denoted by X ∼ D(α), where α = (α1 , α2 , α3 )
and αj > 0, j = 1, 2, 3. Its joint density is given by
1
f (x1 , x2 ; α) = xα1 −1 x2α2 −1 (1 − x1 − x2 )α3 −1 , x1 , x2 ∈ [0, 1]; x1 + x2 ≤ 1,
B(α) 1
where B(·) is the multivariate Beta function defined in terms of the Gamma function Γ(·) as:
QK
j=1 Γ(αj ) Γ(α1 )Γ(α2 )Γ(α3 )
B(α) = P = .
Γ
K
α Γ(α1 + α2 + α3 )
j=1 j
Using the copula representation given in Eq. (6) in the main paper, we have that
f (x1 , x2 ; α) = f1 (x1 ; α) f2 (x2 ; α) c (u1 , u2 ; α) ,
where u1 = F1 (x1 ) and u2 = F2 (x2 ).
P
The marginals of a Dirichlet random vector are known to follow a Beta Xj ∼ Beta αj , i̸=j αi , for j = 1, 2. Thus,
using the copula representation we have that:
f (x1 , x2 ; α)
c (u1 , u2 ; α) =
f1 (x1 ; α) f2 (x2 ; α)
1 −1
xα x2 2 −1 (1 − x1 − x2 )α3 −1
(( ( α
Γ(α +(α(2 + α3 )Γ(α
1 )
Γ(α
2 )Γ(α2 + α3 )Γ(α1 + α3 )
1( 1

= ( (
Γ(α Γ(α
1 ) 2 )Γ(α3 )Γ(α1 + α2 + α3 ) x1 1 −1 (1 − x1 )α2 +α3 −1
2 α
xα 2 −1
(1 − x2 )α1 +α3 −1

2
(1 − x1 − x2 )α3 −1
=
m · (1 − x1 )α2 +α3 −1 (1 − x2 )α1 +α3 −1
(1 − F1−1 (u1 ) − F2−1 (u2 ))α3 −1
= , (8)
m · (1 − F1−1 (u1 ))α2 +α3 −1 (1 − F2−1 (u2 ))α1 +α3 −1
Γ(α3 )Γ(α1 +α2 +α3 )
where m = Γ(α2 +α3 )Γ(α1 +α3 ) denotes the normalizing constant and Fj−1 is the inverse CDF of Xj .
Assume now that α1 = 1, α2 = 1, α3 = a, with a being a positive scalar. If Xj is Beta distributed, then Fj can be
expressed in terms of the incomplete Beta function, denoted by Ij , which gives a closed form solution for both u1 and
u2 , and therefore x1 and x2 :
uj = Fj (xj ) = Ij (xj ; α1 = 1, α2 = 1, α3 = a) = 1 − (1 − xj )a+1 , j = 1, 2
1
xj = 1 − (1 − uj ) a+1 , j = 1, 2. (9)
Taken together, Eq. (8) and Eq. (9) give us, up to the normalizing constant m,
h 1 1
ia−1
(1 − u1 ) a+1 + (1 − u2 ) a+1 − 1
c (u1 , u2 ; α1 = 1, α2 = 1, α3 = a) ∝ a .
[(1 − u1 )(1 − u2 )] a+1
Since x1 , x2 ∈ [0, 1], and x1 + x2 ≤ 1, we also have the following constraints on u1 , u2 , determined by Eq. (9):
1 1
u1 , u2 ∈ [0, 1], (1 − u1 ) a+1 + (1 − u2 ) a+1 ≥ 1.
B Supporting Details for Hyperparameter Choices

In this section, we illustrate the choices we made for the hyperparameters of the different measures evaluated in this
paper, which we reiterate below. We support our choices with simulation studies in which we compare the performances
of our measures for different values of the hyperparameters defined on an appropriate grid.
20
M0 :KDE Direct estimation of the bivariate density. We use KDE, with Gaussian kernel and bandwidth selection
based on the asympotically optimal solution proposed in Chacón et al. (2011), where its adequacy is shown
in general settings, including Gaussian mixture models. Thus, no simulations are performed to empirically
choose the hyperparameters.
M0NPCop :DE Nonparametric indirect density estimation with copula. We use standard KDE with the same optimal
bandwidth of Chacón et al. (2011) for the univariate marginals, and KDE with the transformation local
likelihood estimator and nearest-neighbor bandwidth for the copula density (Nagler and Czado, 2016). No
simulations are performed to empirically choose the hyperparameters.
M0PCop :DE Parametric indirect density estimation with copula. We adopt a fully parametric approach (with maximum
using the AIC criterion; no misspecification is introduced for the marginals. No hyperpameters are involved;
thus, no simulations are required.
M1 :kNN-Eucl Cumulative Euclidean distances from the kNNs. We sum the Euclidean distances between each point
and its k neighbors defined according to the Euclidean metric. The choice of the hyperparameter k is based on
an extensive
hp icross-validation procedure, leading to a general rule of thumb aligned with the existing literature:
k= n/2 , with [x] the integer closest to x. The supporting simulation results are given in Figures 9-12.
M2 :kNN-CDF Cumulative CDF distances from the kNNs. We sum the CDF distances between each point and its k
neighbors, which are defined according to the Euclidean metric. The choice of the hyperparameter k is based
on an extensive cross-validation procedure, suggesting a uniform choice across different scenarios and sample
sizes: k = 30. The supporting simulation results are given in Figures 13-16.
M3 :ϵ-CDF This represents the ϵ-neighborhood multivariate CDF distance, which uses empirical CDF as the CDF
estimator. The optimal choice of the hyperparameter ϵ is based on an extensive cross-validation procedure,
leading to the following heuristic for S1-S16 as a result of an exponential decay model fit with respect to
the sample size: ϵ = exp (2.13 − 0.3 log(n)). For S1-S16, the supporting simulation results are given in
Figures 17-20. For S17 (Dirichlet scenario), performances are robust to sample size, with optimal ϵ = 0.10;
see Figure 21.
M3NPCop :ϵ-CDF Fully nonparametric indirect estimation of the ϵ-CDF measure with copula. We use the empirical CDF
for estimating the univariate marginals, and KDE with the transformation local likelihood estimator and nearest-
neighbor bandwidth for the copula density. The optimal choice of the hyperparameter ϵ follows the same
strategy as the previous measure, leading to the following heuristic for S1-S16: ϵ = exp (1.74 − 0.26 log(n)).
For S1-S16, the supporting simulation results are given in Figures 22-25. For S17 (Dirichlet scenario),
performances depend on the sample size, according to a heuristic given by: ϵ = exp (−1.22 − 0.23 log(n)).
This relationship is obtained by fitting a nonlinear regression (exponential decay model), with the empirical
optimal ϵ and the sample size n as dependent and independent variables, respectively; see Figure 26.
M3PCop :ϵ-CDF Parametric indirect CDF estimation with copula. We adopt a fully parametric approach (with maximum
using the AIC criterion; no misspecification is introduced for the marginals. The optimal choice of the
hyperparameter ϵ follows the heuristic ϵ = exp (1.60 − 0.41 log(n)) for S1-S16 (see Figures 27-30). For S17
(Dirichlet scenario), performances are robust to sample size, with optimal ϵ = 0.02; see Figure 31.
21
n = 50
Total Error Rate (ERR) False Positive Rate (FPR)
0.05
0.08
0.04
0.06
0.03
0.04
0.02
0.02
0.01
0 10 20 30 40 50 0 10 20 30 40 50
k k
False Negative Rate (FNR) Accuracy
0.98
0.7
0.6
0.96
0.5
0.4
0.94
0.3
0.92
0.2
0.1
0 10 20 30 40 50 0 10 20 30 40 50
k k
F1 Score Matthews Correlation Coefficient (MCC)
1.7
0.7
1.6
0.6
1.5
0.5
1.4
0.4
1.3
0.3
1.2
0.2
0 10 20 30 40 50 0 10 20 30 40 50
k k
Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet
Gaussian copula Student-t copula Frank copula Clayton copula Dirichlet copula
Figure 9: Performance results in all evaluated scenarios for different values

hpof theihyperparameter k in M1 :kNN-Eucl.
Gray vertical line refers to the rule of thumb for the optimal choice: k = n/2 = 5, with n = 50.
22
n = 100
0.07
0.04
0.05
0.03
0.02
0.03
0.01
0.01
0.00
0 10 20 30 40 50 0 10 20 30 40 50
k k
0.6
0.99
0.5
0.97
0.4
0.3
0.95
0.2
0.93
0 10 20 30 40 50 0 10 20 30 40 50
k k
1.8
0.8
1.7
0.7
1.6
0.6
1.5
0.5
1.4
0.4
1.3
0 10 20 30 40 50 0 10 20 30 40 50
k k

23
n = 500
0.04
0.06
0.03
0.04
0.02
0.01
0.02
0.00
0.00
0 10 20 30 40 50 0 10 20 30 40 50
k k
1.00
0.5
0.98
0.4
0.3
0.96
0.2
0.94
0.1
0 10 20 30 40 50 0 10 20 30 40 50
k k
1.9
0.9
1.8
0.8
1.7
0.7
1.6
0.6
1.5
0.5
1.4
0.4
0 10 20 30 40 50 0 10 20 30 40 50
k k

24
n = 1000
0.04
0.06
0.03
0.04
0.02
0.01
0.02
0.00
0.00
0 20 40 60 80 100 0 20 40 60 80 100
k k
1.00
0.5
0.4
0.98
0.3
0.96
0.2
0.94
0.1
0 20 40 60 80 100 0 20 40 60 80 100
k k
1.9
0.9
1.8
0.8
1.7
0.7
1.6
0.6
1.5
0.5
1.4
0.4
0 20 40 60 80 100 0 20 40 60 80 100
k k

25
n = 50
0.10
0.05
0.08
0.06
0.03
0.04
0.01
0.02
0 5 10 15 20 25 30 0 5 10 15 20 25 30
k k
0.98
0.8
0.96
0.6
0.94
0.4
0.92
0.90
0.2
0 5 10 15 20 25 30 0 5 10 15 20 25 30
k k
1.7
0.1 0.2 0.3 0.4 0.5 0.6 0.7

1.6
1.5
1.4
1.3
1.2
1.1
0 5 10 15 20 25 30 0 5 10 15 20 25 30
k k
Figure 13: Performance results in all evaluated scenarios for n = 50 and for different values of the hyperparameter k in
M2 :kNN-CDF. Gray vertical line refers to the final choice: k = 30.
26
n = 100
0.00 0.01 0.02 0.03 0.04 0.05

0.08
0.06
0.04
0.02
0 10 20 30 40 50 0 10 20 30 40 50
k k
0.98
0.7
0.96
0.5
0.94
0.3
0.92
0.1
0 10 20 30 40 50 0 10 20 30 40 50
k k
1.2 1.3 1.4 1.5 1.6 1.7 1.8
0.2 0.3 0.4 0.5 0.6 0.7 0.8
0 10 20 30 40 50 0 10 20 30 40 50
k k
Figure 14: Performance results in all evaluated scenarios for n = 100 and for different values of the hyperparameter k
in M2 :kNN-CDF. Gray vertical line refers to the final choice: k = 30.
27
n = 500
0.08
0.04
0.06
0.02
0.04
0.02
0.00
0 10 20 30 40 50 0 10 20 30 40 50
k k
0.2 0.3 0.4 0.5 0.6 0.7 0.8
0.98
0.96
0.94
0.92
0 10 20 30 40 50 0 10 20 30 40 50
k k
1.8
0.8
1.6
0.6
1.4
0.4
1.2
0.2
0 10 20 30 40 50 0 10 20 30 40 50
k k
28
n = 100
0.00 0.01 0.02 0.03 0.04 0.05

0.08
0.06
0.04
0.02
0 10 20 30 40 50 0 10 20 30 40 50
k k
0.98
0.7
0.96
0.5
0.94
0.3
0.92
0.1
0 10 20 30 40 50 0 10 20 30 40 50
k k
1.2 1.3 1.4 1.5 1.6 1.7 1.8
0.2 0.3 0.4 0.5 0.6 0.7 0.8
0 10 20 30 40 50 0 10 20 30 40 50
k k
29
n = 50
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
0.8
1.6
0.6
1.2
0.4
0.8
0.2
0.0
0.4
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
Unimodal Unimodal w heavy tails Bimodal Quadrimodal
Gaussian copula Student-t copula Frank copula Clayton copula
Figure 17: Performance results in all evaluated scenarios for n = 50 and for different values of the hyperparameter ϵ in
M3 :ϵ-CDF. Gray vertical line reflects the following heuristic: ϵ = exp (2.13 − 0.3 log(n)) = 2.60.
30
n = 100
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.0
0.9
0.8
0.7
0.6
0.4
0.5
0.2
0.3
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
0.6 0.8 1.0 1.2 1.4 1.6 1.8
0.8
0.6
0.4
0.2
0.0
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
Figure 18: Performance results in all evaluated scenarios for n = 100 and for different values of the hyperparameter ϵ
in M3 :ϵ-CDF. Gray vertical line reflects the following heuristic: ϵ = exp (2.13 − 0.3 log(n)) = 2.11.
31
n = 500
0.25
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.00
1.0
0.95
0.8
0.90
0.6
0.85
0.4
0.80
0.2
0.75
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.8
0.8
1.6
0.6
1.4
0.4
1.2
0.2
1.0
0.0
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
32
n = 1000
0.06
0.08
0.04
0.04
0.02
0.00
0.00
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.0
0.98
0.8
0.6
0.94
0.4
0.90
0.2
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.8
0.8
1.6
0.6
1.4
0.4
1.2
0.2
1.0
0.0
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
33
Dirichlet(1, 1, 2)
0.15
0.10
0.10
0.06
0.05
0.02
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.00 0.05 0.10 0.15 0.20 0.25 0.30
ε ε
1.0
0.95
0.8
0.6
0.90
0.4
0.85
0.2
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.00 0.05 0.10 0.15 0.20 0.25 0.30
ε ε
1.6
0.6
1.4
0.4
1.2
0.2
1.0
0.0
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.00 0.05 0.10 0.15 0.20 0.25 0.30
ε ε
n = 50 n = 100 n = 500 n = 1000
Figure 21: Performance results in the Dirichlet scenario for all sample sizes and for different values of the hyperparameter
ϵ in M3 :ϵ-CDF. Gray vertical line reflects the optimal choice given by ϵ = 0.10.
34
n = 50
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.6
0.7
0.5
1.2
0.3
0.8
0.1
0.4
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
M3NPCop :ϵ-CDF. Gray vertical line reflects the following heuristic: ϵ = exp (1.74 − 0.26 log(n)) ≈ 2.
35
n = 100
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
0.9
0.7
0.7
0.5
0.5
0.3
0.3
0.1
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.8
0.8
1.4
0.6
1.0
0.4
0.6
0.2
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
in M3NPCop :ϵ-CDF. Gray vertical line reflects the following heuristic: ϵ = exp (1.74 − 0.26 log(n)) = 1.72.
36
n = 500
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.00
0.1 0.2 0.3 0.4 0.5 0.6 0.7
0.95
0.90
0.85
0.80
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
0.9
1.8
0.7
1.6
0.5
1.4
0.3
1.2
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
37
n = 1000
0.06
0.08
0.06
0.04
0.04
0.02
0.02
0.00
0.00
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.00
0.7
0.98
0.5
0.96
0.3
0.94
0.92
0.1
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
1.8
0.8
1.6
0.6
1.4
0.4
1.2
0.2
0 2 4 6 8 10 0 2 4 6 8 10
ε ε
38
0.06
0.08
0.04
0.06
0.04
0.02
0.02
0.00
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.00 0.05 0.10 0.15 0.20 0.25 0.30
ε ε
0.1 0.2 0.3 0.4 0.5 0.6 0.7
0.98
0.96
0.94
0.92
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.00 0.05 0.10 0.15 0.20 0.25 0.30
ε ε
1.8
0.2 0.3 0.4 0.5 0.6 0.7 0.8

1.6
1.4
1.2
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.00 0.05 0.10 0.15 0.20 0.25 0.30
ε ε
n = 50 n = 100 n = 500 n = 1000
ϵ in M3NPCop :ϵ-CDF. The optimal choice reflects the following heuristic: ϵ = exp (−1.22 − 0.23 log(n)).
39
n = 50
0.08
0.05
0.04
0.06
0.03
0.04
0.02
0.02
0.01
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
0.98
0.6
0.5
0.96
0.4
0.94
0.3
0.2
0.92
0.1
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
0.8
1.7
0.7
1.6
0.6
1.5
0.5
1.4
0.4
1.3
0.3
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
M3PCop :ϵ-CDF. Gray vertical line reflects the following heuristic: ϵ = exp (1.60 − 0.41 log(n)) ≈ 1.
40
n = 100
0.07
0.04
0.03
0.05
0.02
0.03
0.01
0.00
0.01
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
0.99
0.6
0.5
0.97
0.4
0.95
0.3
0.2
0.93
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
1.8
0.8
1.7
0.7
1.6
0.6
1.5
0.5
1.4
0.4
1.3
0.3
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
in M3PCop :ϵ-CDF. Gray vertical line reflects the following heuristic: ϵ = exp (1.60 − 0.41 log(n)) = 0.75.
41
n = 500
0.00 0.01 0.02 0.03 0.04

0.06
0.04
0.02
0.00
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
1.00
0.6
0.5
0.98
0.4
0.96
0.3
0.2
0.94
0.1
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
1.3 1.4 1.5 1.6 1.7 1.8 1.9
0.3 0.4 0.5 0.6 0.7 0.8 0.9
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
42
n = 1000
0.00 0.01 0.02 0.03 0.04

0.06
0.04
0.02
0.00
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
1.00
0.6
0.5
0.98
0.4
0.3
0.96
0.2
0.94
0.1
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
1.4 1.5 1.6 1.7 1.8 1.9
0.4 0.5 0.6 0.7 0.8 0.9
0 1 2 3 4 5 0 1 2 3 4 5
ε ε
43
Dirichlet(1, 1, 2)
0.04
0.05
0.03
0.03
0.02
0.01
0.01
0.00 0.05 0.10 0.15 0.20 0.00 0.05 0.10 0.15 0.20
ε ε
0.5
0.98
0.4
0.3
0.96
0.2
0.94
0.1
0.00 0.05 0.10 0.15 0.20 0.00 0.05 0.10 0.15 0.20
ε ε
1.8
0.8
1.7
0.7
1.6
0.6
1.5
0.5
1.4
0.00 0.05 0.10 0.15 0.20 0.00 0.05 0.10 0.15 0.20
ε ε
n = 50 n = 100 n = 500 n = 1000
ϵ in M3PCop :ϵ-CDF. Gray vertical line reflects the optimal choice given by ϵ = 0.02.
44
C Simulation results for all scenarios

In this section, we report the full set of simulation results (for all scenarios and sample sizes) that were omitted in the
main paper for the sake of space.
Table 2: Performance results in terms of mean (standard deviation) of the compared methods in scenario S1 across the
different sample sizes.
n = 50
ERR 0.035 (0.023) 0.029 (0.020) 0.031 (0.021) 0.034 (0.023) 0.057 (0.028) 0.030 (0.020) 0.034 (0.021) 0.030 (0.020)
FPR 0.024 (0.018) 0.020 (0.019) 0.021 (0.019) 0.025 (0.020) 0.035 (0.020) 0.021 (0.019) 0.023 (0.018) 0.021 (0.019)
FNR 0.175 (0.234) 0.120 (0.193) 0.146 (0.221) 0.133 (0.211) 0.452 (0.344) 0.126 (0.196) 0.158 (0.223) 0.122 (0.192)
Accuracy 0.965 (0.023) 0.971 (0.020) 0.969 (0.021) 0.966 (0.023) 0.943 (0.028) 0.970 (0.020) 0.966 (0.021) 0.970 (0.020)
F1 1.628 (0.263) 1.681 (0.263) 1.661 (0.272) 1.649 (0.267) 1.416 (0.305) 1.675 (0.262) 1.643 (0.260) 1.678 (0.261)
MCC 0.698 (0.197) 0.756 (0.176) 0.733 (0.199) 0.722 (0.192) 0.463 (0.298) 0.749 (0.178) 0.715 (0.188) 0.752 (0.174)
n = 100
ERR 0.026 (0.015) 0.021 (0.013) 0.023 (0.014) 0.025 (0.016) 0.038 (0.017) 0.020 (0.014) 0.025 (0.014) 0.020 (0.013)
FPR 0.014 (0.012) 0.011 (0.012) 0.012 (0.013) 0.016 (0.013) 0.020 (0.014) 0.011 (0.012) 0.013 (0.012) 0.011 (0.012)
FNR 0.204 (0.195) 0.152 (0.165) 0.181 (0.179) 0.150 (0.174) 0.359 (0.224) 0.143 (0.165) 0.185 (0.180) 0.143 (0.162)
Accuracy 0.974 (0.015) 0.979 (0.013) 0.977 (0.014) 0.975 (0.016) 0.962 (0.017) 0.980 (0.014) 0.975 (0.014) 0.980 (0.013)
F1 1.707 (0.183) 1.762 (0.173) 1.737 (0.186) 1.728 (0.186) 1.573 (0.217) 1.769 (0.174) 1.726 (0.174) 1.770 (0.171)
MCC 0.733 (0.158) 0.788 (0.139) 0.762 (0.159) 0.754 (0.156) 0.593 (0.209) 0.796 (0.139) 0.752 (0.146) 0.796 (0.136)
n = 500
ERR 0.015 (0.006) 0.013 (0.006) 0.018 (0.007) 0.017 (0.006) 0.015 (0.007) 0.009 (0.006) 0.014 (0.006) 0.009 (0.006)
FPR 0.008 (0.005) 0.007 (0.006) 0.009 (0.007) 0.010 (0.006) 0.008 (0.006) 0.004 (0.005) 0.007 (0.006) 0.005 (0.006)
FNR 0.135 (0.092) 0.117 (0.079) 0.173 (0.075) 0.132 (0.089) 0.139 (0.088) 0.073 (0.087) 0.125 (0.086) 0.075 (0.086)
Accuracy 0.985 (0.006) 0.987 (0.006) 0.982 (0.007) 0.983 (0.006) 0.985 (0.007) 0.991 (0.006) 0.986 (0.006) 0.991 (0.006)
F1 1.842 (0.061) 1.863 (0.063) 1.807 (0.080) 1.826 (0.066) 1.840 (0.075) 1.908 (0.061) 1.854 (0.062) 1.906 (0.061)
MCC 0.846 (0.058) 0.867 (0.059) 0.811 (0.077) 0.830 (0.063) 0.844 (0.072) 0.912 (0.055) 0.858 (0.058) 0.910 (0.055)
n = 1000
ERR 0.012 (0.004) 0.011 (0.004) 0.018 (0.005) 0.014 (0.004) 0.009 (0.004) 0.006 (0.004) 0.011 (0.004) 0.006 (0.004)
FPR 0.006 (0.004) 0.006 (0.004) 0.010 (0.005) 0.008 (0.004) 0.005 (0.004) 0.003 (0.004) 0.006 (0.004) 0.003 (0.004)
FNR 0.113 (0.067) 0.100 (0.056) 0.181 (0.053) 0.115 (0.066) 0.086 (0.065) 0.052 (0.064) 0.098 (0.060) 0.053 (0.064)
Accuracy 0.988 (0.004) 0.989 (0.004) 0.982 (0.005) 0.986 (0.004) 0.991 (0.004) 0.994 (0.004) 0.989 (0.004) 0.994 (0.004)
F1 1.873 (0.048) 1.887 (0.043) 1.803 (0.060) 1.855 (0.045) 1.901 (0.045) 1.937 (0.042) 1.888 (0.040) 1.935 (0.042)
MCC 0.875 (0.046) 0.889 (0.040) 0.806 (0.059) 0.858 (0.044) 0.903 (0.043) 0.939 (0.039) 0.891 (0.038) 0.938 (0.039)
45
n = 50
ERR 0.043 (0.025) 0.043 (0.024) 0.055 (0.026) 0.049 (0.026) 0.042 (0.024) 0.040 (0.023) 0.044 (0.024) 0.037 (0.022)
FPR 0.028 (0.018) 0.028 (0.018) 0.034 (0.019) 0.033 (0.022) 0.027 (0.019) 0.026 (0.019) 0.029 (0.019) 0.025 (0.019)
FNR 0.252 (0.280) 0.251 (0.279) 0.402 (0.330) 0.292 (0.297) 0.247 (0.276) 0.219 (0.262) 0.274 (0.281) 0.195 (0.243)
Accuracy 0.957 (0.025) 0.957 (0.024) 0.945 (0.026) 0.951 (0.026) 0.958 (0.024) 0.960 (0.023) 0.956 (0.024) 0.963 (0.022)
F1 1.549 (0.274) 1.553 (0.275) 1.435 (0.286) 1.509 (0.284) 1.557 (0.276) 1.579 (0.273) 1.535 (0.274) 1.600 (0.269)
MCC 0.623 (0.224) 0.627 (0.224) 0.495 (0.270) 0.580 (0.246) 0.632 (0.224) 0.657 (0.213) 0.607 (0.228) 0.680 (0.198)
n = 100
ERR 0.031 (0.017) 0.032 (0.016) 0.048 (0.019) 0.036 (0.016) 0.029 (0.016) 0.026 (0.016) 0.032 (0.016) 0.025 (0.015)
FPR 0.017 (0.012) 0.017 (0.013) 0.025 (0.013) 0.020 (0.014) 0.015 (0.012) 0.014 (0.013) 0.017 (0.013) 0.013 (0.012)
FNR 0.253 (0.212) 0.263 (0.206) 0.450 (0.235) 0.282 (0.201) 0.228 (0.203) 0.198 (0.193) 0.265 (0.203) 0.186 (0.188)
Accuracy 0.969 (0.017) 0.968 (0.016) 0.952 (0.019) 0.964 (0.016) 0.971 (0.016) 0.974 (0.016) 0.968 (0.016) 0.975 (0.015)
F1 1.655 (0.190) 1.649 (0.191) 1.478 (0.214) 1.615 (0.193) 1.682 (0.191) 1.712 (0.187) 1.649 (0.190) 1.723 (0.185)
MCC 0.680 (0.172) 0.673 (0.173) 0.497 (0.210) 0.640 (0.176) 0.707 (0.169) 0.738 (0.162) 0.673 (0.172) 0.749 (0.157)
n = 500
ERR 0.016 (0.006) 0.018 (0.006) 0.050 (0.009) 0.020 (0.007) 0.014 (0.006) 0.011 (0.006) 0.017 (0.006) 0.011 (0.006)
FPR 0.008 (0.005) 0.010 (0.006) 0.026 (0.006) 0.011 (0.006) 0.007 (0.005) 0.005 (0.006) 0.009 (0.005) 0.005 (0.006)
FNR 0.144 (0.092) 0.172 (0.092) 0.498 (0.094) 0.179 (0.091) 0.123 (0.090) 0.093 (0.088) 0.157 (0.091) 0.092 (0.088)
Accuracy 0.984 (0.006) 0.982 (0.006) 0.950 (0.009) 0.980 (0.007) 0.986 (0.006) 0.989 (0.006) 0.983 (0.006) 0.989 (0.006)
F1 1.834 (0.065) 1.805 (0.069) 1.469 (0.098) 1.789 (0.070) 1.856 (0.062) 1.887 (0.062) 1.821 (0.066) 1.888 (0.063)
MCC 0.838 (0.061) 0.810 (0.067) 0.472 (0.098) 0.794 (0.068) 0.860 (0.058) 0.892 (0.057) 0.825 (0.063) 0.893 (0.058)
n = 1000
ERR 0.011 (0.004) 0.015 (0.004) 0.051 (0.007) 0.016 (0.004) 0.010 (0.004) 0.007 (0.004) 0.013 (0.004) 0.007 (0.004)
FPR 0.006 (0.004) 0.008 (0.004) 0.027 (0.004) 0.009 (0.004) 0.005 (0.004) 0.004 (0.004) 0.007 (0.004) 0.004 (0.004)
FNR 0.105 (0.067) 0.137 (0.069) 0.505 (0.066) 0.139 (0.067) 0.090 (0.066) 0.065 (0.066) 0.120 (0.068) 0.065 (0.066)
Accuracy 0.989 (0.004) 0.985 (0.004) 0.949 (0.007) 0.984 (0.004) 0.990 (0.004) 0.993 (0.004) 0.987 (0.004) 0.993 (0.004)
F1 1.879 (0.044) 1.846 (0.046) 1.465 (0.070) 1.836 (0.047) 1.896 (0.042) 1.922 (0.042) 1.864 (0.045) 1.922 (0.042)
MCC 0.882 (0.042) 0.848 (0.045) 0.466 (0.069) 0.838 (0.046) 0.898 (0.040) 0.924 (0.039) 0.867 (0.043) 0.924 (0.039)
n = 50
ERR 0.046 (0.027) 0.045 (0.025) 0.062 (0.028) 0.051 (0.026) 0.043 (0.024) 0.043 (0.025) 0.044 (0.024) 0.040 (0.023)
FPR 0.030 (0.019) 0.029 (0.018) 0.038 (0.018) 0.035 (0.022) 0.028 (0.019) 0.028 (0.019) 0.029 (0.019) 0.026 (0.019)
FNR 0.289 (0.304) 0.281 (0.294) 0.492 (0.340) 0.299 (0.303) 0.266 (0.283) 0.261 (0.287) 0.270 (0.288) 0.229 (0.267)
Accuracy 0.954 (0.027) 0.955 (0.025) 0.938 (0.028) 0.949 (0.026) 0.957 (0.024) 0.957 (0.025) 0.956 (0.024) 0.960 (0.023)
F1 1.522 (0.287) 1.529 (0.278) 1.363 (0.284) 1.493 (0.280) 1.545 (0.279) 1.549 (0.285) 1.540 (0.279) 1.575 (0.278)
MCC 0.593 (0.248) 0.601 (0.235) 0.415 (0.279) 0.565 (0.243) 0.618 (0.232) 0.623 (0.238) 0.613 (0.233) 0.651 (0.221)
n = 100
ERR 0.033 (0.017) 0.033 (0.017) 0.053 (0.019) 0.036 (0.017) 0.029 (0.016) 0.029 (0.016) 0.031 (0.016) 0.027 (0.016)
FPR 0.018 (0.012) 0.018 (0.013) 0.028 (0.013) 0.021 (0.014) 0.016 (0.012) 0.015 (0.013) 0.017 (0.013) 0.015 (0.013)
FNR 0.271 (0.215) 0.273 (0.206) 0.515 (0.239) 0.277 (0.206) 0.232 (0.202) 0.227 (0.198) 0.255 (0.205) 0.215 (0.195)
Accuracy 0.967 (0.017) 0.967 (0.017) 0.947 (0.019) 0.964 (0.017) 0.971 (0.016) 0.971 (0.016) 0.969 (0.016) 0.973 (0.016)
F1 1.636 (0.194) 1.637 (0.194) 1.416 (0.218) 1.612 (0.199) 1.674 (0.189) 1.682 (0.191) 1.656 (0.195) 1.693 (0.192)
MCC 0.661 (0.176) 0.662 (0.175) 0.434 (0.216) 0.638 (0.180) 0.701 (0.165) 0.708 (0.167) 0.681 (0.175) 0.720 (0.166)
n = 500
ERR 0.017 (0.006) 0.019 (0.006) 0.056 (0.009) 0.020 (0.007) 0.014 (0.006) 0.012 (0.006) 0.016 (0.006) 0.012 (0.006)
FPR 0.009 (0.005) 0.010 (0.005) 0.029 (0.006) 0.011 (0.006) 0.007 (0.005) 0.006 (0.005) 0.008 (0.005) 0.006 (0.005)
FNR 0.155 (0.094) 0.173 (0.091) 0.558 (0.093) 0.171 (0.089) 0.124 (0.090) 0.102 (0.090) 0.143 (0.092) 0.102 (0.090)
Accuracy 0.983 (0.006) 0.981 (0.006) 0.944 (0.009) 0.980 (0.007) 0.986 (0.006) 0.988 (0.006) 0.984 (0.006) 0.988 (0.006)
F1 1.822 (0.064) 1.804 (0.067) 1.408 (0.097) 1.794 (0.068) 1.855 (0.061) 1.878 (0.063) 1.835 (0.063) 1.878 (0.062)
MCC 0.826 (0.062) 0.808 (0.065) 0.410 (0.097) 0.799 (0.066) 0.859 (0.058) 0.882 (0.059) 0.839 (0.061) 0.883 (0.058)
n = 1000
ERR 0.012 (0.004) 0.015 (0.004) 0.056 (0.007) 0.016 (0.004) 0.010 (0.004) 0.008 (0.004) 0.012 (0.004) 0.008 (0.004)
FPR 0.007 (0.004) 0.008 (0.004) 0.030 (0.004) 0.009 (0.004) 0.005 (0.004) 0.004 (0.004) 0.006 (0.004) 0.004 (0.004)
FNR 0.117 (0.068) 0.141 (0.068) 0.561 (0.065) 0.141 (0.067) 0.092 (0.065) 0.071 (0.065) 0.111 (0.067) 0.071 (0.065)
Accuracy 0.988 (0.004) 0.985 (0.004) 0.944 (0.007) 0.984 (0.004) 0.990 (0.004) 0.992 (0.004) 0.988 (0.004) 0.992 (0.004)
F1 1.869 (0.043) 1.843 (0.045) 1.407 (0.069) 1.834 (0.046) 1.895 (0.040) 1.916 (0.041) 1.874 (0.042) 1.917 (0.041)
MCC 0.871 (0.041) 0.846 (0.044) 0.408 (0.069) 0.836 (0.045) 0.897 (0.038) 0.919 (0.038) 0.876 (0.040) 0.919 (0.038)
46
n = 50
ERR 0.061 (0.030) 0.060 (0.028) 0.081 (0.030) 0.068 (0.032) 0.050 (0.027) 0.050 (0.027) 0.055 (0.027) 0.049 (0.027)
FPR 0.038 (0.018) 0.037 (0.018) 0.048 (0.016) 0.046 (0.024) 0.031 (0.019) 0.032 (0.019) 0.034 (0.019) 0.031 (0.019)
FNR 0.464 (0.342) 0.454 (0.336) 0.709 (0.321) 0.444 (0.347) 0.335 (0.314) 0.337 (0.314) 0.400 (0.324) 0.326 (0.310)
Accuracy 0.939 (0.030) 0.940 (0.028) 0.919 (0.030) 0.932 (0.032) 0.950 (0.027) 0.950 (0.027) 0.945 (0.027) 0.951 (0.027)
F1 1.379 (0.287) 1.391 (0.287) 1.188 (0.260) 1.363 (0.284) 1.485 (0.292) 1.481 (0.290) 1.434 (0.287) 1.490 (0.289)
MCC 0.438 (0.277) 0.451 (0.275) 0.221 (0.273) 0.426 (0.274) 0.559 (0.257) 0.555 (0.255) 0.500 (0.265) 0.564 (0.252)
n = 100
ERR 0.046 (0.020) 0.043 (0.020) 0.069 (0.021) 0.050 (0.021) 0.035 (0.017) 0.034 (0.017) 0.038 (0.018) 0.034 (0.017)
FPR 0.024 (0.012) 0.023 (0.013) 0.036 (0.011) 0.030 (0.015) 0.019 (0.013) 0.018 (0.013) 0.020 (0.013) 0.018 (0.012)
FNR 0.410 (0.242) 0.388 (0.238) 0.683 (0.233) 0.380 (0.237) 0.297 (0.218) 0.290 (0.218) 0.335 (0.224) 0.280 (0.217)
Accuracy 0.954 (0.020) 0.957 (0.020) 0.931 (0.021) 0.950 (0.021) 0.965 (0.017) 0.966 (0.017) 0.962 (0.018) 0.966 (0.017)
F1 1.506 (0.207) 1.529 (0.209) 1.259 (0.209) 1.497 (0.203) 1.617 (0.196) 1.623 (0.196) 1.581 (0.200) 1.631 (0.194)
MCC 0.526 (0.202) 0.550 (0.203) 0.270 (0.213) 0.520 (0.195) 0.640 (0.182) 0.647 (0.182) 0.603 (0.189) 0.655 (0.179)
n = 500
ERR 0.027 (0.007) 0.025 (0.007) 0.069 (0.009) 0.030 (0.008) 0.019 (0.006) 0.014 (0.006) 0.019 (0.007) 0.014 (0.006)
FPR 0.014 (0.006) 0.013 (0.006) 0.036 (0.005) 0.018 (0.006) 0.010 (0.006) 0.008 (0.006) 0.010 (0.006) 0.008 (0.006)
FNR 0.253 (0.095) 0.234 (0.094) 0.686 (0.084) 0.256 (0.099) 0.177 (0.087) 0.131 (0.089) 0.180 (0.091) 0.131 (0.089)
Accuracy 0.973 (0.007) 0.975 (0.007) 0.931 (0.009) 0.970 (0.008) 0.981 (0.006) 0.986 (0.006) 0.981 (0.007) 0.986 (0.006)
F1 1.719 (0.079) 1.740 (0.077) 1.274 (0.087) 1.691 (0.080) 1.800 (0.072) 1.846 (0.069) 1.796 (0.072) 1.847 (0.069)
MCC 0.723 (0.078) 0.744 (0.075) 0.275 (0.087) 0.696 (0.079) 0.804 (0.070) 0.851 (0.065) 0.800 (0.069) 0.851 (0.065)
n = 1000
ERR 0.022 (0.005) 0.020 (0.005) 0.069 (0.007) 0.025 (0.005) 0.015 (0.004) 0.010 (0.004) 0.014 (0.004) 0.010 (0.004)
FPR 0.011 (0.004) 0.010 (0.004) 0.036 (0.003) 0.015 (0.004) 0.008 (0.004) 0.005 (0.004) 0.008 (0.004) 0.005 (0.004)
FNR 0.211 (0.067) 0.190 (0.068) 0.684 (0.060) 0.225 (0.072) 0.140 (0.059) 0.092 (0.065) 0.136 (0.066) 0.092 (0.065)
Accuracy 0.978 (0.005) 0.980 (0.005) 0.931 (0.007) 0.975 (0.005) 0.985 (0.004) 0.990 (0.004) 0.986 (0.004) 0.990 (0.004)
F1 1.771 (0.052) 1.792 (0.051) 1.277 (0.062) 1.737 (0.052) 1.845 (0.048) 1.894 (0.042) 1.848 (0.044) 1.894 (0.042)
MCC 0.773 (0.052) 0.794 (0.050) 0.278 (0.062) 0.740 (0.051) 0.847 (0.046) 0.896 (0.040) 0.850 (0.043) 0.897 (0.040)
n = 50
ERR 0.033 (0.021) 0.030 (0.020) 0.034 (0.022) 0.034 (0.022) 0.054 (0.026) 0.031 (0.020) 0.034 (0.021) 0.030 (0.020)
FPR 0.023 (0.019) 0.021 (0.020) 0.023 (0.020) 0.025 (0.020) 0.034 (0.020) 0.021 (0.020) 0.023 (0.019) 0.021 (0.020)
FNR 0.150 (0.210) 0.128 (0.192) 0.179 (0.238) 0.132 (0.206) 0.441 (0.332) 0.132 (0.193) 0.166 (0.221) 0.125 (0.189)
Accuracy 0.967 (0.021) 0.970 (0.020) 0.966 (0.022) 0.966 (0.022) 0.946 (0.026) 0.969 (0.020) 0.966 (0.021) 0.970 (0.020)
F1 1.639 (0.274) 1.663 (0.279) 1.625 (0.291) 1.638 (0.283) 1.426 (0.312) 1.659 (0.277) 1.626 (0.274) 1.666 (0.278)
MCC 0.728 (0.177) 0.754 (0.174) 0.711 (0.212) 0.727 (0.188) 0.485 (0.298) 0.750 (0.172) 0.713 (0.184) 0.757 (0.171)
n = 100
ERR 0.025 (0.015) 0.021 (0.013) 0.026 (0.014) 0.024 (0.015) 0.037 (0.017) 0.020 (0.013) 0.024 (0.014) 0.020 (0.013)
FPR 0.013 (0.012) 0.011 (0.012) 0.014 (0.013) 0.016 (0.013) 0.020 (0.013) 0.011 (0.012) 0.013 (0.012) 0.011 (0.012)
FNR 0.184 (0.192) 0.148 (0.164) 0.212 (0.187) 0.137 (0.163) 0.338 (0.221) 0.141 (0.166) 0.176 (0.179) 0.137 (0.162)
Accuracy 0.975 (0.015) 0.979 (0.013) 0.974 (0.014) 0.976 (0.015) 0.963 (0.017) 0.980 (0.013) 0.976 (0.014) 0.980 (0.013)
F1 1.726 (0.174) 1.764 (0.166) 1.708 (0.186) 1.741 (0.175) 1.590 (0.211) 1.771 (0.165) 1.735 (0.166) 1.774 (0.164)
MCC 0.749 (0.156) 0.787 (0.140) 0.729 (0.170) 0.764 (0.151) 0.608 (0.205) 0.794 (0.139) 0.758 (0.145) 0.798 (0.137)
n = 500
ERR 0.014 (0.006) 0.012 (0.005) 0.024 (0.007) 0.016 (0.006) 0.015 (0.006) 0.009 (0.006) 0.013 (0.006) 0.009 (0.006)
FPR 0.007 (0.005) 0.006 (0.006) 0.013 (0.007) 0.010 (0.006) 0.008 (0.006) 0.005 (0.006) 0.007 (0.005) 0.005 (0.006)
FNR 0.123 (0.095) 0.109 (0.082) 0.238 (0.080) 0.118 (0.087) 0.135 (0.087) 0.076 (0.090) 0.113 (0.091) 0.075 (0.089)
Accuracy 0.986 (0.006) 0.988 (0.005) 0.976 (0.007) 0.984 (0.006) 0.985 (0.006) 0.991 (0.006) 0.987 (0.006) 0.991 (0.006)
F1 1.853 (0.061) 1.869 (0.062) 1.739 (0.091) 1.837 (0.061) 1.842 (0.067) 1.903 (0.060) 1.864 (0.059) 1.903 (0.060)
MCC 0.858 (0.057) 0.874 (0.057) 0.743 (0.089) 0.842 (0.057) 0.847 (0.064) 0.907 (0.054) 0.869 (0.055) 0.908 (0.054)
n = 1000
ERR 0.010 (0.004) 0.010 (0.004) 0.026 (0.006) 0.013 (0.004) 0.010 (0.004) 0.006 (0.004) 0.009 (0.004) 0.006 (0.004)
FPR 0.005 (0.004) 0.005 (0.004) 0.014 (0.005) 0.008 (0.004) 0.005 (0.004) 0.003 (0.004) 0.005 (0.004) 0.003 (0.004)
FNR 0.092 (0.067) 0.091 (0.057) 0.257 (0.059) 0.105 (0.067) 0.087 (0.064) 0.053 (0.064) 0.084 (0.065) 0.053 (0.063)
Accuracy 0.990 (0.004) 0.990 (0.004) 0.974 (0.006) 0.987 (0.004) 0.990 (0.004) 0.994 (0.004) 0.991 (0.004) 0.994 (0.004)
F1 1.893 (0.042) 1.894 (0.041) 1.723 (0.068) 1.864 (0.041) 1.898 (0.040) 1.933 (0.039) 1.901 (0.038) 1.934 (0.039)
MCC 0.896 (0.041) 0.897 (0.040) 0.725 (0.067) 0.866 (0.040) 0.900 (0.038) 0.936 (0.036) 0.903 (0.036) 0.936 (0.037)
47
n = 50
ERR 0.042 (0.025) 0.042 (0.023) 0.062 (0.027) 0.049 (0.025) 0.042 (0.024) 0.039 (0.023) 0.043 (0.023) 0.036 (0.022)
FPR 0.027 (0.019) 0.028 (0.018) 0.038 (0.018) 0.033 (0.021) 0.027 (0.019) 0.026 (0.019) 0.028 (0.019) 0.025 (0.019)
FNR 0.242 (0.287) 0.248 (0.270) 0.495 (0.338) 0.296 (0.293) 0.243 (0.276) 0.212 (0.255) 0.258 (0.274) 0.186 (0.238)
Accuracy 0.958 (0.025) 0.958 (0.023) 0.938 (0.027) 0.951 (0.025) 0.958 (0.024) 0.961 (0.023) 0.957 (0.023) 0.964 (0.022)
F1 1.560 (0.287) 1.557 (0.274) 1.364 (0.286) 1.502 (0.279) 1.560 (0.280) 1.588 (0.277) 1.548 (0.276) 1.609 (0.275)
MCC 0.636 (0.236) 0.632 (0.219) 0.415 (0.281) 0.574 (0.240) 0.636 (0.227) 0.666 (0.212) 0.623 (0.224) 0.690 (0.201)
n = 100
ERR 0.030 (0.016) 0.031 (0.016) 0.058 (0.020) 0.035 (0.017) 0.029 (0.015) 0.026 (0.015) 0.032 (0.016) 0.025 (0.015)
FPR 0.016 (0.012) 0.017 (0.012) 0.030 (0.012) 0.020 (0.014) 0.015 (0.012) 0.014 (0.012) 0.017 (0.012) 0.013 (0.012)
FNR 0.244 (0.204) 0.259 (0.205) 0.559 (0.233) 0.278 (0.208) 0.229 (0.197) 0.206 (0.189) 0.262 (0.206) 0.191 (0.184)
Accuracy 0.970 (0.016) 0.969 (0.016) 0.943 (0.020) 0.965 (0.017) 0.971 (0.015) 0.974 (0.015) 0.968 (0.016) 0.975 (0.015)
F1 1.672 (0.185) 1.656 (0.185) 1.377 (0.211) 1.622 (0.194) 1.683 (0.175) 1.709 (0.180) 1.655 (0.186) 1.723 (0.176)
MCC 0.696 (0.165) 0.680 (0.167) 0.392 (0.213) 0.646 (0.178) 0.708 (0.153) 0.735 (0.155) 0.678 (0.169) 0.749 (0.149)
n = 500
ERR 0.015 (0.006) 0.017 (0.006) 0.060 (0.009) 0.019 (0.006) 0.014 (0.006) 0.011 (0.005) 0.017 (0.006) 0.011 (0.005)
FPR 0.008 (0.005) 0.009 (0.005) 0.032 (0.006) 0.010 (0.006) 0.007 (0.005) 0.006 (0.005) 0.009 (0.005) 0.006 (0.005)
FNR 0.140 (0.089) 0.161 (0.094) 0.601 (0.090) 0.165 (0.092) 0.128 (0.090) 0.096 (0.085) 0.157 (0.093) 0.095 (0.085)
Accuracy 0.985 (0.006) 0.983 (0.006) 0.940 (0.009) 0.981 (0.006) 0.986 (0.006) 0.989 (0.005) 0.983 (0.006) 0.989 (0.005)
F1 1.838 (0.062) 1.816 (0.065) 1.363 (0.096) 1.804 (0.068) 1.851 (0.059) 1.884 (0.058) 1.820 (0.061) 1.886 (0.058)
MCC 0.843 (0.059) 0.820 (0.062) 0.365 (0.096) 0.808 (0.066) 0.855 (0.055) 0.888 (0.053) 0.825 (0.059) 0.890 (0.053)
n = 1000
ERR 0.012 (0.004) 0.014 (0.004) 0.061 (0.007) 0.014 (0.004) 0.011 (0.004) 0.008 (0.004) 0.013 (0.004) 0.008 (0.004)
FPR 0.006 (0.004) 0.007 (0.004) 0.032 (0.004) 0.008 (0.004) 0.006 (0.004) 0.004 (0.004) 0.007 (0.004) 0.004 (0.004)
FNR 0.109 (0.066) 0.130 (0.066) 0.608 (0.066) 0.129 (0.065) 0.100 (0.065) 0.067 (0.063) 0.123 (0.066) 0.067 (0.063)
Accuracy 0.988 (0.004) 0.986 (0.004) 0.939 (0.007) 0.986 (0.004) 0.989 (0.004) 0.992 (0.004) 0.987 (0.004) 0.992 (0.004)
F1 1.876 (0.042) 1.854 (0.044) 1.357 (0.071) 1.849 (0.044) 1.886 (0.040) 1.920 (0.038) 1.861 (0.040) 1.920 (0.038)
MCC 0.878 (0.041) 0.856 (0.043) 0.358 (0.071) 0.851 (0.042) 0.888 (0.038) 0.922 (0.035) 0.863 (0.039) 0.923 (0.035)
n = 50
ERR 0.046 (0.026) 0.043 (0.025) 0.073 (0.030) 0.050 (0.027) 0.043 (0.025) 0.042 (0.024) 0.043 (0.025) 0.040 (0.024)
FPR 0.030 (0.019) 0.028 (0.018) 0.044 (0.017) 0.034 (0.023) 0.028 (0.019) 0.027 (0.019) 0.028 (0.019) 0.026 (0.019)
FNR 0.296 (0.301) 0.258 (0.279) 0.620 (0.334) 0.287 (0.294) 0.257 (0.286) 0.259 (0.277) 0.255 (0.277) 0.226 (0.265)
Accuracy 0.954 (0.026) 0.957 (0.025) 0.927 (0.030) 0.950 (0.027) 0.958 (0.025) 0.958 (0.024) 0.957 (0.025) 0.960 (0.024)
F1 1.522 (0.283) 1.551 (0.274) 1.267 (0.279) 1.511 (0.280) 1.555 (0.283) 1.555 (0.276) 1.556 (0.276) 1.582 (0.277)
MCC 0.589 (0.247) 0.621 (0.226) 0.303 (0.286) 0.581 (0.242) 0.625 (0.237) 0.625 (0.229) 0.626 (0.228) 0.655 (0.221)
n = 100
ERR 0.034 (0.017) 0.033 (0.017) 0.065 (0.021) 0.036 (0.018) 0.030 (0.016) 0.029 (0.016) 0.031 (0.016) 0.028 (0.016)
FPR 0.018 (0.013) 0.017 (0.012) 0.034 (0.012) 0.021 (0.014) 0.016 (0.012) 0.015 (0.012) 0.016 (0.013) 0.014 (0.012)
FNR 0.286 (0.212) 0.274 (0.217) 0.636 (0.236) 0.272 (0.210) 0.245 (0.199) 0.234 (0.201) 0.257 (0.205) 0.222 (0.199)
Accuracy 0.966 (0.017) 0.967 (0.017) 0.935 (0.021) 0.964 (0.018) 0.970 (0.016) 0.971 (0.016) 0.969 (0.016) 0.972 (0.016)
F1 1.633 (0.195) 1.643 (0.194) 1.302 (0.211) 1.621 (0.196) 1.669 (0.180) 1.684 (0.190) 1.662 (0.191) 1.695 (0.189)
MCC 0.657 (0.180) 0.666 (0.178) 0.315 (0.214) 0.646 (0.179) 0.694 (0.159) 0.708 (0.169) 0.686 (0.173) 0.720 (0.166)
n = 500
ERR 0.018 (0.006) 0.018 (0.006) 0.068 (0.009) 0.019 (0.006) 0.015 (0.006) 0.012 (0.005) 0.017 (0.006) 0.012 (0.005)
FPR 0.010 (0.005) 0.010 (0.005) 0.036 (0.005) 0.011 (0.006) 0.008 (0.005) 0.006 (0.005) 0.009 (0.005) 0.006 (0.006)
FNR 0.171 (0.091) 0.172 (0.091) 0.680 (0.088) 0.166 (0.090) 0.139 (0.090) 0.104 (0.087) 0.154 (0.090) 0.103 (0.087)
Accuracy 0.982 (0.006) 0.982 (0.006) 0.932 (0.009) 0.981 (0.006) 0.985 (0.006) 0.988 (0.005) 0.983 (0.006) 0.988 (0.005)
F1 1.806 (0.067) 1.806 (0.067) 1.282 (0.093) 1.798 (0.068) 1.839 (0.062) 1.876 (0.060) 1.824 (0.062) 1.877 (0.061)
MCC 0.811 (0.064) 0.810 (0.064) 0.283 (0.093) 0.802 (0.066) 0.844 (0.059) 0.881 (0.056) 0.828 (0.059) 0.882 (0.056)
n = 1000
ERR 0.015 (0.004) 0.015 (0.004) 0.069 (0.007) 0.016 (0.004) 0.011 (0.004) 0.008 (0.004) 0.013 (0.004) 0.008 (0.004)
FPR 0.008 (0.004) 0.008 (0.004) 0.036 (0.004) 0.009 (0.004) 0.006 (0.004) 0.004 (0.004) 0.007 (0.004) 0.004 (0.004)
FNR 0.142 (0.067) 0.142 (0.067) 0.684 (0.063) 0.142 (0.065) 0.108 (0.067) 0.077 (0.066) 0.124 (0.067) 0.077 (0.066)
Accuracy 0.985 (0.004) 0.985 (0.004) 0.931 (0.007) 0.984 (0.004) 0.989 (0.004) 0.992 (0.004) 0.987 (0.004) 0.992 (0.004)
F1 1.845 (0.044) 1.845 (0.044) 1.279 (0.067) 1.833 (0.044) 1.880 (0.040) 1.913 (0.040) 1.864 (0.040) 1.913 (0.040)
MCC 0.847 (0.043) 0.847 (0.043) 0.280 (0.067) 0.835 (0.043) 0.882 (0.039) 0.915 (0.038) 0.866 (0.038) 0.916 (0.038)
48
n = 50
ERR 0.058 (0.029) 0.057 (0.027) 0.087 (0.031) 0.066 (0.031) 0.046 (0.025) 0.048 (0.025) 0.052 (0.027) 0.048 (0.025)
FPR 0.036 (0.018) 0.035 (0.018) 0.051 (0.015) 0.045 (0.025) 0.029 (0.019) 0.031 (0.018) 0.033 (0.019) 0.030 (0.018)
FNR 0.425 (0.339) 0.428 (0.335) 0.770 (0.295) 0.430 (0.344) 0.291 (0.299) 0.321 (0.306) 0.372 (0.327) 0.311 (0.299)
Accuracy 0.942 (0.029) 0.943 (0.027) 0.913 (0.031) 0.934 (0.031) 0.954 (0.025) 0.952 (0.025) 0.948 (0.027) 0.952 (0.025)
F1 1.413 (0.290) 1.414 (0.288) 1.140 (0.241) 1.378 (0.285) 1.522 (0.284) 1.497 (0.281) 1.460 (0.294) 1.504 (0.278)
MCC 0.471 (0.276) 0.473 (0.274) 0.163 (0.254) 0.439 (0.274) 0.594 (0.243) 0.566 (0.246) 0.524 (0.271) 0.574 (0.241)
n = 100
ERR 0.042 (0.018) 0.041 (0.018) 0.076 (0.021) 0.047 (0.020) 0.033 (0.016) 0.033 (0.017) 0.036 (0.017) 0.033 (0.017)
FPR 0.022 (0.012) 0.021 (0.013) 0.040 (0.011) 0.028 (0.016) 0.017 (0.012) 0.017 (0.013) 0.019 (0.013) 0.017 (0.013)
FNR 0.377 (0.232) 0.369 (0.225) 0.760 (0.215) 0.360 (0.231) 0.275 (0.208) 0.281 (0.214) 0.316 (0.215) 0.278 (0.209)
Accuracy 0.958 (0.018) 0.959 (0.018) 0.924 (0.021) 0.953 (0.020) 0.967 (0.016) 0.967 (0.017) 0.964 (0.017) 0.967 (0.017)
F1 1.544 (0.203) 1.555 (0.202) 1.187 (0.194) 1.524 (0.206) 1.641 (0.188) 1.639 (0.199) 1.605 (0.199) 1.641 (0.197)
MCC 0.565 (0.195) 0.576 (0.194) 0.196 (0.200) 0.547 (0.196) 0.665 (0.171) 0.662 (0.184) 0.627 (0.186) 0.664 (0.180)
n = 500
ERR 0.024 (0.007) 0.023 (0.007) 0.076 (0.010) 0.028 (0.008) 0.017 (0.006) 0.014 (0.006) 0.019 (0.006) 0.014 (0.006)
FPR 0.013 (0.006) 0.012 (0.006) 0.040 (0.005) 0.016 (0.006) 0.009 (0.006) 0.007 (0.006) 0.010 (0.006) 0.007 (0.006)
FNR 0.233 (0.092) 0.222 (0.091) 0.758 (0.082) 0.241 (0.097) 0.162 (0.089) 0.127 (0.088) 0.183 (0.089) 0.126 (0.088)
Accuracy 0.976 (0.007) 0.977 (0.007) 0.924 (0.010) 0.972 (0.008) 0.983 (0.006) 0.986 (0.006) 0.981 (0.006) 0.986 (0.006)
F1 1.744 (0.075) 1.755 (0.074) 1.200 (0.085) 1.711 (0.078) 1.818 (0.068) 1.854 (0.063) 1.796 (0.070) 1.855 (0.063)
MCC 0.748 (0.073) 0.759 (0.072) 0.201 (0.085) 0.716 (0.076) 0.822 (0.065) 0.858 (0.060) 0.800 (0.067) 0.860 (0.060)
n = 1000
ERR 0.020 (0.005) 0.019 (0.005) 0.076 (0.007) 0.024 (0.005) 0.013 (0.004) 0.010 (0.004) 0.015 (0.004) 0.010 (0.004)
FPR 0.011 (0.004) 0.010 (0.004) 0.040 (0.003) 0.013 (0.004) 0.007 (0.004) 0.005 (0.004) 0.008 (0.004) 0.005 (0.004)
FNR 0.197 (0.068) 0.180 (0.067) 0.762 (0.059) 0.206 (0.070) 0.126 (0.064) 0.090 (0.064) 0.146 (0.062) 0.090 (0.064)
Accuracy 0.980 (0.005) 0.981 (0.005) 0.924 (0.007) 0.976 (0.005) 0.987 (0.004) 0.990 (0.004) 0.985 (0.004) 0.990 (0.004)
F1 1.787 (0.052) 1.804 (0.051) 1.196 (0.062) 1.758 (0.051) 1.861 (0.043) 1.899 (0.042) 1.841 (0.044) 1.899 (0.042)
MCC 0.789 (0.051) 0.806 (0.050) 0.197 (0.062) 0.760 (0.050) 0.863 (0.041) 0.901 (0.040) 0.843 (0.043) 0.901 (0.040)
n = 50
ERR 0.037 (0.023) 0.033 (0.020) 0.036 (0.021) 0.038 (0.024) 0.060 (0.027) 0.030 (0.020) 0.038 (0.023) 0.031 (0.021)
FPR 0.025 (0.019) 0.022 (0.019) 0.024 (0.020) 0.028 (0.020) 0.037 (0.019) 0.021 (0.020) 0.026 (0.019) 0.022 (0.020)
FNR 0.188 (0.239) 0.147 (0.202) 0.192 (0.243) 0.165 (0.228) 0.478 (0.340) 0.125 (0.192) 0.197 (0.240) 0.130 (0.199)
Accuracy 0.963 (0.023) 0.967 (0.020) 0.964 (0.021) 0.962 (0.024) 0.940 (0.027) 0.970 (0.020) 0.962 (0.023) 0.969 (0.021)
F1 1.600 (0.275) 1.641 (0.271) 1.607 (0.283) 1.602 (0.276) 1.382 (0.296) 1.660 (0.276) 1.593 (0.271) 1.656 (0.279)
MCC 0.685 (0.198) 0.730 (0.173) 0.691 (0.209) 0.689 (0.195) 0.437 (0.289) 0.751 (0.171) 0.677 (0.196) 0.747 (0.178)
n = 100
ERR 0.029 (0.016) 0.024 (0.015) 0.029 (0.015) 0.029 (0.016) 0.042 (0.018) 0.020 (0.014) 0.028 (0.016) 0.020 (0.014)
FPR 0.015 (0.012) 0.013 (0.012) 0.015 (0.013) 0.018 (0.013) 0.022 (0.013) 0.011 (0.012) 0.015 (0.012) 0.011 (0.012)
FNR 0.230 (0.210) 0.186 (0.188) 0.242 (0.192) 0.186 (0.188) 0.393 (0.224) 0.141 (0.172) 0.227 (0.203) 0.143 (0.175)
Accuracy 0.971 (0.016) 0.976 (0.015) 0.971 (0.015) 0.971 (0.016) 0.958 (0.018) 0.980 (0.014) 0.972 (0.016) 0.980 (0.014)
F1 1.684 (0.186) 1.733 (0.180) 1.683 (0.188) 1.697 (0.182) 1.538 (0.208) 1.775 (0.172) 1.689 (0.180) 1.773 (0.172)
MCC 0.707 (0.169) 0.756 (0.158) 0.705 (0.172) 0.720 (0.161) 0.557 (0.203) 0.799 (0.142) 0.712 (0.162) 0.797 (0.143)
n = 500
ERR 0.017 (0.006) 0.016 (0.005) 0.024 (0.007) 0.018 (0.006) 0.019 (0.007) 0.009 (0.006) 0.015 (0.006) 0.009 (0.006)
FPR 0.009 (0.005) 0.008 (0.006) 0.012 (0.007) 0.011 (0.006) 0.010 (0.006) 0.005 (0.006) 0.008 (0.005) 0.005 (0.006)
FNR 0.156 (0.092) 0.142 (0.084) 0.226 (0.080) 0.142 (0.090) 0.178 (0.085) 0.069 (0.088) 0.137 (0.089) 0.071 (0.088)
Accuracy 0.983 (0.006) 0.984 (0.005) 0.976 (0.007) 0.982 (0.006) 0.981 (0.007) 0.991 (0.006) 0.985 (0.006) 0.991 (0.006)
F1 1.817 (0.063) 1.832 (0.061) 1.748 (0.082) 1.810 (0.066) 1.797 (0.079) 1.907 (0.061) 1.837 (0.061) 1.906 (0.060)
MCC 0.822 (0.060) 0.837 (0.058) 0.752 (0.081) 0.815 (0.063) 0.801 (0.077) 0.912 (0.055) 0.841 (0.058) 0.910 (0.055)
n = 1000
ERR 0.015 (0.005) 0.013 (0.004) 0.023 (0.005) 0.015 (0.004) 0.012 (0.004) 0.006 (0.004) 0.012 (0.004) 0.006 (0.004)
FPR 0.008 (0.004) 0.007 (0.004) 0.012 (0.005) 0.009 (0.004) 0.007 (0.004) 0.003 (0.004) 0.006 (0.004) 0.003 (0.004)
FNR 0.144 (0.068) 0.128 (0.061) 0.229 (0.057) 0.127 (0.068) 0.117 (0.065) 0.051 (0.065) 0.111 (0.064) 0.052 (0.065)
Accuracy 0.985 (0.005) 0.987 (0.004) 0.977 (0.005) 0.985 (0.004) 0.988 (0.004) 0.994 (0.004) 0.988 (0.004) 0.994 (0.004)
F1 1.840 (0.049) 1.857 (0.042) 1.754 (0.062) 1.842 (0.045) 1.868 (0.048) 1.937 (0.044) 1.874 (0.041) 1.937 (0.044)
MCC 0.842 (0.048) 0.860 (0.040) 0.756 (0.061) 0.844 (0.043) 0.871 (0.046) 0.940 (0.040) 0.877 (0.039) 0.939 (0.040)
49
Table 11: Performance results in terms of mean (standard deviation) of the compared methods in scenario S10 across
the different sample sizes.
n = 50
ERR 0.044 (0.026) 0.039 (0.022) 0.058 (0.027) 0.044 (0.025) 0.041 (0.023) 0.037 (0.023) 0.041 (0.023) 0.036 (0.022)
FPR 0.029 (0.019) 0.026 (0.019) 0.037 (0.019) 0.031 (0.022) 0.027 (0.019) 0.025 (0.019) 0.027 (0.019) 0.024 (0.019)
FNR 0.252 (0.288) 0.213 (0.260) 0.459 (0.341) 0.248 (0.280) 0.229 (0.266) 0.194 (0.251) 0.238 (0.272) 0.174 (0.235)
Accuracy 0.956 (0.026) 0.961 (0.022) 0.942 (0.027) 0.956 (0.025) 0.959 (0.023) 0.963 (0.023) 0.959 (0.023) 0.964 (0.022)
F1 1.537 (0.290) 1.577 (0.286) 1.389 (0.298) 1.537 (0.294) 1.560 (0.283) 1.591 (0.288) 1.557 (0.287) 1.608 (0.286)
MCC 0.622 (0.233) 0.667 (0.213) 0.452 (0.287) 0.623 (0.235) 0.647 (0.216) 0.683 (0.209) 0.644 (0.223) 0.702 (0.197)
n = 100
ERR 0.031 (0.016) 0.028 (0.015) 0.051 (0.019) 0.031 (0.016) 0.029 (0.015) 0.024 (0.015) 0.029 (0.015) 0.024 (0.015)
FPR 0.017 (0.012) 0.015 (0.012) 0.027 (0.013) 0.018 (0.014) 0.016 (0.012) 0.013 (0.012) 0.016 (0.013) 0.013 (0.012)
FNR 0.249 (0.206) 0.223 (0.201) 0.487 (0.244) 0.222 (0.199) 0.227 (0.199) 0.181 (0.188) 0.231 (0.200) 0.174 (0.184)
Accuracy 0.969 (0.016) 0.972 (0.015) 0.949 (0.019) 0.969 (0.016) 0.971 (0.015) 0.976 (0.015) 0.971 (0.015) 0.976 (0.015)
F1 1.659 (0.185) 1.687 (0.186) 1.443 (0.219) 1.667 (0.194) 1.681 (0.181) 1.727 (0.180) 1.679 (0.188) 1.732 (0.179)
MCC 0.682 (0.168) 0.710 (0.168) 0.459 (0.218) 0.692 (0.175) 0.704 (0.162) 0.751 (0.156) 0.702 (0.170) 0.757 (0.154)
n = 500
ERR 0.018 (0.007) 0.016 (0.006) 0.053 (0.009) 0.017 (0.006) 0.015 (0.006) 0.010 (0.006) 0.016 (0.006) 0.010 (0.006)
FPR 0.009 (0.006) 0.008 (0.006) 0.028 (0.006) 0.009 (0.006) 0.008 (0.005) 0.005 (0.006) 0.008 (0.005) 0.005 (0.006)
FNR 0.161 (0.094) 0.144 (0.092) 0.530 (0.093) 0.145 (0.090) 0.132 (0.091) 0.088 (0.089) 0.141 (0.091) 0.088 (0.089)
Accuracy 0.982 (0.007) 0.984 (0.006) 0.947 (0.009) 0.983 (0.006) 0.985 (0.006) 0.990 (0.006) 0.984 (0.006) 0.990 (0.006)
F1 1.814 (0.070) 1.832 (0.067) 1.436 (0.099) 1.822 (0.067) 1.845 (0.065) 1.890 (0.064) 1.835 (0.065) 1.890 (0.064)
MCC 0.818 (0.067) 0.837 (0.064) 0.439 (0.099) 0.827 (0.063) 0.849 (0.061) 0.895 (0.058) 0.840 (0.062) 0.895 (0.058)
n = 1000
ERR 0.014 (0.004) 0.013 (0.004) 0.054 (0.007) 0.013 (0.004) 0.011 (0.004) 0.007 (0.004) 0.012 (0.004) 0.007 (0.004)
FPR 0.007 (0.004) 0.007 (0.004) 0.029 (0.004) 0.007 (0.004) 0.006 (0.004) 0.004 (0.004) 0.006 (0.004) 0.004 (0.004)
FNR 0.133 (0.066) 0.118 (0.068) 0.541 (0.069) 0.120 (0.068) 0.103 (0.066) 0.065 (0.065) 0.113 (0.067) 0.065 (0.065)
Accuracy 0.986 (0.004) 0.987 (0.004) 0.946 (0.007) 0.987 (0.004) 0.989 (0.004) 0.993 (0.004) 0.988 (0.004) 0.993 (0.004)
F1 1.852 (0.045) 1.868 (0.044) 1.427 (0.071) 1.859 (0.044) 1.883 (0.042) 1.923 (0.041) 1.873 (0.043) 1.923 (0.041)
MCC 0.854 (0.044) 0.870 (0.042) 0.429 (0.071) 0.861 (0.043) 0.885 (0.040) 0.925 (0.038) 0.875 (0.041) 0.925 (0.038)
n = 50
ERR 0.047 (0.027) 0.041 (0.024) 0.072 (0.029) 0.044 (0.026) 0.043 (0.024) 0.040 (0.024) 0.040 (0.023) 0.038 (0.024)
FPR 0.030 (0.019) 0.027 (0.019) 0.043 (0.017) 0.031 (0.024) 0.028 (0.019) 0.027 (0.020) 0.027 (0.019) 0.026 (0.020)
FNR 0.287 (0.306) 0.234 (0.277) 0.609 (0.339) 0.231 (0.268) 0.253 (0.280) 0.226 (0.269) 0.226 (0.262) 0.203 (0.255)
Accuracy 0.953 (0.027) 0.959 (0.024) 0.928 (0.029) 0.956 (0.026) 0.957 (0.024) 0.960 (0.024) 0.960 (0.023) 0.962 (0.024)
F1 1.512 (0.296) 1.561 (0.294) 1.271 (0.284) 1.548 (0.293) 1.544 (0.289) 1.566 (0.293) 1.567 (0.287) 1.585 (0.290)
MCC 0.595 (0.249) 0.650 (0.228) 0.317 (0.289) 0.639 (0.226) 0.632 (0.227) 0.657 (0.223) 0.657 (0.215) 0.678 (0.212)
n = 100
ERR 0.033 (0.017) 0.028 (0.015) 0.059 (0.020) 0.030 (0.017) 0.029 (0.015) 0.026 (0.015) 0.027 (0.015) 0.026 (0.015)
FPR 0.018 (0.012) 0.015 (0.012) 0.031 (0.012) 0.018 (0.014) 0.015 (0.012) 0.014 (0.012) 0.015 (0.012) 0.014 (0.012)
FNR 0.280 (0.217) 0.228 (0.194) 0.582 (0.238) 0.213 (0.195) 0.231 (0.193) 0.205 (0.188) 0.215 (0.193) 0.196 (0.185)
Accuracy 0.967 (0.017) 0.972 (0.015) 0.941 (0.020) 0.970 (0.017) 0.971 (0.015) 0.974 (0.015) 0.973 (0.015) 0.974 (0.015)
F1 1.636 (0.197) 1.687 (0.183) 1.358 (0.214) 1.679 (0.194) 1.683 (0.180) 1.709 (0.181) 1.698 (0.180) 1.717 (0.177)
MCC 0.658 (0.182) 0.711 (0.162) 0.372 (0.216) 0.704 (0.172) 0.707 (0.159) 0.734 (0.157) 0.722 (0.158) 0.742 (0.152)
n = 500
ERR 0.019 (0.007) 0.016 (0.006) 0.061 (0.009) 0.018 (0.006) 0.015 (0.006) 0.011 (0.006) 0.015 (0.006) 0.011 (0.006)
FPR 0.010 (0.005) 0.008 (0.005) 0.032 (0.006) 0.011 (0.006) 0.008 (0.005) 0.006 (0.005) 0.008 (0.005) 0.006 (0.005)
FNR 0.175 (0.095) 0.144 (0.093) 0.608 (0.092) 0.150 (0.093) 0.139 (0.093) 0.096 (0.089) 0.136 (0.093) 0.096 (0.089)
Accuracy 0.981 (0.007) 0.984 (0.006) 0.939 (0.009) 0.982 (0.006) 0.985 (0.006) 0.989 (0.006) 0.985 (0.006) 0.989 (0.006)
F1 1.800 (0.068) 1.832 (0.064) 1.355 (0.097) 1.810 (0.065) 1.837 (0.061) 1.882 (0.059) 1.841 (0.061) 1.883 (0.059)
MCC 0.804 (0.066) 0.836 (0.061) 0.357 (0.097) 0.815 (0.063) 0.842 (0.058) 0.887 (0.055) 0.845 (0.058) 0.887 (0.054)
n = 1000
ERR 0.015 (0.004) 0.013 (0.004) 0.061 (0.007) 0.015 (0.004) 0.012 (0.004) 0.008 (0.004) 0.012 (0.004) 0.008 (0.004)
FPR 0.008 (0.004) 0.007 (0.004) 0.032 (0.004) 0.009 (0.004) 0.006 (0.004) 0.004 (0.004) 0.006 (0.004) 0.004 (0.004)
FNR 0.147 (0.064) 0.119 (0.068) 0.608 (0.065) 0.132 (0.069) 0.110 (0.066) 0.071 (0.065) 0.107 (0.067) 0.070 (0.065)
Accuracy 0.985 (0.004) 0.987 (0.004) 0.939 (0.007) 0.985 (0.004) 0.988 (0.004) 0.992 (0.004) 0.988 (0.004) 0.992 (0.004)
F1 1.836 (0.046) 1.865 (0.042) 1.357 (0.067) 1.839 (0.045) 1.874 (0.040) 1.916 (0.041) 1.878 (0.040) 1.916 (0.041)
MCC 0.839 (0.045) 0.867 (0.041) 0.358 (0.067) 0.841 (0.044) 0.877 (0.038) 0.919 (0.038) 0.880 (0.038) 0.919 (0.038)
50
n = 50
ERR 0.055 (0.028) 0.052 (0.027) 0.084 (0.031) 0.059 (0.029) 0.045 (0.024) 0.046 (0.026) 0.047 (0.025) 0.045 (0.025)
FPR 0.035 (0.018) 0.033 (0.018) 0.050 (0.015) 0.041 (0.024) 0.029 (0.019) 0.030 (0.019) 0.030 (0.019) 0.029 (0.019)
FNR 0.385 (0.332) 0.357 (0.322) 0.732 (0.321) 0.351 (0.318) 0.275 (0.287) 0.293 (0.306) 0.302 (0.301) 0.271 (0.294)
Accuracy 0.945 (0.028) 0.948 (0.027) 0.916 (0.031) 0.941 (0.029) 0.955 (0.024) 0.954 (0.026) 0.953 (0.025) 0.955 (0.025)
F1 1.429 (0.286) 1.455 (0.289) 1.162 (0.251) 1.430 (0.280) 1.521 (0.283) 1.508 (0.294) 1.503 (0.291) 1.525 (0.291)
MCC 0.500 (0.262) 0.530 (0.258) 0.194 (0.266) 0.507 (0.249) 0.605 (0.230) 0.590 (0.249) 0.585 (0.246) 0.610 (0.239)
n = 100
ERR 0.041 (0.019) 0.036 (0.018) 0.073 (0.022) 0.043 (0.020) 0.031 (0.016) 0.029 (0.015) 0.032 (0.016) 0.029 (0.015)
FPR 0.021 (0.012) 0.019 (0.013) 0.038 (0.011) 0.026 (0.015) 0.016 (0.012) 0.015 (0.012) 0.016 (0.012) 0.015 (0.012)
FNR 0.372 (0.231) 0.323 (0.220) 0.715 (0.217) 0.328 (0.228) 0.265 (0.203) 0.245 (0.198) 0.270 (0.204) 0.240 (0.195)
Accuracy 0.959 (0.019) 0.964 (0.018) 0.927 (0.022) 0.957 (0.020) 0.969 (0.016) 0.971 (0.015) 0.968 (0.016) 0.971 (0.015)
F1 1.559 (0.207) 1.608 (0.198) 1.229 (0.194) 1.561 (0.202) 1.663 (0.181) 1.683 (0.180) 1.658 (0.187) 1.688 (0.180)
MCC 0.577 (0.202) 0.627 (0.191) 0.238 (0.199) 0.581 (0.195) 0.683 (0.170) 0.703 (0.166) 0.678 (0.175) 0.708 (0.165)
n = 500
ERR 0.024 (0.007) 0.019 (0.006) 0.070 (0.009) 0.026 (0.007) 0.017 (0.006) 0.012 (0.006) 0.016 (0.006) 0.012 (0.006)
FPR 0.013 (0.006) 0.010 (0.005) 0.037 (0.005) 0.016 (0.006) 0.009 (0.006) 0.006 (0.006) 0.008 (0.005) 0.006 (0.006)
FNR 0.226 (0.092) 0.174 (0.092) 0.701 (0.086) 0.212 (0.098) 0.157 (0.087) 0.108 (0.087) 0.146 (0.091) 0.108 (0.086)
Accuracy 0.976 (0.007) 0.981 (0.006) 0.930 (0.009) 0.974 (0.007) 0.983 (0.006) 0.988 (0.006) 0.984 (0.006) 0.988 (0.006)
F1 1.748 (0.079) 1.801 (0.068) 1.259 (0.090) 1.734 (0.075) 1.819 (0.065) 1.869 (0.063) 1.830 (0.066) 1.870 (0.063)
MCC 0.752 (0.077) 0.805 (0.066) 0.261 (0.090) 0.739 (0.073) 0.823 (0.062) 0.874 (0.059) 0.834 (0.063) 0.874 (0.058)
n = 1000
ERR 0.020 (0.005) 0.015 (0.004) 0.070 (0.007) 0.022 (0.005) 0.013 (0.004) 0.009 (0.004) 0.013 (0.004) 0.009 (0.004)
FPR 0.010 (0.004) 0.008 (0.004) 0.037 (0.004) 0.013 (0.004) 0.007 (0.004) 0.005 (0.004) 0.007 (0.004) 0.004 (0.004)
FNR 0.189 (0.064) 0.139 (0.068) 0.696 (0.060) 0.190 (0.072) 0.127 (0.064) 0.078 (0.065) 0.118 (0.065) 0.077 (0.065)
Accuracy 0.980 (0.005) 0.985 (0.004) 0.930 (0.007) 0.978 (0.005) 0.987 (0.004) 0.991 (0.004) 0.987 (0.004) 0.991 (0.004)
F1 1.793 (0.051) 1.845 (0.045) 1.265 (0.063) 1.772 (0.052) 1.858 (0.043) 1.909 (0.041) 1.867 (0.042) 1.910 (0.041)
MCC 0.795 (0.050) 0.848 (0.044) 0.266 (0.064) 0.774 (0.051) 0.860 (0.042) 0.912 (0.039) 0.869 (0.041) 0.912 (0.039)
n = 50
ERR 0.037 (0.021) 0.033 (0.021) 0.036 (0.021) 0.036 (0.022) 0.055 (0.026) 0.033 (0.021) 0.037 (0.022) 0.033 (0.021)
FPR 0.025 (0.019) 0.023 (0.020) 0.024 (0.020) 0.026 (0.021) 0.034 (0.020) 0.022 (0.020) 0.025 (0.019) 0.022 (0.020)
FNR 0.185 (0.233) 0.161 (0.221) 0.198 (0.246) 0.167 (0.235) 0.433 (0.337) 0.150 (0.218) 0.192 (0.242) 0.151 (0.216)
Accuracy 0.963 (0.021) 0.967 (0.021) 0.964 (0.021) 0.964 (0.022) 0.945 (0.026) 0.967 (0.021) 0.963 (0.022) 0.967 (0.021)
F1 1.606 (0.270) 1.632 (0.278) 1.604 (0.284) 1.613 (0.283) 1.421 (0.301) 1.640 (0.280) 1.600 (0.275) 1.639 (0.279)
MCC 0.690 (0.190) 0.718 (0.192) 0.687 (0.213) 0.698 (0.206) 0.480 (0.287) 0.727 (0.192) 0.683 (0.201) 0.726 (0.189)
n = 100
ERR 0.028 (0.016) 0.025 (0.014) 0.031 (0.015) 0.028 (0.015) 0.041 (0.018) 0.021 (0.014) 0.028 (0.015) 0.022 (0.014)
FPR 0.015 (0.012) 0.013 (0.013) 0.016 (0.013) 0.017 (0.013) 0.022 (0.013) 0.011 (0.012) 0.015 (0.012) 0.012 (0.013)
FNR 0.213 (0.200) 0.192 (0.183) 0.265 (0.202) 0.187 (0.184) 0.379 (0.227) 0.148 (0.170) 0.220 (0.196) 0.155 (0.172)
Accuracy 0.972 (0.016) 0.975 (0.014) 0.969 (0.015) 0.973 (0.015) 0.959 (0.018) 0.979 (0.014) 0.972 (0.015) 0.978 (0.014)
F1 1.693 (0.179) 1.721 (0.177) 1.657 (0.197) 1.701 (0.185) 1.550 (0.213) 1.761 (0.176) 1.688 (0.178) 1.755 (0.179)
MCC 0.718 (0.158) 0.746 (0.153) 0.680 (0.183) 0.725 (0.162) 0.569 (0.207) 0.787 (0.145) 0.713 (0.158) 0.781 (0.150)
n = 500
ERR 0.017 (0.006) 0.016 (0.006) 0.032 (0.008) 0.017 (0.006) 0.019 (0.006) 0.009 (0.006) 0.016 (0.006) 0.010 (0.006)
FPR 0.009 (0.005) 0.008 (0.006) 0.017 (0.006) 0.010 (0.006) 0.010 (0.006) 0.005 (0.006) 0.008 (0.006) 0.005 (0.006)
FNR 0.153 (0.093) 0.148 (0.083) 0.314 (0.089) 0.141 (0.085) 0.178 (0.084) 0.080 (0.088) 0.149 (0.086) 0.081 (0.087)
Accuracy 0.983 (0.006) 0.984 (0.006) 0.968 (0.008) 0.983 (0.006) 0.981 (0.006) 0.991 (0.006) 0.984 (0.006) 0.990 (0.006)
F1 1.823 (0.065) 1.830 (0.062) 1.658 (0.083) 1.822 (0.065) 1.799 (0.071) 1.900 (0.065) 1.828 (0.063) 1.899 (0.064)
MCC 0.828 (0.062) 0.835 (0.059) 0.662 (0.082) 0.827 (0.061) 0.804 (0.069) 0.905 (0.059) 0.833 (0.059) 0.904 (0.059)
n = 1000
ERR 0.015 (0.004) 0.013 (0.004) 0.034 (0.006) 0.014 (0.004) 0.013 (0.004) 0.007 (0.004) 0.013 (0.004) 0.007 (0.004)
FPR 0.008 (0.004) 0.007 (0.004) 0.018 (0.004) 0.008 (0.004) 0.007 (0.004) 0.003 (0.004) 0.007 (0.004) 0.004 (0.004)
FNR 0.139 (0.067) 0.127 (0.060) 0.337 (0.069) 0.123 (0.063) 0.125 (0.064) 0.061 (0.066) 0.122 (0.064) 0.062 (0.066)
Accuracy 0.985 (0.004) 0.987 (0.004) 0.966 (0.006) 0.986 (0.004) 0.987 (0.004) 0.993 (0.004) 0.987 (0.004) 0.993 (0.004)
F1 1.847 (0.046) 1.860 (0.040) 1.641 (0.064) 1.853 (0.041) 1.861 (0.041) 1.928 (0.041) 1.865 (0.040) 1.927 (0.041)
MCC 0.849 (0.045) 0.862 (0.039) 0.642 (0.063) 0.855 (0.040) 0.863 (0.040) 0.930 (0.039) 0.867 (0.039) 0.929 (0.039)
51
n = 50
ERR 0.046 (0.025) 0.047 (0.025) 0.065 (0.028) 0.051 (0.026) 0.045 (0.024) 0.038 (0.023) 0.048 (0.025) 0.039 (0.024)
FPR 0.029 (0.019) 0.030 (0.019) 0.039 (0.017) 0.034 (0.021) 0.029 (0.019) 0.025 (0.019) 0.031 (0.019) 0.025 (0.020)
FNR 0.301 (0.298) 0.304 (0.297) 0.517 (0.338) 0.342 (0.310) 0.288 (0.292) 0.208 (0.258) 0.323 (0.307) 0.223 (0.271)
Accuracy 0.954 (0.025) 0.953 (0.025) 0.935 (0.028) 0.949 (0.026) 0.955 (0.024) 0.962 (0.023) 0.952 (0.025) 0.961 (0.024)
F1 1.518 (0.284) 1.514 (0.281) 1.344 (0.278) 1.480 (0.288) 1.529 (0.280) 1.596 (0.283) 1.501 (0.287) 1.586 (0.292)
MCC 0.588 (0.245) 0.585 (0.242) 0.393 (0.276) 0.547 (0.258) 0.601 (0.237) 0.675 (0.217) 0.570 (0.253) 0.665 (0.233)
n = 100
ERR 0.033 (0.017) 0.035 (0.017) 0.061 (0.021) 0.040 (0.018) 0.032 (0.016) 0.025 (0.015) 0.038 (0.017) 0.025 (0.015)
FPR 0.017 (0.012) 0.018 (0.012) 0.032 (0.012) 0.023 (0.014) 0.017 (0.012) 0.013 (0.012) 0.020 (0.012) 0.013 (0.012)
FNR 0.276 (0.211) 0.300 (0.212) 0.586 (0.225) 0.325 (0.210) 0.265 (0.205) 0.182 (0.186) 0.332 (0.209) 0.193 (0.188)
Accuracy 0.967 (0.017) 0.965 (0.017) 0.939 (0.021) 0.960 (0.018) 0.968 (0.016) 0.975 (0.015) 0.962 (0.017) 0.975 (0.015)
F1 1.642 (0.177) 1.620 (0.182) 1.350 (0.203) 1.580 (0.194) 1.654 (0.177) 1.733 (0.163) 1.592 (0.188) 1.724 (0.165)
MCC 0.662 (0.168) 0.640 (0.174) 0.362 (0.205) 0.599 (0.187) 0.674 (0.167) 0.755 (0.144) 0.611 (0.181) 0.746 (0.148)
n = 500
ERR 0.020 (0.007) 0.022 (0.007) 0.064 (0.010) 0.025 (0.007) 0.017 (0.006) 0.011 (0.006) 0.023 (0.007) 0.011 (0.006)
FPR 0.011 (0.006) 0.011 (0.006) 0.034 (0.005) 0.013 (0.006) 0.009 (0.006) 0.006 (0.006) 0.012 (0.006) 0.006 (0.006)
FNR 0.189 (0.094) 0.204 (0.092) 0.639 (0.093) 0.227 (0.088) 0.159 (0.091) 0.094 (0.089) 0.224 (0.086) 0.096 (0.088)
Accuracy 0.980 (0.007) 0.978 (0.007) 0.936 (0.010) 0.975 (0.007) 0.983 (0.006) 0.989 (0.006) 0.977 (0.007) 0.989 (0.006)
F1 1.785 (0.073) 1.770 (0.073) 1.320 (0.094) 1.741 (0.079) 1.817 (0.067) 1.884 (0.063) 1.751 (0.076) 1.881 (0.062)
MCC 0.790 (0.070) 0.774 (0.071) 0.323 (0.094) 0.745 (0.077) 0.821 (0.064) 0.888 (0.058) 0.755 (0.074) 0.886 (0.057)
n = 1000
ERR 0.017 (0.005) 0.017 (0.004) 0.063 (0.007) 0.020 (0.005) 0.013 (0.004) 0.008 (0.004) 0.019 (0.004) 0.008 (0.004)
FPR 0.009 (0.004) 0.009 (0.004) 0.033 (0.004) 0.011 (0.004) 0.007 (0.004) 0.004 (0.004) 0.010 (0.004) 0.004 (0.004)
FNR 0.162 (0.067) 0.167 (0.066) 0.631 (0.068) 0.183 (0.061) 0.126 (0.065) 0.069 (0.065) 0.183 (0.060) 0.071 (0.065)
Accuracy 0.983 (0.005) 0.983 (0.004) 0.937 (0.007) 0.980 (0.005) 0.987 (0.004) 0.992 (0.004) 0.981 (0.004) 0.992 (0.004)
F1 1.821 (0.050) 1.816 (0.047) 1.333 (0.069) 1.794 (0.051) 1.859 (0.045) 1.918 (0.041) 1.800 (0.050) 1.917 (0.041)
MCC 0.823 (0.049) 0.818 (0.046) 0.334 (0.069) 0.796 (0.050) 0.861 (0.044) 0.920 (0.038) 0.802 (0.049) 0.919 (0.039)
n = 50
ERR 0.049 (0.027) 0.048 (0.026) 0.075 (0.031) 0.054 (0.027) 0.046 (0.025) 0.042 (0.024) 0.049 (0.025) 0.042 (0.024)
FPR 0.031 (0.019) 0.031 (0.019) 0.045 (0.017) 0.036 (0.023) 0.030 (0.018) 0.027 (0.019) 0.031 (0.019) 0.027 (0.019)
FNR 0.333 (0.315) 0.324 (0.308) 0.629 (0.343) 0.337 (0.315) 0.290 (0.294) 0.249 (0.277) 0.335 (0.307) 0.250 (0.278)
Accuracy 0.951 (0.027) 0.952 (0.026) 0.925 (0.031) 0.947 (0.027) 0.954 (0.025) 0.958 (0.024) 0.951 (0.025) 0.958 (0.024)
F1 1.487 (0.292) 1.496 (0.288) 1.249 (0.276) 1.468 (0.291) 1.520 (0.282) 1.556 (0.286) 1.488 (0.287) 1.557 (0.289)
MCC 0.559 (0.259) 0.569 (0.251) 0.289 (0.285) 0.541 (0.259) 0.596 (0.236) 0.637 (0.228) 0.560 (0.252) 0.638 (0.232)
n = 100
ERR 0.037 (0.018) 0.037 (0.018) 0.067 (0.022) 0.040 (0.018) 0.033 (0.017) 0.028 (0.016) 0.037 (0.017) 0.028 (0.016)
FPR 0.019 (0.012) 0.019 (0.012) 0.035 (0.012) 0.023 (0.014) 0.017 (0.012) 0.014 (0.012) 0.019 (0.013) 0.014 (0.012)
FNR 0.319 (0.213) 0.328 (0.219) 0.648 (0.236) 0.321 (0.218) 0.274 (0.211) 0.224 (0.200) 0.322 (0.216) 0.229 (0.204)
Accuracy 0.963 (0.018) 0.963 (0.018) 0.933 (0.022) 0.960 (0.018) 0.967 (0.017) 0.972 (0.016) 0.963 (0.017) 0.972 (0.016)
F1 1.603 (0.188) 1.597 (0.196) 1.290 (0.206) 1.581 (0.198) 1.647 (0.188) 1.697 (0.182) 1.603 (0.195) 1.693 (0.184)
MCC 0.624 (0.178) 0.618 (0.186) 0.302 (0.210) 0.602 (0.188) 0.669 (0.174) 0.720 (0.164) 0.624 (0.185) 0.716 (0.167)
n = 500
ERR 0.023 (0.007) 0.023 (0.007) 0.068 (0.010) 0.024 (0.007) 0.018 (0.006) 0.012 (0.006) 0.022 (0.007) 0.012 (0.006)
FPR 0.012 (0.006) 0.012 (0.006) 0.036 (0.005) 0.014 (0.006) 0.009 (0.006) 0.006 (0.006) 0.012 (0.006) 0.006 (0.006)
FNR 0.221 (0.094) 0.217 (0.094) 0.681 (0.090) 0.220 (0.092) 0.162 (0.091) 0.105 (0.088) 0.211 (0.090) 0.107 (0.088)
Accuracy 0.977 (0.007) 0.977 (0.007) 0.932 (0.010) 0.976 (0.007) 0.982 (0.006) 0.988 (0.006) 0.978 (0.007) 0.988 (0.006)
F1 1.753 (0.078) 1.756 (0.076) 1.278 (0.089) 1.743 (0.078) 1.813 (0.067) 1.872 (0.061) 1.763 (0.075) 1.871 (0.062)
MCC 0.757 (0.076) 0.760 (0.074) 0.279 (0.089) 0.748 (0.076) 0.818 (0.064) 0.877 (0.056) 0.768 (0.073) 0.875 (0.058)
n = 1000
ERR 0.020 (0.005) 0.019 (0.004) 0.067 (0.007) 0.020 (0.005) 0.014 (0.004) 0.008 (0.004) 0.018 (0.004) 0.008 (0.004)
FPR 0.010 (0.004) 0.010 (0.004) 0.035 (0.004) 0.011 (0.004) 0.007 (0.004) 0.004 (0.004) 0.009 (0.004) 0.004 (0.004)
FNR 0.192 (0.066) 0.179 (0.065) 0.666 (0.066) 0.181 (0.065) 0.130 (0.066) 0.077 (0.066) 0.171 (0.061) 0.078 (0.065)
Accuracy 0.980 (0.005) 0.981 (0.004) 0.933 (0.007) 0.980 (0.005) 0.986 (0.004) 0.992 (0.004) 0.982 (0.004) 0.992 (0.004)
F1 1.791 (0.051) 1.804 (0.047) 1.297 (0.067) 1.793 (0.049) 1.856 (0.044) 1.911 (0.042) 1.813 (0.049) 1.911 (0.042)
MCC 0.793 (0.051) 0.807 (0.046) 0.298 (0.067) 0.795 (0.048) 0.859 (0.042) 0.914 (0.040) 0.816 (0.048) 0.913 (0.039)
52
n = 50
ERR 0.062 (0.030) 0.058 (0.029) 0.086 (0.032) 0.069 (0.032) 0.050 (0.026) 0.048 (0.027) 0.055 (0.028) 0.048 (0.027)
FPR 0.038 (0.018) 0.036 (0.018) 0.051 (0.015) 0.046 (0.025) 0.032 (0.018) 0.031 (0.019) 0.035 (0.019) 0.030 (0.019)
FNR 0.471 (0.343) 0.428 (0.341) 0.754 (0.310) 0.452 (0.342) 0.341 (0.305) 0.318 (0.310) 0.400 (0.331) 0.316 (0.311)
Accuracy 0.938 (0.030) 0.942 (0.029) 0.914 (0.032) 0.931 (0.032) 0.950 (0.026) 0.952 (0.027) 0.945 (0.028) 0.952 (0.027)
F1 1.371 (0.288) 1.406 (0.291) 1.148 (0.246) 1.359 (0.284) 1.478 (0.281) 1.497 (0.294) 1.432 (0.293) 1.500 (0.297)
MCC 0.430 (0.279) 0.469 (0.276) 0.176 (0.261) 0.422 (0.274) 0.551 (0.245) 0.573 (0.256) 0.499 (0.271) 0.576 (0.259)
n = 100
ERR 0.048 (0.020) 0.044 (0.019) 0.076 (0.022) 0.052 (0.021) 0.036 (0.017) 0.032 (0.017) 0.040 (0.018) 0.032 (0.017)
FPR 0.025 (0.012) 0.023 (0.013) 0.039 (0.010) 0.031 (0.016) 0.018 (0.012) 0.016 (0.013) 0.021 (0.012) 0.017 (0.013)
FNR 0.439 (0.235) 0.409 (0.235) 0.749 (0.213) 0.407 (0.235) 0.313 (0.211) 0.274 (0.211) 0.361 (0.221) 0.283 (0.212)
Accuracy 0.952 (0.020) 0.956 (0.019) 0.924 (0.022) 0.948 (0.021) 0.964 (0.017) 0.968 (0.017) 0.960 (0.018) 0.968 (0.017)
F1 1.491 (0.205) 1.523 (0.210) 1.198 (0.192) 1.485 (0.206) 1.615 (0.195) 1.654 (0.196) 1.569 (0.199) 1.648 (0.199)
MCC 0.509 (0.201) 0.542 (0.205) 0.206 (0.196) 0.506 (0.199) 0.637 (0.182) 0.676 (0.180) 0.589 (0.190) 0.669 (0.184)
n = 500
ERR 0.029 (0.007) 0.027 (0.008) 0.072 (0.010) 0.032 (0.008) 0.021 (0.007) 0.014 (0.006) 0.024 (0.007) 0.014 (0.006)
FPR 0.015 (0.006) 0.014 (0.006) 0.038 (0.005) 0.019 (0.007) 0.011 (0.006) 0.007 (0.006) 0.013 (0.006) 0.008 (0.006)
FNR 0.283 (0.092) 0.256 (0.093) 0.720 (0.088) 0.276 (0.095) 0.197 (0.089) 0.126 (0.089) 0.226 (0.088) 0.129 (0.089)
Accuracy 0.971 (0.007) 0.973 (0.008) 0.928 (0.010) 0.968 (0.008) 0.979 (0.007) 0.986 (0.006) 0.976 (0.007) 0.986 (0.006)
F1 1.689 (0.080) 1.716 (0.081) 1.238 (0.089) 1.670 (0.083) 1.777 (0.074) 1.850 (0.067) 1.748 (0.077) 1.848 (0.069)
MCC 0.693 (0.079) 0.720 (0.080) 0.239 (0.090) 0.675 (0.082) 0.782 (0.072) 0.855 (0.064) 0.752 (0.075) 0.852 (0.066)
n = 1000
ERR 0.026 (0.005) 0.022 (0.005) 0.070 (0.007) 0.027 (0.006) 0.017 (0.004) 0.010 (0.004) 0.019 (0.005) 0.010 (0.004)
FPR 0.014 (0.004) 0.012 (0.004) 0.037 (0.003) 0.016 (0.005) 0.009 (0.004) 0.005 (0.004) 0.010 (0.004) 0.005 (0.004)
FNR 0.255 (0.065) 0.215 (0.067) 0.699 (0.062) 0.239 (0.073) 0.162 (0.062) 0.093 (0.067) 0.183 (0.064) 0.094 (0.066)
Accuracy 0.974 (0.005) 0.978 (0.005) 0.930 (0.007) 0.973 (0.006) 0.983 (0.004) 0.990 (0.004) 0.981 (0.005) 0.990 (0.004)
F1 1.725 (0.057) 1.767 (0.053) 1.262 (0.065) 1.722 (0.056) 1.823 (0.050) 1.894 (0.042) 1.800 (0.052) 1.893 (0.043)
MCC 0.727 (0.056) 0.769 (0.053) 0.263 (0.065) 0.724 (0.055) 0.825 (0.049) 0.896 (0.040) 0.802 (0.051) 0.895 (0.041)
n = 50
ERR 0.072 (0.027) 0.053 (0.027) 0.060 (0.026) 0.070 (0.033) 0.049 (0.025) 0.041 (0.024) 0.053 (0.026) 0.041 (0.024)
FPR 0.047 (0.015) 0.037 (0.017) 0.041 (0.017) 0.052 (0.025) 0.035 (0.017) 0.031 (0.019) 0.037 (0.017) 0.030 (0.019)
FNR 0.620 (0.358) 0.343 (0.346) 0.466 (0.360) 0.420 (0.368) 0.295 (0.322) 0.200 (0.276) 0.342 (0.338) 0.200 (0.277)
Accuracy 0.928 (0.027) 0.947 (0.027) 0.940 (0.026) 0.930 (0.033) 0.951 (0.025) 0.959 (0.024) 0.947 (0.026) 0.959 (0.024)
F1 1.229 (0.267) 1.417 (0.291) 1.342 (0.290) 1.321 (0.272) 1.454 (0.289) 1.531 (0.298) 1.422 (0.290) 1.538 (0.300)
MCC 0.279 (0.278) 0.503 (0.263) 0.412 (0.282) 0.399 (0.265) 0.546 (0.247) 0.635 (0.226) 0.508 (0.260) 0.639 (0.230)
n = 100
ERR 0.064 (0.020) 0.042 (0.020) 0.056 (0.019) 0.052 (0.023) 0.036 (0.018) 0.029 (0.017) 0.038 (0.019) 0.028 (0.017)
FPR 0.037 (0.011) 0.026 (0.011) 0.033 (0.011) 0.035 (0.016) 0.022 (0.012) 0.018 (0.013) 0.023 (0.012) 0.018 (0.013)
FNR 0.646 (0.249) 0.350 (0.258) 0.549 (0.254) 0.372 (0.275) 0.269 (0.234) 0.199 (0.217) 0.307 (0.247) 0.197 (0.219)
Accuracy 0.936 (0.020) 0.958 (0.020) 0.944 (0.019) 0.948 (0.023) 0.964 (0.018) 0.971 (0.017) 0.962 (0.019) 0.972 (0.017)
F1 1.269 (0.206) 1.515 (0.213) 1.354 (0.212) 1.455 (0.218) 1.589 (0.203) 1.661 (0.214) 1.557 (0.212) 1.666 (0.215)
MCC 0.284 (0.210) 0.543 (0.201) 0.373 (0.212) 0.486 (0.209) 0.619 (0.182) 0.693 (0.186) 0.586 (0.196) 0.696 (0.189)
n = 500
ERR 0.063 (0.009) 0.027 (0.008) 0.057 (0.010) 0.029 (0.008) 0.020 (0.007) 0.019 (0.007) 0.022 (0.007) 0.019 (0.007)
FPR 0.037 (0.005) 0.018 (0.006) 0.034 (0.006) 0.019 (0.006) 0.014 (0.006) 0.014 (0.006) 0.015 (0.006) 0.013 (0.006)
FNR 0.653 (0.098) 0.215 (0.121) 0.578 (0.114) 0.219 (0.116) 0.136 (0.099) 0.127 (0.097) 0.160 (0.103) 0.127 (0.096)
Accuracy 0.937 (0.009) 0.973 (0.008) 0.943 (0.010) 0.971 (0.008) 0.980 (0.007) 0.981 (0.007) 0.978 (0.007) 0.981 (0.007)
F1 1.283 (0.096) 1.694 (0.091) 1.353 (0.108) 1.679 (0.090) 1.771 (0.079) 1.780 (0.083) 1.748 (0.082) 1.785 (0.086)
MCC 0.286 (0.096) 0.702 (0.090) 0.357 (0.108) 0.688 (0.087) 0.779 (0.074) 0.788 (0.077) 0.755 (0.079) 0.792 (0.081)
n = 1000
ERR 0.063 (0.006) 0.023 (0.006) 0.052 (0.007) 0.024 (0.006) 0.016 (0.005) 0.018 (0.005) 0.018 (0.005) 0.019 (0.005)
FPR 0.036 (0.003) 0.015 (0.004) 0.030 (0.004) 0.017 (0.004) 0.012 (0.004) 0.013 (0.004) 0.013 (0.004) 0.013 (0.004)
FNR 0.650 (0.064) 0.178 (0.084) 0.515 (0.083) 0.188 (0.085) 0.104 (0.072) 0.130 (0.072) 0.119 (0.070) 0.132 (0.074)
Accuracy 0.937 (0.006) 0.977 (0.006) 0.948 (0.007) 0.976 (0.006) 0.984 (0.005) 0.982 (0.005) 0.982 (0.005) 0.981 (0.005)
F1 1.291 (0.065) 1.744 (0.061) 1.420 (0.077) 1.727 (0.059) 1.817 (0.056) 1.792 (0.058) 1.802 (0.054) 1.788 (0.060)
MCC 0.293 (0.064) 0.749 (0.061) 0.423 (0.077) 0.732 (0.058) 0.822 (0.054) 0.796 (0.056) 0.807 (0.052) 0.793 (0.058)
53

A A C H - D R: Lternative Pproaches For Omputing Ighest Ensity Egions

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A A C H - D R: Lternative Pproaches For Omputing Ighest Ensity Egions

Uploaded by

Copyright:

Available Formats

A LTERNATIVE A PPROACHES FOR C OMPUTING

H IGHEST-D ENSITY R EGIONS

Nina Deliu Brunero Liseo

MRC – Biostatistics Unit, University of Cambridge (UK) brunero.liseo@uniroma1.it

Keywords Statistical intervals · Density estimation · Copula models · k-nearest neighborhood

100(1 − α)% HDR

Coverage = 100(1 - α)% = 95%

95% HDR with KDE

2 HDR: Formulation and Computation

2.1 Density quantile approach

2.2 One-class neighbor machines (OCNM) approach

The function g is called either a (a) sparsity or a (b) concentration measure.

where I denotes the indicator function, and

2.3 Neighborhood-quantile approach

3 Alternative Measures for Building HDRs

3.1 kNN-Euclidean distance

3.2 kNN-CDF distance

-4 -3 x1 = -2 x2 = -1.6 x3 = -0.72 x4 = 0.5 1 2 3 4

• If x = x(k) , M2 (x, x(k) , 1) = 0 by definition.

3.3 ϵ-neighborhood multivariate CDF distance

with V = {(x1 + ϵ1 , x2 + ϵ2 ), (x1 − ϵ1 , x2 + ϵ2 ), (x1 + ϵ1 , x2 − ϵ2 ), (x1 − ϵ1 , x2 − ϵ2 )}.

The proof follows from basic probability and calculus theory.

3.4 Copula-based measures

represented by the joint distribution of F1 (X1 ), . . . , Fd (Xd ) and is derived as:

For d = 2, it is straightforward to see that

Unimodal: Unimodal: Bimodal: Quadrimodal:

S9 S10 S11 S12

S13 S14 S15 S16

4.2 Evaluated measures

4.3 Performance metrics

negative the others:

4.4 Simulation results

4.4.1 Simplex scenario: Dirichlet

Scenario S1 (Unimodal): Gaussian copula – Student-t marginals

Scenario S6 (Unimodal): Student-t copula – Gaussian marginals

Scenario S16 (Quadrimodal): Clayton copula – Gaussian mixture marginals

M3 :ϵ-CDF Performances are robust to sample size, with optimal ϵ = 0.10.

S17: Dirichlet (1,1,2) Dirichlet (1,1,2) Copula

4.5 MAGIC data

5 Discussion and Conclusion

50 100 250 500 1000 50 100 250 500 1000

False Positive Rate (FPR) False Negative Rate (FNR)

50 100 250 500 1000 50 100 250 500 1000

95% HDR on MAGIC Data

95% HDR on MAGIC Data (Measure Averaging)

Wand, M. P. and Jones, M. C. (1994a). Kernel Smoothing. Chapman and Hall/CRC.

A Derivation of the Dirichlet copula

B Supporting Details for Hyperparameter Choices

Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet

Figure 9: Performance results in all evaluated scenarios for different values

Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet

hpof theihyperparameter k in M1 :kNN-Eucl.

Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet

hpof theihyperparameter k in M1 :kNN-Eucl.

Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet

hpof theihyperparameter k in M1 :kNN-Eucl.

0.1 0.2 0.3 0.4 0.5 0.6 0.7

Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet

0.00 0.01 0.02 0.03 0.04 0.05

0.2 0.3 0.4 0.5 0.6 0.7 0.8

Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet

Unimodal Unimodal w heavy tails Bimodal Quadrimodal Dirichlet