Accred Qual Assur
DOI 10.1007/s00769-017-1292-6
GENERAL PAPER
Bayesian analysis of homogeneity studies in the production
of reference materials
Adriaan M. H. van der Veen1
Received: 18 March 2017 / Accepted: 30 September 2017
Ó Springer-Verlag GmbH Germany 2017
Abstract For almost two decades, the batch homogeneity
in the production of reference materials has been evaluated
using analysis of variance (ANOVA) to determine the
between-bottle standard deviation. This approach replaced
at that time the use of the F-test in ANOVA to determine
whether the ratio of the mean squares MSbetween =MSwithin is
statistically significant. Problems arise when
MSbetween \MSwithin , because classical ANOVA provides a
negative between-bottle variance, which is then often set to
zero. By using a Bayesian hierarchical model, based on the
same assumptions as traditional ANOVA, we show that
even if MSbetween \MSwithin , there can be a relevant level of
between-bottle inhomogeneity to account for. The Baye-
sian analysis produces a nonzero value for the between-
bottle standard deviation, which dismisses the practice of
setting this standard deviation to 0. At the same time, it
dismisses the current guidance given in ISO Guide 35
under these circumstances. Finally, it is shown that tradi-
tional ANOVA, meta-analysis methods and Bayesian
analysis give very similar answers as long as
MSbetween [ MSwithin , so there is no need to discourage
using these methods in favour of a Bayesian analysis,
provided that the repeatability of the measurement method
used to conduct the between-bottle homogeneity study is
sufficient to characterise the dispersion across the bottles
(items) in a batch of a reference material.
Keywords Measurement uncertainty  Bayesian analysis 
Hierarchical model  Meta-analysis  Analysis of variance 
Reference material  Homogeneity study
& Adriaan M. H. van der Veen
avdveen@vsl.nl
1
VSL, Thijsseweg 11, 2629 JA Delft, The Netherlands
Introduction
The current practice of viewing the batch (‘‘between-bot-
tle’’) homogeneity as an uncertainty component to be
included in the uncertainty budget of the certified value of
a reference material started with a publication by Pauwels
et al. [1] in the late 1990s. In the following years, the idea
was further developed and extended to stability studies [2],
and eventually, an evaluation method based on analysis of
variance (ANOVA) was proposed [3, 4]. The evaluation of
the between-bottle homogeneity has become an integral
part of reference material production since the publication
of the third edition of ISO Guide 35 [5], implementing
relevant requirements of ISO Guide 34 [6] (recently
superseded by ISO 17034 [7]). Also in proficiency testing
[8], this type of assessment is essential to characterise the
dispersion of the property across the items (bottles) pro-
duced that are going to be used with a single assigned
value, valid for the entire batch [9].
The classical approach to using ANOVA in homo-
geneity studies of reference materials relies on the
relationships between the mean squares between
(MSbetween ) and within groups (MSwithin ) on one hand and
the expectations of the repeatability and between-bottle
variances on the other [3, 4]. As long as
MSbetween  MSwithin , this way of evaluating homogeneity
study data works well. When this condition is not fulfilled,
the between-bottle standard deviation cannot be computed,
because it would involve taking the square root of a neg-
ative number. Situations where MSbetween \MSwithin arise
when the differences due to between-bottle homogeneity
are small and the repeatability of the measurement method
is not sufficient to quantify such differences, or in the
presence of outliers or stragglers in the dataset. Till to date,
there are still experimenters that consider an F-test telling
123

that there is ‘‘no significant inhomogeneity’’ as desired
outcome, see, e.g. [10], but such an assessment is inap-
propriate, for it assesses the batch homogeneity against the
wrong metric, namely the repeatability standard deviation
of the method, whereas it is supposed to be the standard
uncertainty from the characterisation of the reference
material, possibly combined with an uncertainty contribu-
tion due to the instability of the property value.
Insufficient repeatability of the method can be the result
of choosing a poorly repeatable method for conducting the
homogeneity study, a too low number of replicates, but
such situations should be avoided [1, 4]. Nevertheless, after
material processing, homogenising and bottling, the level
of between-bottle heterogeneity can be so small that it is
hard to characterise it, even if a suitable measurement and
experimental set-up is chosen. The problem arising from
insufficient repeatability, and consequently
MSbetween \MSwithin has been addressed by two proposals to
set the between-bottle variance to zero, or a nonzero value,
based on the uncertainty associated with the mean squares
‘‘within groups’’ [11]. Gelman et al. [12] comment that
setting the between-group standard deviation in ANOVA to
zero when MSbetween \MSwithin is not necessarily what the
data tells. This attitude is followed by ISO Guide 35 [5],
by providing an alternative. The uncertainty contribution is
computed as [4, 5]
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffipffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ubb ¼ MSwithin =n 4 2=ðaðn  1ÞÞ
where a denotes the number of bottles and n the number of
replicate measurements in the homogeneity study. It
reflects the uncertainty associated with the repeatability of
the method [11]. This alternative has been criticised as it
would not properly address the dispersion due to between-
bottle heterogeneity [13]. In this paper, we present an
alternative that sounds from a statistical point of view and
characterises even for datasets where MSbetween \MSwithin ,
the between-bottle homogeneity.
The alternative to using traditional ANOVA [4], meta-
analysis, or restricted maximum-likelihood estimation [13] is
to use a Bayesian approach to ANOVA [12, 14] which enables
computing the within- and between-bottle standard devia-
tions, even if MSbetween \MSwithin , at the expense that the
Bayesian approach is computationally more involved. The
Bayesian analysis provides (samples of) the probability dis-
tributions for the within- and between-bottle standard
deviations. From these probability distributions, values for the
corresponding standard deviations can be obtained. The
model presented builds forth on a model developed for
reworking the ANOVA example in the GUM (Guide to the
expression of uncertainty in measurement) [15, example H.5].
To assess the performance of a Bayesian analysis for
evaluating batch homogeneity, a dataset has been chosen
123
Accred Qual Assur
for a homogeneity study of ten gas mixtures. These gas
mixtures were obtained by decanting a gravimetrically
prepared synthetic natural gas mixture [16]. The set chosen
was one that confirmed the successful operation of the
decanting procedure, thus a dataset for which small values
for the between-bottle standard deviation are expected. For
one component (nitrogen), MSbetween \MSwithin . The out-
come of the Bayesian analysis is compared with two
classical methods: traditional ANOVA and a meta-analysis
method using the DerSimonian–Laird (DL) model [17],
based on the same random-effects model. Neither of these
methods satisfactorily deals with the situation where
MSbetween \MSwithin , but both methods would be considered
as simpler alternatives for the Bayesian analysis otherwise.
Experimental
Ten gas mixtures were prepared by connecting ten evacu-
ated gas cylinders with a water volume of 1 L to a
manifold, equipped with 10 positions. A gravimetrically
prepared synthetic natural gas mixture was obtained from a
commercial speciality gas supplier. This mixture was
decanted into the 10 cylinders by connecting it to a mass
flow controller which in turn was connected to the mani-
fold. The gas flow rate was controlled during decanting, to
mitigate the gas temperature drop due to Joule–Thomson
ð1Þ
cooling [16]. The selected dataset originates from the
experiments performed after optimising the settings of the
decanting method.
The analyses were performed on an Agilent 7980A gas
chromatograph, configured for the analysis of the compo-
sition of natural gas and biogas. The back channel consists
of a HayeSep Q pre-column and HayeSep T? column and
a thermal conductivity detector (TCD) and is used for the
separation of methane and carbon dioxide from the other
components. Nitrogen, hydrogen and oxygen are separated
and detected on the auxiliary channel equipped with a
Hayesep/Molsieve column and TCD. The front channel is
equipped with a PDMS-column and a flame ionisation
detector (FID). This channel is used for the separation of
the hydrocarbons save methane. The carrier gas is helium
for the front and back channels and argon for the auxiliary
channel. The data collection is performed using ChemS-
tation software. The temperature programme is the same
for all channels and starts with 35  C for 6 min, followed
by increasing the temperature with 10  C min1 to 100  C
and hold for 0.5 min.
The GC was calibrated with a suite of Primary Standard
Gas Mixtures (PSMs). The calibration was done using a
multipoint calibration approach with at least 7 PSMs for
each component [18]. The data processing was done in
accordance with ISO 6974 [19, 20]. For all components, a
Accred Qual Assur
quadratic polynomial was used. The peak areas observed
during calibration and the homogeneity study were cor-
rected for instrument drift. The validity of the calibration
was confirmed using a quality control mixture. The cur-
vatures of the calibration functions are irrelevant for the
calculations involved in the homogeneity study; the cur-
vature is small, and the amount-of-substance fractions of
the ten gas mixtures subjected to the homogeneity study are
close enough that nonlinearity of the analyser does not
influence the outcome. The measurements on the 10 gas
mixtures were taken in a single sequence under repeata-
bility conditions. For each of the gas mixtures, six replicate
injections were performed, of which the first one was dis-
carded to eliminate carry-over effects.
Modelling and data analysis
Basic model
The methods for evaluating a between-bottle homogeneity
in this paper serve the same purpose as the ones proposed
previously for evaluating between-bottle homogeneity
studies [2, 3, 4] and are based on largely the same
assumptions. The main objective is still to compute the
between-bottle standard deviation sbb from a dataset of a
fully nested one-way design. As before, the measurement
data (e.g. group means and replicates within the groups) is
assumed to be normally distributed, but this is not a
requirement for applying Bayesian methods; quite on the
contrary, Bayesian methods are easier to adapt when other
assumptions concerning the distribution of the measure-
ment data are needed [12, 21]. All methods can of course
be adapted accordingly when another experimental set-up
is chosen.
All data analysis methods in this paper are based on the
same random-effects model. This model has been the basis
for the development of the traditional ANOVA method for
assessing the batch homogeneity of reference materials [4].
This random-effects model can be written as [3, 4]
Yij ¼ l þ Ai þ eij ð2Þ
where Yij denotes the value of the jth measurement of the
ith bottle (item), l the expected value, Ai the bias of the ith
bottle (item), and eij the random measurement error in
observation j on item i. In most cases, it is assumed that
eij  Nð0; r2 Þ where r2 denotes the repeatability variance.
Furthermore, it is assumed that Ai  Nð0; s2 Þ where s2
denotes the between-group variance. In case of a between-
bottle homogeneity study, s is the between-bottle standard
deviation, often denoted by rbb or sbb [4].
Traditional ANOVA
Traditional ANOVA is well covered in the literature [3–5]
and the most widely used statistical method in the litera-
ture. The expression for the between-bottle variance is
obtained from the relationships between the mean squares
between and within groups and the expectations of the
corresponding variances [3, 12]. The expression for the
between-bottle standard deviation, here denoted by s, takes
the form
MSbetween  MSwithin
s2 ¼ ð3Þ
n
where n denotes the number of replicates in one group. s is
identical to the sbb as used elsewhere in the literature (e.g.
[4, 5, 11]). The pooled repeatability
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ffi standard deviation of
the method equals MSwithin [3].
Meta-analysis
Meta-analysis is widely used to combine results from dif-
ferent studies [17, 22]. It also finds use to combine results
from different laboratories in an interlaboratory compar-
ison [23] and reference material characterisation [24].
Meta-analysis can also be used to determine the between-
group standard deviation s in a between-bottle homo-
geneity study, as an alternative to traditional ANOVA (see
section Traditional ANOVA).
Based on the same random-effects model (see Eq. (2)), a
meta-analysis can be performed, which, among others, can
provide a value for s2 . In fact, traditional ANOVA can be
viewed as belonging to the larger family of models used for
meta-analysis [22]. Several studies have been performed to
study the performance of the various models used for meta-
analysis. From these studies, it can be inferred that the
DerSimonian–Laird (DL) model pairs computational ease
with good characterisation of the dispersion of the property
value across groups. The expression for the between-bottle
variance reads as [17]
P  2
i wi Yi  Y ða  1Þ
s2 ¼ P P 2 P ð4Þ
i wi  i wi = i wi
where Yi denotes a group mean, wi ¼ 1=s2i , si the within-
P P
group standard deviation, and Y ¼ i wi Yi = i wi . Similar
to traditional ANOVA, in meta-analysis s2 ¼ 0 when the
result of Eq. (4) is negative. The meta-analysis using the
DL-method does not use pooling of the within-group
variances.
The weights wi in Eq. (4) will for well-designed
between-bottle homogeneity studies usually be very simi-
lar. Even if the within-group standard deviations si are
assumed to be samples of the same probability distribution,
123

these will be different. It is then still appropriate to use
weighing, albeit that the influence of the weighing will
generally be small.
Bayesian analysis
Bayesian methods are in some respects fundamentally
different from classical statistical methods. Bayesian
methods use the measurement data to update the proba-
bility density function of one or more parameters. Hence,
the probability density functions of the parameter before
the updating need to be specified [12].
Two variants of the Bayesian model are proposed, one
with pooling of the within-group standard deviations, and
another without pooling. In the case of pooling, there are
three parameters in the Bayesian model: the grand mean l,
the within-group standard deviation r and the between-
group standard deviation s [12]. The Bayesian analysis
provides for these three parameters posterior probability
density functions, from which, among others, the between-
bottle standard deviation can be obtained. For all parame-
ters, weakly informative prior distributions are selected, to
(1) allow the data to dominate in the Bayesian analysis and
(2) improve the performance of the Markov Chain Monte
Carlo method (MCMC), used for calculating a sample of
the posterior distributions [12, 25].
In most real-world situations, Bayesian computations
are much more involved than their classical counterparts.
Only in a very limited number of cases, analytical
expressions exist for, e.g., the mean and standard deviation
of a parameter after a Bayesian inference. Such cases
typically involve the use of non-informative, conjugate or
semi-conjugate probability density functions to model prior
information [26]. Whereas in developing Bayesian models
such priors can be convenient as a starting point [12], priors
should be elicited on the knowledge at hand [27].
Weakly informative priors are often appropriate in
metrology, because (1) they allow the data to dominate and
(2) they ensure that the posterior is ‘‘proper’’, i.e. the
posterior fulfils the requirements of a probability density
function. Using priors often implies that a Monte Carlo
method is required to generate a sample from the posterior.
From this sample, the mean, standard deviation and cov-
erage interval, among others, can be computed. This kind
of Monte Carlo method is known as Markov Chain Monte
Carlo (MCMC) [12]. These methods can be implemented
in various ways, but require resources well beyond those
usually employed for data processing in metrology (e.g.
mainstream spreadsheet software). Furthermore, coding
models involving MCMC requires a great deal of knowl-
edge about the properties and performance of these
computational methods sampling and simulation
123
Accred Qual Assur
techniques and ways to evaluate the performance of the
calculation.
All calculations involving MCMC were implemented in
R [28], using the package RStan [29]. This package
enables writing the models in a specific language called
Stan [12]. To monitor the performance of the MCMC
methods, the statistic R^ was used; this parameter is defined
as [12]
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
n1 1 B
R^ ¼ þ
n nW
where B and W are the between-chain and within-chain
variance, and n is the length of the chain. At convergence,
R^  1 for each scalar parameter in the model. Furthermore,
trace plots [12] were used for inspecting the behaviour of
the chains in the MCMC calculation. The sampler in
RStan has some advantages for non-experts in Bayesian
computation in that they attempt to configure themselves
during a warm-up phase, after which it generates the
samples for an MCMC chain. Multiple chains can be used
and combined, which is not a property of all samplers
[12, 30].
As prior for the mean l, the normal distribution is
chosen with the nominal value of the amount-of-substance
fraction of the parent mixture as mean and a suitably large
standard deviation. This standard deviation is chosen to be
wider than the dispersion of the group means and set to 5
times the repeatability standard deviation. For all standard
deviations, the folded Cauchy distribution is used as prior,
which performs well for both between-group and within-
group standard deviations [21]. Whereas choosing a non-
informative probability distribution as prior for r would
lead to acceptable results, this is not the case for the
between-group standard deviation s [21]. The scale
parameter (A) for the folded Cauchy distribution is inferred
from prior knowledge about the analytical system (for the
within-group standard deviation) and the envisaged batch
homogeneity (for the between-group standard deviation),
respectively. The location parameter of the half Cauchy
distribution is set to 0. Most of the density of the half
Cauchy distribution is then concentrated on the interval
[0, A] [21].
The hierarchical model with pooling is shown in
Table 1. The model without pooling is obtained by
declaring sig as an array (for each group an unknown
within-group standard deviation) and adapting the code
accordingly. The package RStan [29] enables writing the
models in a specific programming language called Stan
(see Table 1) [12] and running the calculations from R. The
sampler in RStan has some advantages for non-experts in
Bayesian computation in that it attempts to configure itself
during a warm-up phase, after which it generates the
samples for an MCMC chain.
Accred Qual Assur

Table 1 Hierarchical model for a between-bottle homogeneity study with pooling

When performing the MCMC calculations, four chains of
sufficient length were used. The efforts in optimising the
chain length have been kept to a minimum, as the compu-
tation is fast and the only concern has been that the values for
the parameters could be reproduced to all meaningful digits.
It should be emphasised that these calculations, like all
Monte Carlo calculations, give slightly different answers
each time the calculation is run. As long as these differences
are small enough to be meaningless, the calculation was
deemed sufficiently accurate. Four chains have been used
with 25 000 iterations. The warm-up phase was set to be 5000
samples long, leaving 20 000 per chain for the posterior.

123

To improve the numerical stability of the model, the
data are rescaled by dividing the group means by their
mean and adapting the within-group standard deviations
accordingly. In the parameter block (see Table 1), the
scaled parameters are declared. The unscaled parameters
are computed in the block generated quantities.
The rescaling also aids when using the model for other
datasets with a similar structure.
Results
Measurement data
The results of the homogeneity study are shown in Fig. 1.
The datasets for the various components are quite different
in terms of the dispersion of the group means and how this
dispersion relates to the expanded uncertainty computed as
2  si where si denotes the standard deviation of the group
mean. For all components save ethane and nitrogen, the
dispersion of the group means is larger than would be
expected based on the computed expanded uncertainty.
The variability of the within-group standard deviations
differs appreciably among components. The datasets for n-
hexane and carbon dioxide show the largest heteroscedas-
ticity in this respect. The datasets from, e.g., methane,
ethane and the pentanes look much more homoscedastic,
i.e. have similar within-group standard deviations.
Pooling of the within-group standard deviation is often
appropriate in between-bottle homogeneity studies,
because the data are usually obtained under repeatability
conditions [4]. it is in most cases unlikely that the
repeatability standard deviation fluctuates appreciably from
bottle to bottle. There is of course variability in the cal-
culated standard deviations, which is to a large extent due
to the limited number of repeat observations.
Traditional ANOVA
The results of evaluation of the between-bottle homo-
geneity from Fig. 1 using one-way ANOVA [4] are given
in Table 2. As the within- and between-group standard
deviations are small, these are given in lmol mol1 (parts-
per-million). Save nitrogen, all other datasets have a non-
zero between-bottle standard deviation. For nitrogen, the
MSbetween is smaller than MSwithin . This dataset alone would
be sufficient motivation for using Bayesian inference
instead of the traditional method for evaluating this
homogeneity study.
For pnitrogen, using
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ffi Eq. (1), ubb ¼ 1:48 lmol mol1
where MSwithin =n ¼ 3:13 lmol mol1 . The latter has
been proposed as a ‘‘safe’’ value for sbb in case
123
Accred Qual Assur
MSbetween \MSwithin [13]. For this dataset, these two alter-
natives differ appreciably.
The results in Table 2 have been used in this work as a
benchmark in the validation of the Bayesian models with
and without pooling, save for nitrogen for which traditional
ANOVA is unable to provide a nonzero between-bottle
standard deviation.
Meta-analysis
The results of the meta-analysis using the DL-model are
shown in Table 3. The mean values l^ using meta-analysis
are very close to those from traditional ANOVA (see
Table 2). The between-bottle standard deviation s^ is larger
in the meta-analysis than for the traditional analysis of
variance for ethane, iso-butane, n-butane, carbon dioxide
and methane. The between-bottle standard deviation is
smaller for propane and n-hexane. For the other compo-
nents, s^ from the meta-analysis is very similar to that from
traditional ANOVA. Also, meta-analysis provides for all
components nonzero values for s, save for nitrogen.
Bayesian analysis, pooled within-group standard
deviation
In the case of a Bayesian analysis involving MCMC, it is
important to assess the convergence of the iterative cal-
culation [12]. Trace plots give, as function of the iteration
number, the value of the parameter. For both standard
deviations, the trace plots are shown for the scaled
parameters tau_ and sig_, which are defined as s=y and
r=y, respectively. These scaled standard deviations are
good approximations to the corresponding coefficients of
variation and can be interpreted as such. This presentation
supports easier comparison of the posterior distributions
(posteriors) for the standard deviations across components.
The trace plots for ethane are shown in Fig. 2. These
plots are typical for parameters for which the MCMC
calculation converged. The results for the four chains are
very similar, and there is no meaningful between-chain
^ which is
variability. This is also reflected in the value for R,
indeed very close to one for the three parameters (see
Table 4). The results are given with the number of mean-
ingful digits derived from the standard error of the MCMC
(SE).
In Table 4, the column labelled mean gives the mean of
the parameter, SE the standard deviation of the mean due to
the random error in the MCMC and SD the standard
deviation of parameter mean. The next two columns give
the limits of the symmetric 95 % credible interval. The
column labelled neff gives the approximate number of
independent samples and the last column the value of R. ^ R
returns also the characteristics for all other model
Accred Qual Assur

ethane propane

4.703
3.501
fraction (cmol/mol)

fraction (cmol/mol)
3.499

4.701
3.497

4.699
3.495

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
mixture mixture
0.12348

iso−butane n−butane

0.12254
fraction (cmol/mol)

fraction (cmol/mol)
0.12342

0.12248
0.12336

0.12242
1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
mixture mixture
iso−pentane n−pentane
0.03124
0.03059

fraction (cmol/mol)
fraction (cmol/mol)

0.03121
0.03056

0.03118
0.03053

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
mixture mixture
n−hexane nitrogen
0.07600
fraction (cmol/mol)

fraction (cmol/mol)
0.426
0.07594

0.424
0.07588

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
mixture mixture
carbon dioxide methane
5.036

85.98
fraction (cmol/mol)

fraction (cmol/mol)
5.032

85.95
85.92
5.028

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
mixture mixture

Fig. 1 Amount-of-substance fractions of the components (in cmol mol1 , or, equivalently, %) in the 10 synthetic natural gas mixtures. The half-
widths of the uncertainty bars represent the expanded uncertainty, computed 2si where si denotes the standard deviation of the group mean

123
 Accred Qual Assur

Table 2 Results of the evaluation of the between-bottle homogeneity data using traditional analysis of variance
Component l MSbetween MSwithin s^ r^
cmol mol1 cmol2 mol2 cmol2 mol2 lmol mol1 lmol mol1

Ethane 3.49779 1:694 106 1:184 106 3.19 10.88

6 7
Propane 4.70073 4:278 10 2:213 10 9.01 4.70
iso-Butane 0.123405 2:139 109 1:796 1010 0.20 0.13
n-Butane 0.122493 2:783 109 1:993 1010 0.23 0.14
iso-Pentane 0.030568 2:667 1010 9:220 1011 0.06 0.10
n-Pentane 0.031213 4:436 1010 4:385 1011 0.09 0.07
9 10
n-Hexane 0.075964 4:408 10 1:193 10 0.29 0.11
Nitrogen 0.425144 3:249 107 4:903 107 7.00
Carbon dioxide 5.03265 9:014 106 1:320 106 12.4 11.5
4
Methane 85.9601 6:472 10 8:288 105 106 91
Each dataset comprises 10 gas mixtures and 5 replicates per mixture [16]

Table 3 Results of the evaluation of the between-bottle homogeneity data using meta-analysis and the DerSimonian–Laird model
Component l^ uðlÞ
^ s^
cmol mol1 cmol mol1 lmol mol1

Ethane 3.497798 0.000201 4.49

Propane 4.700733 0.000257 7.86
iso-Butane 0.123404 0.000005 0.15
n-Butane 0.122493 0.000006 0.18
iso-Pentane 0.030568 0.000002 0.07
n-Pentane 0.031212 0.000003 0.09
n-Hexane 0.075965 0.000006 0.19
Nitrogen 0.425069 0.000073
Carbon dioxide 5.03267 0.00040 11.7
Methane 85.9602 0.0038 114

0.00125

3.499
0.0004
0.00100

3.498 chain chain chain

m (cmol/mol)

0.00075
1 1 0.0003 1
σ/y
τ /y

2 2 2
3 0.00050 3 3

3.497 4 4 4

0.00025 0.0002

3.496
0.00000
5000 10000 15000 20000 25000 5000 10000 15000 20000 25000 5000 10000 15000 20000 25000
Iteration number Iteration number Iteration number

Fig. 2 Trace plots for l, s=y, and r=y for ethane

123
Accred Qual Assur

Table 4 Results of the MCMC calculation for ethane

Mean SE SD q0:025 q0:975 neff R^

l 3.4977911 0.0000008 0.0003141 3.4971687 3.4984109 145081 1.0000

s 0.0003508 0.0000009 0.0002891 0.0000139 0.0010754 109659 1.0001
r 0.0008777 0.0000003 0.0000990 0.0006905 0.0010781 152000 1.0000
Given are the mean, the standard error from the MCMC (SE), the standard deviation (SD), the 2.5 % and 97.5 % quantiles (q0:025 and q0:975 ;
respectively), all expressed in cmol mol1 , the effective number of samples ðneff Þ and R^

parameters, but these are omitted for brevity. It is worth estimated to be s^ ¼ 1:72 lmol mol1 , which lies in
noting that the s^ from ANOVA lies in the 95 % coverage between
p ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ubb ¼ 1:48 lmol mol1 and
1 
interval of the same parameter from the Bayesian analysis. MS =n
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
within ¼ 3:13 lmol mol . Both u bb and
The repeatability standard deviation r^ from ANOVA lies MSwithin =n lie in the 95 % coverage interval of s.
just above the 97.5 % quantile of the same parameter from Comparing the results of nitrogen with those of ethane
the Bayesian analysis. (see Table 4), it is worth noting that the standard deviation
The posterior distributions of l, s, and r for ethane are of s for both parameters is quite large. These datasets give
shown in Fig. 3. The posterior of s is very asymmetric. It is with the Bayesian model very similar estimates for s and r
worth noting that the mode of the posterior of r=y is much (when expressed as coefficients of variation), yet the
larger than that of s=y. In the context of the GUM [15], classical methods (ANOVA and meta-analysis) treat the
from the posteriors of s, and r, only the location is of two datasets very differently.
relevance; the ‘‘uncertainty of the uncertainty’’ is not fur- The value of s^ for nitrogen compares well neither to the
ther relevant to the evaluation of measurement uncertainty. practice of setting it to zero, nor to the value of ubb from
The means of the posteriors of s and r are taken as esti- Eq. (1). Expressed as coefficient of variation (0.040 %), it
mates for the corresponding standard deviations. is substantially larger than pthat of carbon ffi dioxide (0.003 %)
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
The fitting of the model shown in Table 1 for the and methane (0.010 %). MSwithin =n is substantially larger
datasets of other components converged well. Notwith- than s^, thus providing a ‘‘safe value’’ for this uncertainty
standing the fact that the Bayesian model uses the scaled component.
parameters for the computations, the results for propane The results of the Bayesian analysis using the model
and n-hexane indicated that convergence was slower than with pooling are summarised in Table 6. The between-
for the other datasets. bottle standard deviation is larger than that from traditional
Computationally, the dataset of nitrogen was not more ANOVA (see Table 2) for ethane, propane and carbon
challenging than the others. The trace plots of the Bayesian dioxide and smaller for iso-pentane and methane. For the
analysis (see Fig. 4) show good convergence. The poste- other components, the between-bottle standard deviations
riors are shown in Fig. 5. The posterior of s for nitrogen from both approaches are very similar.
looks as skewed as that for ethane (see Fig. 3). Notwithstanding that a weakly informative prior is used
The output for nitrogen for l, s, and r is shown in for s, it is useful to perform an analysis of the sensitivity of
Table 5. The between-bottle standard deviation is the estimated between-bottle homogeneity standard
6000
1000 1500 2000

12000
4000

8000
density

density

density
2000

4000
500
0

3.4960 3.4975 3.4990 0.0000 0.0004 0.0008 0.0012 0.00015 0.00030

μ (cmol/mol) τ/ y σ/ y

Fig. 3 Posteriors of l, s=y, and r=y for ethane

123
 Accred Qual Assur

0.003 0.0018

0.4255

0.0015
chain 0.002 chain chain
m (cmol/mol)

1 1 1

σ/ y
τ/y
0.4250 2 2 2
3 3 0.0012 3
4 4 4
0.001

0.4245 0.0009

0.000

5000 10000 15000 20000 25000 5000 10000 15000 20000 25000 5000 10000 15000 20000 25000
Iteration number Iteration number Iteration number

Fig. 4 Trace plots for l, s=y, and r=y for nitrogen

1500

2500
3000

1000
density

density

density
1500
500
1000

0 500
0

0.4245 0.4250 0.4255 0.0000 0.0010 0.0020 0.0030 0.0008 0.0012 0.0016
μ (cmol/mol) τ /y σ /y

Fig. 5 Posteriors of l, s=y, and r=y for nitrogen

Table 5 Results of the MCMC calculation for nitrogen

Mean SE SD q0:025 q0:975 neff R^

l 0.4251443 0.0000004 0.0001680 0.4248122 0.4254757 152000 1.0000

s 0.0001720 0.0000004 0.0001439 0.0000061 0.0005320 152000 1.0000
r 0.0004808 0.0000001 0.0000557 0.0003752 0.0005935 152000 1.0000
Given are the mean, the standard error from the MCMC (SE), the standard deviation (SD), the 2.5 % and 97.5 % quantiles (q0:025 and q0:975 ,
respectively), all expressed in cmol mol1 , the effective number of samples (neff ) and R^

deviation s^ as a function of the scale parameter A of the
Cauchy distribution. For ease of comparison between
components, the results and scale parameters are given
relative to the corresponding l^ for the component. The
superiority of this distribution over reference priors for a
between-group standard deviation is well covered in the
literature [21].
The results of the sensitivity analysis for selected
components are shown in Fig. 6. Only the computed
between-bottle homogeneity standard deviation for nitro-
gen shows a noticeable influence of the scale parameter A
of the Cauchy distribution. It should however be kept in
mind that this parameter has been evaluated over a very
wide range, in this case, from 0.05 % to 1 %. Usually, from
inspecting the group means (¼ means of the bottles), it
should be possible to obtain a realistic value for the scale
parameter A in accordance with the recommendations from
Gelman et al. [12].
Bayesian analysis, no pooling of the within-group
standard deviation
There can be circumstances where, even in a between-
bottle homogeneity study, it is unclear whether pooling of

123
Accred Qual Assur

Table 6 Values and standard deviations of the fitted parameters l, s, and r for the homogeneity data expressed as amount-of-substance fractions
using the model with pooling of the within-group standard deviations
Component l^ sðlÞ
^ s^ sÞ
sð^ r^ sðrÞ
^
cmol mol1 cmol mol1 lmol mol1 lmol mol1 lmol mol1 lmol mol1

Ethane 3.497790 0.000314 3.51 2.89 8.78 0.99

Propane 4.700730 0.000335 9.35 2.97 3.98 0.44
iso-Butane 0.123405 0.000008 0.20 0.08 0.11 0.01
n-Butane 0.122493 0.000009 0.24 0.08 0.11 0.01
iso-Pentane 0.030568 0.000003 0.04 0.03 0.07 0.01
n-Pentane 0.031213 0.000003 0.08 0.04 0.06 0.01
n-Hexane 0.075964 0.000010 0.30 0.08 0.06 0.01
Nitrogen 0.425144 0.000168 1.72 1.44 4.81 0.56
Carbon dioxide 5.03265 0.00050 13.6 4.9 6.7 0.7
Methane 85.9601 0.0041 90 46 79 9

ethane rescaling the data and lengthening the warm-up phase of

0.05 propane
the MCMC. The trace plots look very similar to those
iso-butane
n-butane obtained for the model with pooling (see Figs. 2 and 4) and
0.04 nitrogen are therefore not presented here. The same applies to the
carbon dioxide
methane
obtained posteriors, which are in shape also very similar to
their counterparts in section Bayesian analysis, pooled
0.03
within-group standard deviation (see Figs. 3 and 5).
τ /μ (%)

The results of the Bayesian analysis using a hierarchical

0.02 model without pooling are summarised in Table 7. The
results from the model without pooling of the within-group
0.01 standard deviations (see Table 7) compare well to those for
the model with pooling (see Table 6). This is not surprising,
because the measurements have been taken under repeata-
0.00
0.001 0.01
bility conditions in the same experiment. The between-bottle
A/μ standard deviation for ethane is larger than for the model with
pooling and that for nitrogen smaller. That these standard
Fig. 6 Sensitivity analysis of s^ for different values of the scale deviations differ most (in relative sense) is not surprising,
parameter A of the Cauchy distribution. According to Gelman [21],
because for these components the posteriors of s are rela-
this scale parameter should not be smaller than the between-group
standard deviation tively broad, indicating that the derived estimates s^ are more
uncertain than those for the other components.

the within-group standard deviations is justified or appro-

priate. The underlying assumption for pooling is that all
within-group standard deviations can be viewed as draws
from the same probability distribution. Some experi-
menters would prefer not at all to rely on such an
assumption. Hence, the same dataset (see Fig. 1) has been
evaluated with a model without pooling as well.
The model without pooling is somewhat more difficult
to handle computationally than the model with pooling.
This is largely due to the fact that now the within-group
standard deviations are calculated based on 5 of degrees of
freedom and this is for the sampler harder than with the
large number of degrees of freedom in the case of pooling.
Eventually, the problem was satisfactorily solved by
Discussion and conclusions
A Bayesian model is presented for the evaluation of between-
bottle homogeneity studies in the production of reference and
proficiency test materials. The model provides, for all com-
ponents in a challenging dataset with small values for the
between-bottle standard deviation, a nonzero value for the
latter standard deviation. The models with and without pool-
ing give very similar answers for most components. The larger
differences for the between-bottle standard deviation (s) for
ethane and nitrogen are partly due to the fact that the posterior
distributions of s are wider for these components, which
indicates more uncertainty about the estimates s^.

123
 Accred Qual Assur

Table 7 Values and standard deviations of the fitted parameters l and s for the homogeneity data, expressed as amount-of-substance fractions
using the model without pooling of the within-group standard deviations
Component l^ sðlÞ
^ s^ sÞ
sð^
cmol mol1 cmol mol1 lmol mol1 lmol mol1

Ethane 3.497790 0.000212 4.23 2.23

Propane 4.700731 0.000321 9.65 2.65
iso-Butane 0.123404 0.000007 0.20 0.06
n-Butane 0.122493 0.000008 0.24 0.07
iso-Pentane 0.030568 0.000003 0.07 0.02
n-Pentane 0.031213 0.000003 0.10 0.03
n-Hexane 0.075964 0.000010 0.31 0.08
Nitrogen 0.425086 0.000096 1.21 0.92
Carbon dioxide 5.03267 0.00048 13.5 3.9
Methane 85.9601 0.0041 115 33

The estimated between-bottle standard deviation s^ for
nitrogen is more sensitive to the choice of the scale
parameter of the prior (the Cauchy distribution) than for the
other components. The estimates of the between-bottle
standard deviation of the other components are very
weakly dependent on the choice of this scale parameter.
Whether this difference in behaviour has a relationship
with the fact that the classical statistical methods are
unable to provide a value for the between-bottle standard
deviation for nitrogen cannot be concluded on the basis of
this dataset alone.
Notwithstanding that there is a desire to make the GUM
‘‘Bayesian’’ [31], classical statistical methods can be
employed as well in most circumstances, as they provide
generally very similar values for the parameters of interest
(i.e. l, s, and r). This conclusion is supported by per-
forming the evaluation of the homogeneity data using
traditional analysis of variance (ANOVA) and meta-anal-
ysis using the DerSimonian–Laird model. Save for
nitrogen, the results from these classical methods are very
similar to the results from the Bayesian models.
The Bayesian analysis gives a value for the between-
bottle standard deviation which is not even close to the ubb
as has been proposed [4] and described in ISO Guide 35
[5] in case of MSbetween \MSwithin . In case of classical
statistical methods, it would be more appropriate to use
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ffi s007690000261
MSwithin =n [13] as a ‘‘safe value’’ under these
circumstances.
Acknowledgements The work presented in this paper was funded by
the Ministry of Economic Affairs of the Netherlands.
References
1. Pauwels J, Lamberty A, Schimmel H (1998) Homogeneity testing
of reference materials. Accred Qual Assur 3(2):51–55. doi:10.
1007/s007690050186
2. Pauwels J, van der Veen AMH, Lamberty A, Schimmel H (2000)
Evaluation of uncertainty of reference materials. Accred Qual
Assur 5(3):95–99. doi:10.1007/s007690050020
3. van der Veen AMH, Pauwels J (2000) Uncertainty calculations in
the certification of reference materials. 1. Principles of analysis of
variance. Accred Qual Assur 5(12):464–469. doi:10.1007/
s007690000237
4. van der Veen AMH, Linsinger TP, Pauwels J (2001) Uncertainty
calculations in the certification of reference materials. 2. Homo-
geneity study. Accred Qual Assur 6(1):26–30. doi:10.1007/
s007690000238
5. ISO Guide 35 (2006) Reference materials—general and statistical
principles for certification. International Organization for Stan-
dardization, Geneva
6. ISO Guide 34 (2009) General requirements for the competence of
reference material producers. International Organization for
Standardization, Geneva
7. ISO 17034 (2016) General requirements for the competence of
reference material producers. International Organization for
Standardization, Geneva
8. ISO 13528 (2015) Statistical methods for use in proficiency
testing by interlaboratory comparison. International Organization
for Standardization, Geneva
9. ISO/IEC 17043 (2010) Conformity assessment—general
requirements for proficiency testing. International Organization
for Standardization, Geneva
10. Ulrich JC, Sarkis JES, Hortellani MA (2015) Homogeneity study
of candidate reference material in fish matrix. J Phys Conf Ser
575:012,040. doi:10.1088/1742-6596/575/1/012040
11. Linsinger TPJ, Pauwels J, van der Veen AMH, Schimmel H,
Lamberty A (2001) Homogeneity and stability of reference
materials. Accred Qual Assur 6(1):20–25. doi:10.1007/
12. Gelman A, Carlin J, Stern H, Dunson D, Vehtari A, Rubin D
(2013) Bayesian data analysis, 3rd edn. Chapman and Hall/CRC,
Boca Raton
13. Ellison SLR (2015) Homogeneity studies and ISO Guide
35:2006. Accred Qual Assur 20(6):519–528. doi:10.1007/s00769-
015-1162-z
14. Hoff PD (2009) A first course in Bayesian statistical methods.
Springer, New York. doi:10.1007/978-0-387-92407-6
15. BIPM, IEC, IFCC, ILAC, ISO, IUPAC, IUPAP, OIML (2008)
Guide to the expression of uncertainty in measurement, JCGM
100:2008, GUM 1995 with minor corrections. BIPM, Sèvres
16. Beelen R (2016) Preparation of a homogeneous set of PT mate-
rials. Technical report, VSL, Delft

123
Accred Qual Assur
17. DerSimonian R, Laird N (1986) Meta-analysis in clinical trials.
Controlled Clin Trials 7(3):177–188. doi:10.1016/0197-
2456(86)90046-2. http://www.sciencedirect.com/science/article/
pii/0197245686900462
18. ISO 6143 (2001) Gas analysis—comparison methods for deter-
mining and checking the composition of calibration gas mixtures,
2nd edn. International Organization for Standardization, Geneva
19. ISO 6974-1 (2012) Natural gas—determination of composition
with defined uncertainty by gas chromatography—part 1:
guidelines for tailored analysis. International Organization for
Standardization, Geneva
20. ISO 6974-2 (2012) Natural gas—determination of composition
with defined uncertainty by gas chromatography—part 2: mea-
suring-system characteristics and statistics for processing of data.
International Organization for Standardization, Geneva
21. Gelman A (2006) Prior distributions for variance parameters in
hierarchical models (comment on article by Browne and Draper).
Bayesian Anal 1:515–534. doi:10.1214/06-ba117a
22. DerSimonian R, Kacker R (2007) Random-effects model for
meta-analysis of clinical trials: an update. Contemp Clin Trials
28(2):105–114. doi:10.1016/j.cct.2006.04.004
23. Kacker RN (2004) Combining information from interlaboratory
evaluations using a random effects model. Metrologia
41(3):132–136. doi:10.1088/0026-1394/41/3/004
24. Rivier C, Désenfant M, Crozet M, Rigaux C, Roudil D, Tufféry
B, Ruas A (2014) Use of an excess variance approach for the
certification of reference materials by interlaboratory comparison.
Accred Qual Assur 19(4):269–274. doi:10.1007/s00769-014-
1066-3
25. Stan Developers Team (2016) Stan modeling language. User’s
guide and reference manual. http://mc-stan.org/documentation/
26. Klauenberg K, Wübbeler G, Mickan B, Harris P, Elster C (2015)
A tutorial on bayesian normal linear regression. Metrologia
52(6):878–892. doi:10.1088/0026-1394/52/6/878
27. O’Hagan A (2014) Eliciting and using expert knowledge in
metrology. Metrologia 51(4):S237–S244. doi:10.1088/0026-
1394/51/4/s237
28. R Core Team (2016) R: a language and environment for statis-
tical computing. R Foundation for Statistical Computing, Vienna,
Austria. https://www.R-project.org/
29. Carpenter B, Gelman A, Hoffman M, Lee D, Goodrich B,
Betancourt M, Brubaker MA, Guo J, Li P, Riddell A (2016) Stan:
a probabilistic programming language. J Stat Softw (in press)
30. Klauenberg K, Elster C (2016) Markov chain monte carlo
methods: an introductory example. Metrologia 53(1):S32–S39.
doi:10.1088/0026-1394/53/1/s32
31. Bich W, Cox MG, Dybkaer R, Elster C, Estler WT, Hibbert B,
Imai H, Kool W, Michotte C, Nielsen L, Pendrill L, Sidney S, van
der Veen AMH, Wöger W (2012) Revision of the ‘‘Guide to the
expression of Uncertainty in Measurement’’. Metrologia
49(6):702–705 http://stacks.iop.org/0026-1394/49/i=6/a=702
123