Sultan Qaboos University

Department of Electrical and Computer

Engineering

Lecture Notes

For the course

MCTE4155-MEASUREMENT AND INSTRUMENTATION

Chapter 3: Error Analysis

By: Faïçal Mnif

Error Analysis 2

CHAPTER 3
MEASUREMENT AND ERROR

Our discussion in this chapter will consider the analysis of data to determine errors, precision
and general validity of experimental measurements. The experimentalist should always know
the validity of data. The automobile test engineer must know the accuracy of the speedometer
and gas gage in order to express the fuel-economy performance with confidence. A nuclear
engineer must know the accuracy and precision of many instruments just to make some
simple radioactivity measurements with confidence. In order to specify the performance of
an amplifier, an electrical engineer must know the accuracy with which the appropriate
measurements of voltage, distortion, etc. have been conducted.
Errors will creep into all experiments regardless of the care which is exerted. Some of these
errors are of random nature, and some will be due to gross blunders on the part of the
experimenter.

3.1. Causes and types of experimental errors

An experimental error is an experimental error. If the experimenter knew what the error was,
he would correct it and it would no longer be an error. In other words, the real errors in
experimental data are those factors that are always vague to some extent and carry some
amount of uncertainty. Our task is to determine just how uncertain a particular observation
may be and to devise a consistent way of specifying the uncertainty in analytical form. A
reasonable definition of experimental uncertainty as the possible value the error may have.
This uncertainty may vary a great deal depending on the circumstances of the experiment.

Types of errors: At this point we may mention some of the types of errors that may cause
uncertainty in an experimental measurement:
- First: there can always be gross blunders in apparatus or instrument construction
which may invalidate the data. Hopefully, the careful experimenter will be able
to eliminate most of these errors.
- Second, there may be certain fixed errors which will cause repeated readings to
be in error by roughly the same amount but for some unknown reason. The fixed
errors are sometimes called the systematic errors.
- Third, there are the random errors, which may be caused by personal
fluctuations, random electronic fluctuations in the instrument, various influences
Chapter 3: Measurement and error 3

of external conditions, etc. These random errors usually follow a certain

statistical distribution.

3.2. Accumulation of Experimental errors

All elements of an instrumentation system have accuracy limits specified by the
manufacturer. For instance a Voltmeter may have an accuracy specified as ± 2% of full scale
values. The voltmeter can be expected to operate within these limits if it is used with care,
properly maintained and periodically calibrated. Because of limits on accuracy, the
instrument will introduce error in a measurement when it is placed in an instrumentation
system. However this error is known provided the Voltmeter is operating within
specifications.
Consider the input-output curve in Fig. 3.1 That characterizes the Voltmeter. The deviation d
is defined as the product of the accuracy and the full scale value of the response of the
Voltmeter. Lines drawn parallel to true response of the Voltmeter, but displaced by ± d,
form the upper and lower accuracy bounds defining the actual response of the
instrument. The area between these two bounds gives the region where the Voltmeter
(or any other instrument) is operation within the manufacturer’s specifications. If the
instrument is operated at one-half scale, the deviation d remains constant.
d = % accuracy  full scale

Figure 3.1 Accuracy bounds for an instrument operating with specifications

Example 3.1:
Error Analysis 4

A voltage V is read using the Voltmeter described above. The full scale range of the
instrument is 220 Volts. The reading indicates 50 Volts. Determine the absolute and the
relative error of the reading.

Solution:
The accuracy of the Voltmeter is 2% of the full scale range, then
2
d  220 = 4.4 Volts
100
The absolute error is then 4.4 Volts. The reading of the Voltmeter can be then expressed as
V  50  4.4 Volts
By tacking the worst possible variations in the reading, we could mention that
Vmax  54.4 Volts

Vmin  45.6 Volts

The percent of the error can be determined by
4.4
 100  8.8%
50
The reading can be now expressed as
V  50Volts  8.8%

Notation: we note the absolute error as wV and the relative error of a reading as wV (%) .

3.3 Uncertainty analysis

3.3.1 Error analysis on a commonsense basis

Suppose that we have satisfied ourselves with the uncertainty in some basic experimental
measurements, taking into consideration such factors as instrument accuracy, competence of
the people using the instruments, etc. Eventually the primary measurements must be
combined to calculate a particular result that is desired. We shall be interested in knowing the
uncertainty in the final result due to the uncertainties in the primary measurements. One rule
of thumb that could be used is that the error in the result is equal to the maximum error in any
parameter used to calculate the result. Another commonsense analysis would combine all the
errors in the most detrimental way in order to determine the maximum error in the final result.

Example 3.2
Consider the calculation of electric power from
P  VI
Chapter 3: Measurement and error 5

where measurements are performed to V and I as

V  100V  2V
I  10A  0.2A

the nominal value of the power is 100 10  1000 Watts. By taking the worst possible
variations in voltage and current, we could calculate

Pmax  (100  2)(10  0.2)  1040.4 Watts

Pmin  (100  2)(10  0.2)  960.4 Watts

Thus using this method of calculation, the uncertainty in the power is +4.04 % and –3.96 %.
It is quite unlikely that the power would be in error by these amounts because the voltmeter
variations would not correspond with the ammeter variations. When the voltmeter reads an
extreme “high”, there is no reason that the ammeter must also read an extreme “high” at that
particular instant; indeed the combination is most unlikely.

3.3.2 Uncertainty analysis

Suppose a set of measurements is made an the uncertainty in each measurement may be
expressed with the chances. These measurements are then used to calculate some desired
result of the experiments. We wish to estimate the uncertainty in the calculated result on the
basis of the uncertainties in the primary measurements. The result R is a given function of the
independent variables x1 , x2 , x3 ,..., xn . Thus

R  R( x1 , x2 , x3 ,..., xn )

Let wR be the uncertainty in the result and w1 , w2 , w3 ,  , wn be the uncertainties in the

independent variables. If the uncertainties in the dependant variables are all given with the
same chances, the uncertainty in the result having these chances is given as
1/ 2
 R  2  R 2
  R 
2
 R  
2

wR   w1    w2    w3   ...   wn  
 x1   x 2   3 x 
 nx  

This method is called Method of Kline and McClintock.

3.3.3 Uncertainty of common figures functions

Error Analysis 6

a- Uncertainty for product functions

In many case the result function takes the form of a product of the respective primary
variables raised to exponents and expressed as
R  x1a1 x2a2 x3a3 ...xnan
when the partial differentiations are performed, we obtain
R
 x1a1 x2a2 x3a3 ...(ai xiai 1 )...xnan
xi

Dividing by R this equation becomes

1 R ai

R xi xi

inserting this relation in wR gives

2 1/ 2
wR   ai wi 
    
R   xi  
 
Example 3.3
Consider the same data as in Example 3.2. Calculate the uncertainty to which the power is
calculated using the Method of Kline and McClintock.

Solution:
P  VI
1/ 2
wP  wI   wV  
2 2
     
P  I   V  

1/ 2
 0.2  2  2  2 
       0.0004  0.0004  0.028  2.8%
 10   100  

wP
It should be noted here that the result is directly a percent, since we are calculating . If we
P
want to determine the absolute uncertainty we only have to write that
wP  0.028 P  0.028 1000  28 Watts

b- Uncertainty for additive functions

When the result function has an additive form, R will be expressed as
R  a1 x1  a2 x2  ...  an x   ai xi
and the partial derivatives for use are then
Chapter 3: Measurement and error 7

R
 ai
xi
The uncertainty in the result may then be expressed as
1/ 2
  2 
  R  2  
wR    w 
 xi 
i
 
  

  (a w ) 
i i
2 1/ 2

Example 3.4: Selection of measurement method

A resistor has a nominal stated value of 10  1% . A voltage is impressed on the resistor,
and the power dissipation is to be calculated in two different ways:
V2
(1) from P 
R
(2) from P  VI
In (1) only a voltage measurement will be made, while both current and voltage will be
measured in (2). Calculate the uncertainty in the power determination in each case when the
measured values of V and I are
V  100 V  1%
I  10 A  1%
Solution:
For the first case we have
P V P V2
2  2
V R R R
and then
1/ 2
 2V  2 2  V 2  2 
wP    wV    2  wR 
 R   R  

V2
dividing by P  gives
R
1/ 2
wP   wV  w  
2 2
 4   R  
P   V   R  

with numerical values
wP
 4(0.01) 2  (0.01) 2  2.236%
P

For the second case

Error Analysis 8

P P
I V
V I
and after some algebraic manipulation we get
1/ 2
wP  wV   wI  
2 2

     
P  V   I  
 
Inserting the numerical values of uncertainties yields
wP
 (0.01) 2  (0.01) 2  1.414%
P
The second method of power determination provides considerably less uncertainty than the
first method, even though the primary uncertainties in each quantity are the same. In this
example the utility of the uncertainty analysis is that it affords the experimenter a basis for
selection of a measurement method to produce a result with less uncertainty.

Example 3.5: way to reduce uncertainties

A certain flowmeter shown in Fig. 3.2 below is used to measure the flow rate of air. The
relation describing the flow rate is
1/ 2
 2 gp1 
  CA  ( p1  p 2 )
 RT1 

Figure 3.2 Flowmeter

where
C = empirical coefficient
A = flow area
p1 and p2 = pressure in (1) and (2) respectively
T1= Temperature
R= constant
Calculate the percent uncertainty in the air flow rate for the following conditions:
C  0.92  0.005
p1  25  0.5 pa

T1  70  2 C
Chapter 3: Measurement and error 9

p  p1  p2  1.4  0.005 pa measured directly

A  1  0.001cm 2
Solution: In this example the flow rates a function of several variables, each subject to an
uncertainty
  f (C, A, p1 , p, T1 )
thus, we form the derivatives
1/ 2
  2 gp1 
 A p 
C  RT1 
1/ 2
  2 gp1 
 C  p 
A  RT1 
1/ 2
  2g 
 0.5CA p  p11 / 2
p1  RT1 
1/ 2
  2 gp1 
 0.5CA  p1 / 2
p  RT1 
  2 gp1 
 0.5CA p T13 / 2
T1  R 

The uncertainty in the flow rate may be calculated by assembling these derivatives as

 w 2 2 2 2 2 1/ 2
w   w   wP  w  w 
  C    A   14  1   1  p   1  T1  
  C   A   4  p  4  T  
  p1     1  
Inserting numerical values of uncertainties yields
1/ 2
w  0.005  2  0.001 2  0.5 
2 2
  1.4 2 1 2
    14    1 4  
  0.92   1   25 
4
 70  


 29.5  106  1.0  106  1.0  104  3.19  106  2.0  104 
1/ 2

= 1.82 %

Comment: The main contribution to uncertainty in  is the T1 and p1 measurements. Thus to

improve the overall situation, the accuracy of this measurements should be improved first.
Error Analysis 10

3.4 Problems (set 1)

Problem 1
A certain resistor draws 110.2 V and 5.3 A. The uncertainties in the measurements are
 0.2 V and  0.06 A, respectively. Calculate the power dissipated in the resistor and the
uncertainty in the power.

Problem 2
The resistance of a certain size of copper wire is given as
R  R0 [1   (T  20)]

where R0  6   0.3% at 20 C,   0.004C 1  1% is the temperature coefficient of

resistance, and the temperature of the wire is T  30  1C . Calculate the resistance of the
wire and its uncertainty.

Problem 3
The steady state pressure-flow-current characteristics for an electrohydraulic servovalve can
be approximated by the following equation
Ps  Pm
Qm  K v I


Kv: Size factor (constant)

I: Current input to the servovalve
Ps=Supply pressure
Pm: Pressure drop across the servomotor
Qm: Control flow rate
: mass density of fluid
Determine the expression of the uncertainty on Qm, wQm , in terms of wI, wPs, wPm and w.

Problem 4
When two parallel plates of area A are placed at a distance d from each other, the capacitance
C according to Faraday’s law is defined by
 r0 A
C
d

1- Derive the dimension of  0

2- The actual nominal value (at 25 ºC) of C is 1 F. Suppose that the variation of
plate’s lengths with temperature is 0.02%/ºC. What would be the value of C at
50ºC.?
Chapter 3: Measurement and error 11

3.5 Statistical analysis of measurement systems

Suppose that we have a large box containing thousands of similar resistances. To get an idea
of the value of the resistances, we might measure two dozen for example. The resulting
resistance values form a data set, which we use to infer something about the value of the
resistances in the box as a whole. Such as average value and value variation. But how close
are the values from our data to the actual average value and variation of all the resistances in
the box? It we took another data set, should we expect the values to be exactly the same?
Engineering measurements taken repeatedly under seemingly identical conditions
will normally show variations in measured values. Sources that contribute to these variations
include the following
Measurement system
 Resolution
 Repeatability
Measurement procedure and technique
 Repeatability
Measured variable
 Temporal variation
 Spatial variation

For a given set of measurements, we want to be able to quantify

1- a single representative value that best characterizes the average of the data set,
2- a representative value that provides a measure of the variation in the measured data
set, Furthermore we want to know how well this single average value represents the
true average value of the variable measured, this is done by establishing
3- an interval about the representative average value in which the true value is expected
to lie.
Statistics and probability provides to tools needed to arrive at these three estimates.

When a set of readings of an instrument is taken, the individual readings will vary somewhat
from each other, and the experimenter may be concerned with the mean of the reading. If
each reading is denoted by xi, and there are n readings, the arithmetic mean is given by
n
1
x
n
 xi
i 1

The deviation di for each reading is defined by

d i  xi  x
We may note that the average of the deviations of all the readings is zero since
Error Analysis 12

n n
1 1
di 
n
 d i  n  ( xi  x )
i 1 i 1

1
x (nx )  0
n
The average of the absolute values of the deviations is given by
n n
1 1
di 
n
 di 
n
 xi  x
i 1 i 1

Note that this quantity is not necessarily zero.

The standard deviation or root-mean-square deviation is defined by

1/ 2
1 n 
 
 n
 ( xi  x m ) 

2

i 1

and the square of the standard deviation  2 is called the variance.

In many circumstances the engineer will not be able to collect as many data as necessary to
describe the underlying population. Generally speaking, it is desired to have at least 20
measurements in order to obtain reliable estimates of standard deviation and general validity
of data.

For small sets of data the standard deviation is defined by

1/ 2
 1 n 
 
 n  1 i 1

( xi  x m ) 2 

Note that the factor n – 1 is used instead of n.

3.6 The Gaussian or normal error distribution

Suppose an experimental observation is made and some particular result is recorded. We
know (or would strongly suspect) that the observation has been subjected to many random
errors. These random errors may make the final reading either too large or too small,
depending on many circumstances which are unknown to us. Assuming that there are many
small errors that contribute to the final error and that each small error is of equal magnitude
and equally likely to be positive or negative, the Gaussian or normal error distribution may be
then derived. If the measurement is designated by x, the Gaussian distribution gives the
probability that the measurement will lie between x and x + dx and is defined by
1 2
/ 2 2
P( x)  e ( x  x )
 2
Chapter 3: Measurement and error 13

where x is the mean reading and  is the standard deviation. P(x) is sometimes called the
probability density. A plot of this equation is given in Fig. 3.3.

Figure 3.3 The Gaussian or normal error distribution

The standard deviation is a measure of the width of the distribution curve; the larger the value
of  , the flatter the curve and hence the larger the expected error of all measurements. The
equation of P(x) is normalized so that the total area under the curve is unity. Thus,

 P( x)dx  1.0

3.7 Probability analysis and random errors

We may now anticipate the next step in the analysis as one of trying to determine the
precision of a set of experimental measurements through an application of the normal error
distribution. One may ask: How do you know that the assumptions pertaining to deviation of
the normal error distribution apply to experimental data? The answer is that for sets of data
where a large number of measurements are taken, experiments indicate that the measurements
do indeed follow a distribution like that shown in Fig.3.3.
By inspection of the Gaussian distribution function, we see that the maximum probability
occurs at x  x and the value of this probability is
Error Analysis 14

1
P( x ) 
 2
It can be seen from this equation that smaller values of the standard deviation produce larger
values of the maximum probability, as would be expected in an intuitive sense. P(x ) is a
measure of precision of the data since it has larger values for smaller values of standard
deviation.
We next wish to examine the Gaussian distribution to determine the probability that certain
data points will fall within a specified deviation from the mean of all data points. The
probability that a measurement will fall within a certain range x1 of the mean reading is
x  x1 1 2
/ 2 2
P x  x 
1 2
e ( x  x ) dx

making the change of variable

xx


This equation becomes
1 1  2 / 2
P
2 1
e d

where
x1
1  .

Example 3.6
Derive the expression of probabilities that a measurement will fall within 1, 2 and 3 standard
deviations of the mean value.

Solution
1 = 1, 2 and 3
We can observe that
1  2 / 2 1  2 / 2
 1
e d  2 0 e d

so that
1 1 0.5 2
P(1 ) 
2 0 e d

1 2 0.5 2
P(2 ) 
2 0 e d

1 3 0.5 2
P(3 ) 
2 0 e d
Chapter 3: Measurement and error 15

1 1  2 / 2
The following table gives the values of the integral I =
2 0 e d for different

values of the argument 1 .

 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

I 0 0.03983 0.07926 0.11791 0.15554 0.19146 0.22575 0.25084 0.28814 0.31594

 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9

I 0.34134 0.36433 0.38493 0.4032 0.41924 0.43319 0.4452 0.45543 0.46407 0.47128

 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9

I 0.47725 0.48214 0.4861 0.48928 0.4918 0.49379 0.49534 0.49653 0.49744 0.49813

 3 3.5 4 4.5 5

I 0.49865 0.49976 0.49997 0.499996 0.4999997

Using this table, we can calculate

P(1 )  2  0.34134 0.6827

P(2 )  2  0.47725 0.9545
P(3 )  2  0.49865 0.9973

Figure 3.4: Probability distribution

Error Analysis 16

Confidence interval and level of significance

The confidence interval expresses the probability that the mean value will lie within a certain
number of  values and is symbolized by z. Thus
x  x  z (% confidence level)
The confidence level in % could be expresses as in table 3.1. We that expect that the mean
value of a measurement set will lie within  2.57 with less than 1 % error (confidence level
of 99 %). The level of significance is 1 minus the confidence level. Thus for z = 2.57 the
level of significance is 1 %.

Table 3.1
Confidence Interval Confidence Level (%) Level of Significance
(%)
3.30 99.9 0.1
3.0 99.7 0.3
2.57 99.0 1.0
2.0 95.4 4.6
1.96 95.0 5.0
1.65 90.0 10.0
1.0 68.3 31.7

For small data samples z should be replaced by

z

n
where n is the number of samples.

Example 3.7
A certain length is measured with a device which has a known precision of  0.5 mm when a
large number of measurements is taken. How many measurements are necessary to establish
the mean length x with a 5% level of significance such that
x  x  0.2 mm
Solution:
For a large number of measurements the 5 % level of significance is obtained at z = 1.96 and
for the population here
Chapter 3: Measurement and error 17

z 1.96  0.5 mm
  0.2 mm =
n n
which yields to
n = 24.01

So, for 25 measurements or more we could state with a confidence level of 95 % that the
population mean value will be within  0.2 mm of the sample mean value.

3.8 Calibration
A very important issue in instrumentation is the calibration of instruments and measurement
systems. The relationship between the value of the input to the measurement system
and the system’s indicated output value is established during a calibration of the
measurement system. A calibration is the act of applying a known value of input to a
measurement system for the purpose of observing the system output. The known
value used for the calibration is called the standard. There two types of calibration;
static and dynamic calibrations.

3.8.1 Static calibration

The most common type of calibration is the static calibration. In this procedure, a known
value is input to the system under calibration and the system output is recorded. In static
calibration, only the magnitudes of the known input and the measured output are important.
By application of a range of known values for the input and observation of the system output,
a direct calibration curve can be developed for the measurement system. On such a curve the
input, x, is plotted on the abscissa against the measured output, y, on the ordinate such as
indicated in Figure 3.4.
16

14 + Measured values

12 ___ curve fit, y=f(x)

10
output values

0
0 0.5 1 1.5 2 2.5 3 3.5 4
input values

Figure 3.4. Static calibration curve

Error Analysis 18

The static calibration curve describes the static input-output relationship for a measurement
system and forms the logic by which the indicated output can be interpreted during an actual
measurement. For example, the calibration curve is the basis for fixing the output display
scale on a measurement system. Alternatively, a calibration curve can be used as part of
developing a functional relationship, an equation known as correlation, between input and
output. A correlation will have the form y  f (x) and is determined by applying physical
reasoning and curve fitting techniques to the calibration curve. The calibration can then be
used is later measurements to ascertain the unknown input value based on the output value,
the value indicated by the measurement system.

3.8.2 Dynamic calibration

When the variables of interest are time dependant and time based information is required we
need dynamic information. In a broad sense, dynamic variables have time dependant
magnitudes. A dynamic calibration determines the relationship between an input of known
dynamic behavior and the measurement system output. Dynamic calibration is out of the
scope of this course

3.8.3 Static sensitivity

The slope of a static calibration curve yields the static sensitivity or the static gain of the
measurement system. The static sensitivity, K, at any particular static value input, say x1 , is
evaluated by
 dy 
K  K ( x1 )   
 dx  x  x1

where K is a function of x. The static sensitivity is a measure relating the change in the
indicated output associated with a given change in the static input. Since calibration curves
can be linear or nonlinear depending on the measurement system and on the variables being
measured, K may or may not be constant over a range of input values.

3.8.4 Range
The proper procedure for calibration is to apply known inputs ranging from the minimum
values for which the measurement system is to be based. These limits define the operating
range of the system. The input operating range is defined as extending from x min to x max .
This range defines the input span, expressed as
ri  xmax  xmin
Chapter 3: Measurement and error 19

Similarly, the output operating range is specified from y min to y max . The output span, or full
scale operating range (FSO), is expressed as
yi  y max  y min
3.8.5 Accuracy
The accuracy of a measurement system can be estimated during calibration. If the input value
of calibration is known exactly, then it can be called the true value. The accuracy of a
measurement system refers to its ability to indicate a true value exactly. Accuracy is related
to absolute error. Absolute error, e, is defined as the difference between the true value
applied to a measurement system and the indicated value of a system:
e = true value – indicated value
from which the percent accuracy is found by
 e 
A(%)  1    100
 true value 
 
By definition, accuracy can be determined only when the true value is known, such as during
calibration.

An alternative form of calibration curve is the deviation plots. Such a curve plots the
difference or deviation between a true or expected value y’ and the indicated value, y, versus
the indicated value, y, as shown in Fig. 3.5. Deviation curves are extremely important when
the difference between the true and the indicated value are too small to suggest possible
trends on direct calibration plots.

Figure 3.5 Calibration curve in form of a deviation plot

Error Analysis 20

3.8.6. Precision
The repeatability or precision of a measurement system refers to the ability of the system to
indicate a particular value upon repeated but independent applications of a specific value of
input. Note that a system that repeatedly indicates that same wrong value upon repeated
application of a particular input would be considered to be very precise regardless of its
accuracy, as shown in Fig. 3.6.

Figure 3.6 Precision

3.8.7 Sequential test (Hysteresis)

A sequential test applies a sequential variation in the input value over the desired input range.
This may be accomplished by increasing the input value (upscale direction) or by decreasing
the input value (downscale direction) over the full input range.
A sequential test is an effective diagnostic technique for identifying and quantifying a
Hysteresis error in a measurement system. Hysteresis error refers to the difference in the
value found between going upscale and downscale in a sequential test. The effect of
Hysteresis in a calibration curve is illustrated in Fig. 3.7. For a particular input value, the
Hysteresis error is found from the difference in the upscale and downscale output by
eh = (y)upscale – (y)downscale

Hysteresis is usually specified for a measurement system in terms of the maximum Hysteresis
error found in the calibration, %eh max , as a percentage of full scale output range:
Chapter 3: Measurement and error 21

eh max
%eh max   100
ro

Figure 3.7 Hysteresis error

3.8.8 Linearity error

Many instruments are designed to achieve a linear relation between an applied static input and
indicated output value. Such a linear static calibration curve would have the general form
y L ( x)  a0  a1 x

where the curve fit y L (x) provides a predicted output value based on a linear relation
between x and y. However, in real systems, truly linear behavior is only approximately
achieved. As a result, measurement device specifications usually provide a statement as to
the expected linearity of the static calibration curve for the devices. The relation between
y L (x) and measured value y(x) is a measure of the nonlinear behavior of a system:

e L ( x )  y ( x)  y L ( x)

where e L (x) is the linearity error. Such behavior is illustrated in Fig. 3.8.

Figure 3.8 Linearity error

Error Analysis 22

For a measurement system that essentially linear in behavior, the extent of possible
nonlinearity in a measurement device is often specified in terms of the maximum expected
linearity error for the calibration as a percentage of full-scale output range:
e L max
%e L max   100
r0

3.8.9 Instrument repeatability

Repeatability of measurement system is its ability to reproduce the same value upon repeated
but independent application of the same input. Repeatability, as shown in Fig. 3.9, is based
on the standard deviation of the statistical measure,  x a measure of the variation in the output
for a given input. The value claimed is usually in terms of the maximum expected error as a
percentage of full-scale output range
2 x
%(e R ) max   100
r0

Figure 3.9 Repeatability

3.8.10. Overall instrument error

An estimate of the overall instrument error is based on all know]n errors. This error is often
misleadingly referred to as the instrument accuracy in some instrument specifications. An
estimate is computed from the square root of the sum of the squares of all known errors. For
M known errors, the instrument error, ec, is estimated by

ec  e12  e22    eM
2

For example, for an instrument having known Hysteresis,, linearity, sensitivity and
repeatability errors, the instrument error is estimated by

ec  eh2  e L2  e K2  e R2
Chapter 3: Measurement and error 23

3.8.11 Resolution and zero-order uncertainty

We define the resolution of a measurement system as the smallest detectable change in
measured value as indicated by the measuring system. It is usually expressed in terms of the
smallest increment which can be measured or sensed. The higher the resolution of a display,
the smaller the increment it is able to measure. For example, a five-digit display which can
measure a quantity to 0.0001 units has a higher resolution than four-digit display measuring to
0.001 units.
Even when all errors are otherwise zero, the value of the measurand must be affected by the
ability to resolve the information provided by the instrument. We call this the zero-order
uncertainty of the instrument, u0. Basically e0, is an estimate of the expected uncertainty
caused by the data scatter that results when the instrument is read.
As an arbitrary rule, assign a numerical value to e0 of one half of the instrument resolution
with a probability of 95%:
e0   12 resolution

3.8.12 Design stage uncertainty

The design-stage uncertainty ed, for an instrument is approximated by combining the overall
instrument uncertainty and the zero-order uncertainty as:

ed  e02  ec2

Example 3.8
Consider the force measuring instrument described by the catalog data that follows
Resolution: 0.25 N
Range: 0-100 N
Linearity: within 0.20 N over range
Repeatability: within 0.30 N over range
Provide an estimate of the uncertainty to this instrument and the design stage instrument
Solution:
e L  0.20 N eR=0.30 N
then

ec   0.2 2  0.32  0.36 N

The instrument resolution is given as 0.25 N, from which
e0  0.125 N
the design-stage error of this instrument is then

ed   e02  ec2   0.362  0.1252  0.38 N

Error Analysis 24

3.9 Problems (set 2)

Problem 1
For the calibration data of Table 3.2,
1. Specify the range of calibration
2. Determine the static sensitivity of the system at:
a- X = 5;
b- X = 10;
c- X = 20;
3. For which input values the system is more sensitive?

Table 3.2: Calibration data

X(cm) Y(V) X(cm) Y(V)

0.5 0.4 10.0 15.8
1.0 1.0 20.0 36.4
2.0 2.3 50.0 110.1
5.0 6.9 100.0 253.2

Problem 2
Consider the voltmeter calibration data in Table 3.3, plot the data by using a suitable scale.
Specify the percent maximum Hysteresis based on full scale range.

Table 3.3: Voltmeter calibration data

Increasing Input (mV) Decreasing input (mV)

X Y X Y
0.0 0.1 5.0 5.0
1.0 1.1 4.0 4.2
2.0 2.1 3.0 3.2
3.0 3.0 2.0 2.2
4.0 4.1 1.0 1.2
5.0 5.0 0.0 0.2
Chapter 3: Measurement and error 25

Problem 3
For the calibration data of table 3.2, plot the results by using linear and log-log scales.
Discuss the apparent advantages of each representation.

Problem 4
Plot the following data on an appropriate set of axes. Estimate the static sensitivity K at each
X.
Y(V) X(cm)
2.9 0.5
3.5 1.0
4.7 2.0
9.0 5.0

Problem 5

The following data have the form y  axb . Plot the data in an appropriate format to estimate
the coefficients a and b. Estimate the static sensitivity K at each value of X. How is K
affected by X?

Y(V) X(cm)
0.14 0.5
2.51 2.0
15.30 5.0
63.71 10.0

Problem 6
A force measurement system has the following specifications

Range: 0 – 1000 N
Linearity error: 0.10 % FSO
Hysteresis error: 0.10 % FSO
Repeatability error: 0.15 % FSO

Estimate the overall instrument error for this system based on available information
Error Analysis 26

Problem 7
The transducer specified in Table 3.4 is chosen to measure a nominal pressure of 500 cm
H2O. The ambient temperature is expected to vary between 18°C and 25°C during tests.
Estimate the magnitude of each elemental error affecting the measured pressure.

Table 3.4 Manufacturer’s specifications: Pressure transducer

Operation
Input range 0-1000 cm H20
Excitation ±15V DC
Output range 0–5V
Performance
Linearity error ± 0.5 % FSO
Hysteresis error Less than ± 0.15 % FSO
Sensitivity error ± 0.25% of reading
Thermal sensitivity error ±0.02 %/ °C of reading
Temperature range 0 – 50 °C

Problem 8 (Old exam question)

A Temperature transducer has the following specifications
Input Range: -30 to 100 C
Output range: 0 to 10 volts
Resolution: 0.1 V
Linearity error: 2 % FSO
Hysteresis error: 1 % FSO
Repeatability error: 1 % FSO
a- Estimate the design stage uncertainty for this transducer based on available information.
b- To calibrate this transducer, the following experimental results were obtained

T(C) -30 -20 -10 0 10 20 30 40 50 60 70 80 90 100

V 1 1.3 1.5 2.1 2.5 3.1 3.6 4.1 4.6 5 5.4 5.9 6.6 8

b-1. Draw V(T)

b-2 Estimate the static sensitivity of the transducer.
b-3 Does the Linearity error provided by the manufacturer make sense? Why?
b-4 Recalculate the design stage uncertainty and compare it to a.
b-4 Propose a solution to avoid this large error.
Chapter 3: Measurement and error 27

3.10 Graphical Analysis and curve fitting

Engineers are well known for their ability to plot many curves of experimental data and to
extract all sorts of significant facts from these curves. The better one understands the
physical phenomena involved in a certain experiment, the better one is able to extract a wide
variety of information from graphical displays of experimental data.
Assuming that the engineer knows what is to be examined with graphical presentations; the
plots may be carefully prepared and checked against appropriate theories. Frequently, a
correlation of the experimental data is desired in terms of an analytical expression between
variables that were measured in the experiment. When, for instance, the data may be
approximated by a straight line, the analytical relation is easy to obtain; but when any other
functional variation is present, difficulties are usually encountered. The curve could be
polynomial, exponential, or complicated logarithmic function. If the experimenter has a good
idea of the type of function that will be represent the data, then the type of plot is easily
selected.

3.10.1 Alternative displays and correlation trendlines for an experimental data.

For simplicity we may treat this section by tacking an example and by using the Software
Excel for the interpolation and the display of graphical result.

Example 3.9 : The data for a series of experiments are shown in the table below

Table 3.5 Experimental data

Input (X) Output (Y)
1 2.05
2 1.67
3 1.37
4 1.12
5 0.92
6 0.75
7 0.61
8 0.50
9 0.41
10 0.34
13 0.19
17 0.08
20 0.05
Error Analysis 28

a- The experimental data are displayed with data markers, but without connecting line
segments

Y vs X
2.5

1.5
Y
1

0.5

0
0 5 10 X 15 20 25

b- The experimental data are plotted with data markers along with connection line
segments.

Y vs X
2.5

1.5
Y

0.5

0
0 5 10 X 15 20 25

c- A smooth curve is plotted through the points with data markers omitted

Y vs X
2.5
2
1.5
Y

1
0.5
0
0 5 10 15 20 25
X
Chapter 3: Measurement and error 29

d- The data points are displays without connecting line segments along with a
least square linear fit to the data. The linear relation obviously does not work, and
is evidenced by a low value of r2.

y = -0.0956x + 1.5376
Y vs X
R2 = 0.8014
2.5
2
1.5
1
Y

0.5
0
-0.5 0 10 20 30
X

e- In this case the plot shows a second order polynomial (quadratic) fit of the data on
linear coordinates. The results are fairly good, except for larger values of x.

Y vs X
y = 0.0077x2 - 0.2499x + 2.0348
2.5
R2 = 0.9905
2

1.5
Y

0.5

0
0 5 10 15 20 25
X

f- In plots below a cubic polynomial fit is performed. Very good results of

interpolation are obtained. Note the almost perfect value of r2.

Y vs X
3 2
y = -0.0002x + 0.0148x - 0.3014x + 2.1086
2.5 2
R = 0.9953
2

1.5
Y

0.5

0
0 5 10 15 20 25
X
Error Analysis 30

g- An exponential relation is fitted to the data with a least square analysis. An

almost perfect fit is obtained with an r2 almost of 1.

Y vs X y = 2.3512e-0.194x
R2 = 0.9978
2.5

1.5
Y

0.5

0
0 5 10 X 15 20 25

h- The plot of Fig. 3. displays the exponential trend of the experimental data but
plotted on a semi logarithmic coordinates.

Y vs X y = 2.3512e-0.194x
R2 = 0.9978
10

1
0 5 10 15 20 25
Y

0.1

0.01
X

Comments
- In all of these plots the software Excel was used to make the trends of the
experimental data as well as the determination of the fitting function and the
factor r2.
- The factor r2 is called the coefficient of correlation. For a perfect fit r2 = 1. A
high value of r2 (approaching the unity) will be an indicative of a very good
correlation.
- The fitting technique used by Excel is based on the least square method such a
method uses the minimization of the square of the error between the data and the
curve fitting used. The theory related to the least square method (LSQ) is beyond
the scope of this course).
Chapter 3: Measurement and error 31

3.11. Problems (set 3)

Problem 1
For the experimental data shown below
x 0 1 2 3 4 5 6 7 8 9 10

y 1.9 3.6 5.5 9.8 14.5 19 25 35 40 53 60

1. Use Excel to analyze a linear and quadratic approximation fitting of the data.
Display the fitting function and r2.
2. Plot the data in such a manner that the graphical display will be a straight line if a
quadratic function relation best fit the data.

Problem 2
The data of a certain experimental test are tabulated below. Examine the data with multiple
graphs using Excel to arrive at a suitable correlation for the data y = f(x). Display the fitting

function and r 2 .
x.103 20 25 35 45 47 50 62 65 70 75 80 90
y 60 79 90 135 110 130 160 150 180 190 191 200

x.103 100 105 110 120 135 140 145 150

y 210 250 240 280 300 290 330 340

Problem 3
A meter behaves according to the following table
x.104 4 5 6 8 10 15 20 30 100
y 0.957 0.962 0.966 0.973 0.977 0.982 0.984 0.984 0.984

Using Excel plot y vs. log x and obtain an appropriate relationship between y and x.

Problem 4
The following calibration data are available for a certain temperature measurement device.
Using Excel plot y vs. x and obtain second-, third-, and fourth order polynomial fits the data.
x (C) -150 -100 -50 -25 0 25 50 75

y (V) -4.648 -3.379 -1.189 -0.94 0 0.993 2.036 3.132

x (C) 100 150 200 300 400

y (V) 4.279 6.704 9.288 14.862 20.872

Error Analysis 32

Case Study: calibration exercise

Table 3.5 shows the results obtained when a pressure transducer was calibrated. The data is
shown graphically in Figure 3.10.
Table 3.6
kPa mV
Force transducer calibration data
0 0
1200
20 48
1000
40 97
800
60 145
mV

600 mV
400 80 193
200 100 241
0
120 290
0 200 400 600 800
kPa 140 338
160 386
Figure 3.10: Force transducer calibration data
180 435
200 483
220 530
240 578
260 625
280 672
300 719
320 767
340 814
360 861
380 909
400 956
420 990
440 1023
460 1052
480 1071
500 1090
520 1099
540 1099
560 1099
580 1099
600 1099
Chapter 3: Measurement and error 33

Q.1 Estimate the range over which the transducer can be considered to have a linear response

..……………………….

Linear regression is performed over the range 0-400kPa. The regression coefficients are

Sensitivity, S: 2.390mV/kPa
Intercept b: 2.325mV

Q.2 Use the calibration data to find the pressure when the output voltage has the values
indicated below. For the non-shaded boxes, give the value of pressure which was obtained
during calibration and calculate the error in the predicted load.

Output voltage Predicted pressure Pressure during Error

calibration
58
193
220
483
625
743
827
956

The residual values for the regression are shown in Figure 3.11.
Error Analysis 34

3
2

Residuals (mV)
1
0
-1 0 200 400 600

-2
-3
Load (kPa)

Figure 3.11: Residual values for regression analysis

Q.3 Give the values of the largest positive and negative residuals ……………………

Q.4 Give the corresponding non-linearity errors ………………………………………

The non-linearity error can be reduced by using the transducer over a shorter range .

Q.5 What do you think would be a suitable shorter range over which the non-linearity error
would be

significantly reduced? ……………………………………

Linear regression analysis is done using the range 0-200kPa. The co-efficients are now:

Sensitvity S: 2.415mV/kPa
Intercept: b: -.0217mV

Q.6 Use the calibration data to find the pressure for the output voltages indicated below. For
the non-shaded boxes, give the value of pressure which was obtained during calibration and
calculate the error in the predicted load.
Chapter 3: Measurement and error 35

Output voltage Predicted pressure Pressure during Error

calibration
67
145
204
275
386
483

The residuals are shown in Figure 3.12.

X Variable 1 Residual Plot

0.2
Residuals

0
0 50 100 150 200 250
-0.2
X Variable 1

Figure 3.12: Residual values for regression 0-200kPa.

Q.7 Give the largest positive and negative non-linearity errors for this regression analysis.

………………………
Error Analysis 36

Case Study 2
A force measurement system is to be set up using a force transducer and a digital display
whose parameters are given below. Find the range and resolution of the system and calculate
the error in the measured force when the input force is i) 200kN ii) 500kN and iii) 1000kN
assuming that the temperature varies between 15 and 25C.

Transducer

Range 10-1000kN
Sensitivity at 200C 0.135mV/kN
Linearity error 2kN
Repeatability error 0.2% reading
Temperature error 0.2kN/0C

Display

Output 4 digits
Minimum input for max reading 1mV
Maximum input for max reading 10V
Accuracy (0.3% reading + 1lsd)
Stability after 5 minutes warm up 1lsd

Range

The range of the transducer is 10-1000kN.

Ideally, the maximum value, 1000kN, should be displayed as 1000

To see if this is possible, we need to find the maximum output voltage from the transducer.

This is given by 0.135 x 1000mV = 0.135V

The minimum voltage required for a reading of 9999 is 1mV, therefore the minimum voltage
required for a reading of 1000 is

1  1000
 0.1mV
9999
Chapter 3: Measurement and error 37

The maximum voltage required for a reading of 9999 is

10  1000
 1V
9999

The output voltage for the maximum force of 1000kN of .135V lies between these two values,
therefore it is possible to set the display to read 1000 as required.

Resolution

The resolution will then be 1kN.

Error
Transducer errors are (a) linearity (b) repeatability and (c) temperature.

Transducer 200 500 1000

Linearity 2 2 2
Repeatibility 0.002 * reading 0.4 1 2
Temperature 0.2*temp change (5) 1 1 1

Total error Sqrt (sum of squares) 2.3 2.4 3.0

Display
Accuracy 1lsd 1 1 1
0.003*reading 0.6 1.5 3
Sum of these 1.6 2.5 4

Stability - 1lsd 1 1 1

Total error Sum of squares 1.9 2.7 4.1

Combined error Sum of squares overall 3.0 3.6 5.1

As a percentage 1.5 0.7 0.5

Display errors are Accuracy and Stability

Quote as

i) 3kN, 1.5%
ii) 4kN, 0.7%
iii) 5kN, 0.5%