Materi Sesi i General

Fundamentals of uncertainty estimation on measurement, calibration and testing Alexandre ALLARD LNE, France

Laboratoire national de métrologie et d’essais

Outline • Introduction •

The concept of measurement uncertainty

•

Reference documents

•

Basic statistical concepts

•

Step 1 : Analysis of the measurement process

• GUM methodology • Supplement 1 to the GUM (Monte Carlo) • Case studies •

Linear example

•

Mass calibration examples

• Conclusion •

Scope of the GUM/GUM-S1

•

Bayesian methods

•

General conclusion 20/05/2015

Workshop measurement uncertainty Jakarta

2

The concept of uncertainty Uncertainty arises with any kind of measurements

Measurement result (VIM 2012) : set of quantity values being attributed to a measurand together with any other available relevant information Note 2 : A measurement result is generally expressed as a single measured quantity value and a measurement uncertainty. 20/05/2015


3

Why Must an Uncertainty in a Measurement Result Be Indicated ? GUM Introduction : When reporting the result of a measurement of a physical quantity, it is obligatory that some quantitative indication of the quality of the result be given so that those who use it can assess evaluate its reliability. Without such an indication, measurement results cannot be compared, either among themselves or with reference values given in a specification or standard […]

20/05/2015


4

Why Must an Uncertainty in a Measurement Result Be Indicated ? GUM 3.3.1 : • The uncertainty of the result of a measurement reflects the lack of exact knowledge of the value of the measurand. • The result of a measurement after correction for recognized systematic effects is still only an estimate of the value of the measurand because of the uncertainty arising from random effects and from imperfect correction of the result for systematic effects.

20/05/2015


5

Usage : Comparison between Two Measurement Results Values

Laboratory B Laboratory A

?

How to compare results from laboratories A and B ? 20/05/2015


6

Usage : Comparison between Two Measurement Results Values

Laboratory B Laboratory A

The statement of a measurement uncertainty enables to compare results from laboratories A and B ? 20/05/2015


7

Comparison between a Measurement Result and a Specification • Manufacturing processes are never perfect and lead to a dispersion of product specificities • Define a tolerance area in which the product is considered to be in compliance with its specifications

Max. Tolerance LImit

Min. Tolerance Limit Specification Area 20/05/2015


8

Comparison between a Measurement Result and a Specification

Measurement Result and Uncertainty

Max. Tolerance Limit

Min. Tolerance Limit Specification Area

20/05/2015


9

Comparison between a Measurement Result and a Specification Neither conformity or non-conformity can be assessed Measurement Result and Uncertainty


Min. Tolerance Limit Specification Area

20/05/2015


10

Comparison between a Measurement Result and a Specification Guard band

Min. Tolerance Limit

Guard band

Acceptation Area


Specification Area

The client benefits from the doubt 20/05/2015


11

Comparison between a Measurement Result and a Specification Guard band

Guard band

Acceptation Area Min. Tolerance Limit Specification Area


The producer benefits from the doubt 20/05/2015


12

Comparison between a Measurement Result and a Specification « shared risk »

« shared risk »

Acceptation Area Min. Tolerance Limit


Specification Area

Client and producer « share the risk » (GUM-Conformity) 20/05/2015


13

Documentation - JCGM

Joint Committee for Guides in Metrology (JCGM)

WG1

GUM + Supplements

20/05/2015

WG2

International Vocabulary of Metrology (VIM)


14

Documentation - VIM • VIM (JCGM 200:2012) : International Vocabulary of Metrology – Basic and general concepts and associated terms, 2012. •

Bilingual English/French document

Harmonization of the vocabulary usd in metrology in order to have everyone concerned with metrology all over the world use the same terms for the same concepts.

20/05/2015


15

Documentation – GUM + Supplements • GUM (JCGM 100:2008) : Guide to the expression of uncertainty in measurement, 2008. •

1st version published in 1994.

•

Minor modifications until the last version in 2008

• GUM-S1 (JCGM 101:2008) : Supplement 1 to the Guide to the expression of uncertainty in measurement – Propagation of distributions using a Monte-Carlo method, 2008. •

Alternative method when the conditions for a valid application of the GUM are not fulfilled

•

Requires a software

20/05/2015


16

Documentation – GUM + Supplements

• GUM-S2 (JCGM 102:2012) : Supplement 2 to the Guide to the expression of uncertainty in measurement – Extension to any number of output quantities, 2012. •

Deals with multiple measurands : common applications in chemistry, electricity, dimensional measurements, …

•

Extension of both GUM and GUM-S1 methodologies

• GUM-Conformity (JCGM 106:2012) : The role of measurement uncertainty in conformity assessment , 2012.

20/05/2015


17

Terminology • Measurand (VIM 2012, 2.3) : « quantity intended to be measured » ● The measurand has a unique and unknowable true value • Uncertainty (VIM 2012, 2.26) : « non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used » ● Standard uncertainty (VIM 2012, 2.30) : « Measurement uncertainty

●

expressed as a standard deviation ». Denoted as u or uc. Expanded uncertainty (VIM 2012, 2.31) : « product of a combined standard measurement uncertainty and a factor larger than the number one ». Denoted as U.

• Coverage interval (VIM 2012, 2.36) : « interval containing the set of true quantity values of a measurand with a stated probability, based on the information available » 20/05/2015


18

Terminology : Measurement error • Measurement error (VIM 2012, 2.16) : « measured quantity value minus a reference quantity value » •

The result of a measurement (after correction) can unknowably be very close to the value of the measurand (and hence have a negligible error) even though it may have a large uncertainty. Thus the uncertainty of the result of a measurement should not be confused with the remaining unknown error.

20/05/2015


19

Basic statistical concepts • ISO 3534-1 : Statistics – Vocabulary and symbols – General statistical terms and terms used in probability • Mean : Parameter that characterizes the location of a series of measurements •

Consider n repeated measurements

=

1

=

,⋯,

+ ⋯+

• Standard deviation : Parameter that characterizes the dispersion of a series of measurements = 20/05/2015

∑

− −1 Workshop measurement uncertainty Jakarta

20


• Variance : Parameter that characterizes the dispersion of a series of measurements. Equals the squared standard deviation =

=

∑

− −1

• Interesting mathematical properties •

Central in the Law of Propagation of Uncertainty (LPU – GUM)

20/05/2015


21

Basic statistical concepts • Quantile (p-quantile) qp : value of x equal to the infimum of all x such that ≤ ≥ •

Useful for the determination of a coverage interval associated with a Monte Carlo simulation

•

q0.50 is the median : half possible values of X are below the median and the other half are over the median.

2.5%

q0.025 20/05/2015


22

Basic statistical concepts – Estimation • Statistics are used when we don’t know something : in general, a sample is used to estimate properties related to a larger population. •

Surveys, conformity assessment, Risk analysis, …

• In metrology, we don’t know the value of the measurand : measurements are performed in order to obtain a sample. The value of the measurand is then estimated using the sample (the measured values). • Estimation (ISO 3534-1, 1.36) : procedure that obtains a statistical representation of a population from a random sample drawn from this population 20/05/2015


23

Basic statistical concepts – Estimation

Population ,⋯,

Data

=

Mean

Standard deviation

=

20/05/2015

1

Sample ,⋯,

1

̅= −

=

1

1 −1


− ̅

24

Basic statistical concepts – Probability distribution CONTINUOUS RANDOM VARIABLES g ( x ) is the probability density function for X = x probability de nsity function - continous v ariable

g(x)

-2

0

2

4

6

8

x +∞

∫

g ( x ) dx = 1

−∞

{x ; g(x)} the probability distribution of the random variable X 20/05/2015


25

Uncertainty of Measurement Evaluation : 4 steps

Step 3: Propagation of Uncertainty (LPU, MCM)

Step 2: Quantification of the sources of uncertainty of the xi Standard Uncertainty or distribution

Step 1: Analysis of the Measurement Process Step 4: Uncertain Input Quantities : xi

Mathematical Model of the Measurement Process

Measurand Y = f(Xi)

Final Expression of the Result y±U

f(X1,..,Xn)

Step 3’: Sensitivity Analysis, Prioritization

Feedback 20/05/2015


26

Step 1 : Analysis of the measurement process • Detailed definition of the measurand Y • List of potential uncertainty sources • Selection of the most important uncertainty sources , ! = 1, … , • Creation of the measurement function # : $=# 20/05/2015

,…, Workshop measurement uncertainty Jakarta

27

Step 1 : The Measurand •

“quantity intended to be measured” (VIM 2.3)

•

[…] The specification of the measurand requires knowledge of the kind of quantity, the description of the phenomenon, the body or the substance carrying the quantity […]

•

The definition of the measurand may require indications about quantities such as temperature, pressure, etc. measurement conditions reference temperature 20 °C the principle of measurement in physics etc. 20/05/2015


28

Step 1 : Examples of Measurand Definitions

1)1) Distance Distance between centre and upper side entre the le centre dethe la face of the gauge and joint plan, 20°Csur and at a supérieure de the la cale et leatplan lequel elle est adhérée, à 20 °C et vertical position. en position verticale.

2)2) Distance Distance between centres of the sides of entre the lestwo deux centres des facesatde la cale, à 20 is °C, lahorizontal the gauge, 20°C. The gauge at a cale étant en position horizontale. position. 3) Distance entre deux plans 3) parallèles, Distance between plans, at 20°C. à 20 two °C,parallel la cale étant en horizontale. Theposition gauge is at a horizontal position.

20/05/2015


29

Step 1 : Measurement error • The errors are due to: • The instrumentation (standards, instruments, environment ...) • The measured object (measurand)

True value (unknown) ̅ systematic

!

random

error Measurement error (VIM 2012, 2.16) : « measured quantity value minus a reference quantity value » Systematic measurement error (VIM 2012, 2.17) : « component of measurement error that in replicate measurements remains constant or varies in a predictable manner » Random mesurement error (VIM 2012, 2.19) : « component of measurement error that in replicate measurements varies in an unpredictable manner » 20/05/2015


30

Step 1 : How can errors be reduced ?

• Random errors are usually reduced by increasing the number of independent observations (replicate measurements) and by taking into account the mean of the values • Systematic errors are reduced by applying corrections • These two rules are the basis of the metrologist’s endeavour to give the best possible measurand evaluation

20/05/2015


31

Step 1 : The Measurement Process

• Do not focus your attention solely on the instrument but observe the process that leads to the measurement result instead • The uncertainty characterizes the result, not the instrument • The following impact the result : • Operators • Instruments, calibrators • The measurement method and operating mode • The measurement conditions (temperature, pressure, etc.) • The measured object or material

20/05/2015


32

Step 1 : Why Must the Measurement Process be Analyzed ?

• Identification of the factors which influence the measurement result (causes of error) to provide a list as exhaustive as possible. •

So as to reduce their influence by: • Using corrections, • Repeating the measurements and calculating the arithmetical mean out of the series of observations

•

To determine the measurement model associated to the measurement process

20/05/2015


33

Step 1 : Full analysis of the measurement process • The objective of a measurement is to determ ine the measurand value […] • As a consequence, a measurement begins with an appropriate definition of the measurand an appropriate definition of the measurement method

● generic description of a logical organization of operations used in a measurement

and an appropriate definition of the measurement procedure

● detailed description of a measurement in according to one or more measurement principles and to a given measurement method, based on a measurement model and including any calculation destined to obtain a measurement result • Measurement principle : Phenomenon serving as a basis of a measurement • Measurement model : Mathematical relation among all quantities known to be involved in a measurement 20/05/2015


34

Step 1 : Measurement Modelling •

In many cases, a measurand Y is not measured directly but is determined from N other quantities X1, X2, … XN through a functional relationship f : $=#

,…,

•

The input quantities X1, … XN represent all the usable data for the calculation of the result.

•

The input quantities X1, … XN – which the output quantity depends on – can also be taken as measurands as well as depend on other quantities, including, in the case of systematic effects, the corrections.

•

The f function does not merely express a physical law, but the measurement process. More specifically, it must contain all the quantities that contribute significantly to the final resulting uncertainty.

20/05/2015


35

Step 1 : Mathematical Model of an Atmospheric Pressure Measurement with a Barometer

0 = % × ℎ 1 + ( ) − 20 × cos / × 1 + 1 ) − 20 corrective term barometer verticality corrective term

gravity acceleration

dilatation

density of mercury at measurement temperature

Reading of the rule

20/05/2015


36

Summary Step 1 : Analysis of the measurement process Define the measurand Y Identify the input quantities Xi,…,XN Obtain the measurement function f : $=#

,…,

Main step for the evaluation of measurement uncertainty

20/05/2015


37



•

Reference documents

•


•



Linear example

•


• Conclusion •


•

Bayesian methods

•



38

Measurement uncertainty evaluation

Measurement Process analysis

Reproducibility Tests

no

?

yes

Analytical Method

Reference documents

Does the model exist ? Estimation of standard uncertainties

Analytical method assumptions?

yes

?

ISO 5725

ISO GUM

ISO GUM non supplément 1

Variance Analysis

Law of Propagation

Monte Carlo Method

Result and Uncertainty "Validation" and update 20/05/2015


39

Measurement uncertainty evaluation Propagation of Uncertainty (GUM) x 1, u(x 1) x 2, u(x 2)

Y = f(X1, X2, X3)

y, u(y)

y ± U (k=2)

x 3, u(x 3)

Propagation of Distributions (GUM-S1)

Y = f(X1, X2, X3)

y~ ;u ( y~)

[ylow;yhigh] 20/05/2015


40



Step 2: Quantification of the sources of uncertainty of the xi Standard Uncertainty u(xi) or distribution



Measurand Y = f(Xi)


f(X1,..,Xn)

Step 3’: Sensitivity Analysis, Hierarchisation

Feedback 20/05/2015


41

GUM – Step 2 : Quantification of the standard uncertainties 2 Experimental Data

Type A

No Experimental Data A Priori Information

Type B

Presentation of the Statistical Tools Used by GUM : each quantity is considered as a random variable

20/05/2015


42

GUM Step 2 – Random variable • RANDOM VARIABLE [ ISO 3534-1,1.2] o

A function which can be applied to any value of all potential results and to which a distribution of probability is attached

o

note 1 : a r.v which can only be applied to isolated values is called discrete. A r.v. which can be applied to all values within a finite or infinite interval is called « continuous ».

o

note 2 : the probability of an event A is noted as Pr(A) or P(A)

• Examples : o

The result of a dice throw (discrete : possible values {1;2;3;4;5;6} )

o

The size of an individual chosen at random from a given population (continuous)

o

A measurement result (continuous)

20/05/2015


43

GUM – Step 2 : Statistical parameters • Mean or Expectation o

Location parameter of a probability distribution/a sample

o

Linear operator

o

μ=4 ∑

•

Variance

o

Scale parameter : quantifies the spread of a distribution or a sample

o

Non linear operator

o o

=

∑

=∑

5

5

=∑

54

+ 2∑

5

<

∑6

;

5 56 789

;

6

If independent random variables =

20/05/2015

5

=

5


44

GUM – Step 2 : Type A n repeated measurements under repeatability conditions (same operator, apparatus, short time interval, independent measurements) Measure

Temperature of the laboratory (°C)

1

22.3

2

22.7

3

23.5

4

23.1

5

23.3

6

22.8

7

23.0

8

22.9

9

22.4

10

23.2 20/05/2015

Mean value ====

1

= = 22.92 °7

Standard deviation

2 = =

=

1 −1

= −=


= 0.38 °7

45

GUM – Step 2 : Type B Investigation of the available knowledge for the input quantity Xi Transcription of that knowledge in terms of a probability distribution Determination of the standard uncertainty as the standard deviation of the chosen probability distribution 20/05/2015

Available knowledge ?

Probability distribution

Standard deviation u(Xi)


46

GUM – Step 2 : Type B (available information) Where can you find available information ?

Available knowledge ?

•

Calibration certificate (with an associated expanded uncertainty U)

•

Verification certificate

•

Resolution of a displaying device

•

Maximum Permissible Errors (MPE)

•

Bounds for an oscillating phenomenon

•

Expert knowledge

•

Physical limits

•

…

20/05/2015


47

GUM – Step 2 : Type B (choice of the probability distribution) • Common probability distributions in metrology : Available information


Best estimate xi with an associated expanded uncertainty U (calibration certificate)

Gaussian

Resolution q of a displaying device

Rectangular

2

=

Maximum Permissible Error (± MPE)

Rectangular

2

=

Oscillating phenomenon between two values a and b

Arcsine (U-shaped)

2

=

…

…

…


Standard deviation u(Xi)

20/05/2015

Standard deviation D = E

2


12 F 4

3 G−5 8

48

GUM – Step 2 : Type B

o Interval of values

o A distribution of probability

-a

a

or

-a

20/05/2015

a

u(xi)

(GUM)

σ or S (Stats)

...


49

GUM – Step 2 : Covariance/Correlation The covariance may be either positive or negative : •

Covariance between the outside temperature and the inside temperature ? •

•

Positive : the two quantities vary in the same direction

Covariance between the outside temperature and the power consumption ? •

Negative : the two quantities vary in opposite directions

The influence of the covariance on measurement uncertainty may be positive or negative, depending on : •

The sign of the covariance

•

The measurement model

The correlation coefficient r may be easier to interpret : • • •

−1 ≤ H ≤ 1 789

;

6

=H

;

6

.2

.2

6

It quantifies the intensity of the relationship between the variations of the two involved quantities 20/05/2015


50

GUM – Step 2 : intensity of the Covariance/Correlation

35

Correlation between Tout and Tins

35

25

25

15

15

5

5 15

20

25

Old building, single glazing windows, bad thermal insulation : r = 0.97

20/05/2015

Correlation between Tout and Tins

15

20

25

New building, double glazing windows, good thermal insulation : r = 0.55


51

GUM – Step 2 : Calculation of the covariance n Simultaneous observations of the two quantities X and Z. 1 − ̅ J − J̅ I89 , J = Measure Tin Tout −1 1

22.3

23.2

2

22.7

15.0

3

23.5

23.6

4

23.1

24.6

5

23.3

25.3

6

22.8

15.8

7

23.0

19.2

8

22.9

20.6

9

22.4

20.3

10

23.2

22.2

Mean

22.92

20.98

Std dev

0.38

3.52

20/05/2015

I89 = , =KLM = 0.493 H

;

6

= 0.41


52

GUM – Step 2 : Covariance arising from the measurement model Consider L independent random variables (r.v): Q1, Q2, Q3,….., QL

X1 r.v (F) function of previous r.v:

X2 r.v (G) function of previous r.v:

X1= 2Q1-5Q3+Q4-3Q8

X2 =Q1+6Q2+Q8

u ( x 1 , x 2 ) = u ( F (Q 1 , Q 2 ,..., Q L ), G (Q 1 , Q 2 ,..., Q L )) =

L

∑

i =1

∂F ∂G u 2 (Q i ∂Q i ∂Q i

)

u ( x 1 , x 2 ) = u (2Q 1 - 5Q 3 + Q 4 - 3Q 8 , Q 1 + 6Q 2 + Q 8 ) = (2) (1) u 2 (Q1 ) + ( −3) (1) u 2 (Q8 )

Example : Two temperature measurements T1 and T2 with the same sensor/thermometer : o

The trueness error is the same for the two temperatures

o

T1 and T2 have a common uncertainty component : the trueness of the sensor 20/05/2015


53



Step 2: Quantification of the sources of uncertainty of the xi Standard Uncertainty u(xi) or distribution



Measurand Y = f(Xi)


f(X1,..,Xn)


Feedback 20/05/2015


54

GUM – Step 3 : Law of Propagation of Uncertainty (LPU) • $=#

,…,

,…,

• Uncertainty --> deviations from the expectation values o ∆Y = Y - E(Y) o ∆Xi = Xi - E(Xi) o E(Y) and E(Xi) being constant V(∆Y )=V(Y) and V(∆Xi )= V(Xi) • Hypothesis 1 : all deviations are small enough for order 1 of the Taylor development to be applied exclusively:

Error Propagation Formula

20/05/2015


55

GUM – Step 3 : Law of Propagation of Uncertainty (LPU) • Hypothesis 2 : all partial derivatives are constant around the point applying the formula of the variance of a sum, the result is :

Variance Propagation Formula or Law of the Propagation of Uncertainty (GUM)

20/05/2015


56

GUM – Step 3 : Law of the Propagation of Uncertainty (LPU) 2

N −1 N   f ∂f ∂f ∂ 2 ( ) u c2 (y ) = ∑  u x + 2 u xi, x ∑ ∑  i i =1 j = i +1 ∂ x i ∂ x j i =1  ∂ x i  N

(

2 2 c ∑ i u (x i ) + 2 ∑ i =1

∑ ci

i =1 j = i + 1

N −1

N

u c2 (y ) = ∑ c i2u 2 ( x i ) + 2∑ i =1

u 2 (y ) =

)

c j u xi , x j

)

N −1 N

N

u c2 (y ) =

j

(

∑ c c u (x ) u (x )r (x , x ) N

i

j

i

j

i

j

i =1 j = i +1

N

N −1

i =1

i =1 j = i +1

2 2 c ∑ i u ( xi ) + 2∑

N

∑

ci c j u (xi , x j )

Revised GUM 20/05/2015


57

GUM : Uncertainty budget/Sensitivity analysis • In the case of independent input quantities 2 $ =

I 2

• Sensitivity coefficient associated to the input quantity Xi. O# I = O • Sensitivity = Contribution of the input quantity Xi to the overall measurement uncertainty I 2 = 2 $ Identification of the most influential input quantities in order to : Reduce the uncertainty associated to the output quantity by the reduction of the uncertainty associated to the most influential input quantities Fix the non influential input quantities to their best estimate 20/05/2015


58


Step 3: Propagation of Uncertainty (LPU, MCM) Step 4: Step 2: Quantification of the sources of uncertainty of the xi Standard Uncertainty u(xi) or distribution

Step 1: Analysis of the Measurement Process

Uncertain Input Quantities : xi


Measurand


Y = f(Xi)

f(X1,..,Xn)


Feedback 20/05/2015


59

GUM – Step 4 : Final expression of the result

y ± U unit (value of k )

Expanded uncertainty

Coverage interval

20/05/2015

Coverage factor

P − Q; P + Q


60

GUM – Step 4 : Definitions

Standard uncertainty (GUM 2.3.1)

Expanded uncertainty (GUM 2.3.5)

« uncertainty of the result of a

« Quantity defining an

measurement

around

expressed

as

a

standard deviation »

about

measurement

the

interval

result

that

may

of

a be

expected to encompass a large fraction of the distribution of values

u(y)

that could reasonably be attributed to the measurand »

k : coverage factor

U = k u(y) 20/05/2015


61

GUM – Step 4 : Expanded uncertainty

U = k uc(y)

Expanded uncertainty

Coverage Factor

Combined standard uncertainty of y (Step 3 Result)

(GUM § 2.3.6) « numerical factor used as a multiplier of the combined standard uncertainty in order to obtain an expanded uncertainty » (VIM 3 § 2.38) « number larger than 1 by which a combined standard measurement uncertainty is multiplied to obtain an expanded measurement uncertainty » 20/05/2015


62

GUM – Step 4 : Determining the coverage factor Model y = f(x1, x2, …, xN) Does Y have a gaussian distribution or approximative gaussian ? (Valid assumptions of Central Limit Theorem (CLT)) Yes

k =2

So that p =

p ≈ 95%

compute

No

(1 − α )%

ν

eff

(dof)

k = Student Quantile ν eff

(1 − α )%

k=2

So that p =

p=?

Monte Carlo Simulations

See GUM 20/05/2015


63

GUM – Step 4 : Determining the coverage factor The Central Limit Theorem (CLT) The Central Limit Theorem: any sum of a large number of independent random variables, identically distributed, is a normal asymptotic variable Corollary : the only necessary condition for its application is that regardless of the distributions of the different variables, no variable or group is to be predominant compared to the others, within the sum.

In simpler terms: A phenomenon may be represented by a normal distribution if it results from a great number of additional, small and independent effects

20/05/2015


64

GUM – Step 4 : Final expression of the result

y ± U unit (value of k ) • In order to have a physical significance, a measurement result must be rounded according to the uncertainty related to the result. o U is rounded up to at most 2 significant digits o The measurement result y is rounded accordingly • For instance, there is no need to announce a length measurement by the micron if the uncertainty related to the measurement with a sliding caliper corresponds to the tenth of a millimeter

• The rounding must not be carried out in several stages but only once, when reporting the final result

20/05/2015


65

GUM – Step 4 : Summary

U = k u(y) u(y) obtained at Stage 3 Very often k = 2 U rounded to 2 significant digits

y ± U

unit

(k=2)

y rounded to the same position as U Note the unit

20/05/2015


66



•

Reference documents

•


•



Linear example

•


• Conclusion •


•

Bayesian methods

•



67

Different methods for the evaluation of measurement uncertainty Measurement Process analysis


no

?

yes

Does the model exist ? Analytical Estimation of standard method uncertainties hypotheses ?

Analytical Method

Reference documents

yes

?

ISO 5725

ISO GUM


Variance Analysis

Law of Propagation

Monte Carlo Method



68

GUM-S1 : The Monte Carlo method • Name derived from the Monaco roulette, which is considered to be a mechanism designed specifically to display numbers randomly. • The objective of the Monte Carlo method is to artificially reproduce a random phenomenon • More specifically, it consists of simulating a factitious sample of realisations of that phenomenon based on random variable assumptions. The statistical analysis made on a large sample will be close to reality 20/05/2015


69

GUM-S1 : « GUM uncertainty framework » Application of the Law of Propagation of Uncertainty f approximated by the 1st order of the serial development of Taylor uc2

2

N −1 N   (y ) = ∑  ∂f  u 2 (x i ) + 2 ∑ ∑ ∂f ∂f u x i , x j i =1  ∂x i  i =1 j =i +1 ∂x i ∂x j N

(

)

• low deviations are considered • f is linear (linearisable) • values of the derivatives available

y, u(y) output quantity Y is Gaussian The Central Limit Theorem is to be applied • independence of xi s • The variance associated to Y is greater than any c²iσ ²(xi) component for an xi whose distribution is not gaussian

20/05/2015


y±U

70

Measurement uncertainty evaluation Propagation of Uncertainty (GUM) x 1, u(x 1) x 2, u(x 2)

Y = f(X1, X2, X3)

y, u(y)

y ± U (k=2)

x 3, u(x 3)


Y = f(X1, X2, X3)

y~ ;u ( y~)

[ylow;yhigh] 20/05/2015


71

GUM-S1 : Different steps of the Monte Carlo method Writing of the mathematical model

Step 1

Y=f(x1,x2,x3)

Choice of distributions for input variables

Step 2

Step nb.1

Step nb.2

Step 3 Step nb. M

X1

X2

 x11   x12   x1M

x21 x22 x2M

X3

Y

Calculation of M values resulting from Y Empirical distribution of the measurand

Step 3

 y1  x31     x32  f  y 2   ⇒      x3M  y M  Random generation of M input variable samples

20/05/2015

Step 4

~ y ; u( ~ y) [ylow;yhigh] Extraction of the: Mean standard deviation interval at p%


72

GUM-S1 : Choice of the distributions Propagation of Uncertainty (GUM) x 1, u(x 1) x 2, u(x 2)

Y = f(X1, X2, X3)

y, u(y)

y ±U

x 3, u(x 3)


Y = f (X1, X2, X3)

20/05/2015


y~ ;u ( y~)

[ylow;yhigh]

73

Reminder : GUM – Type B

o Interval of values

o A distribution of probability

-a

a

or

-a

20/05/2015

a

u(xi)

(GUM)

σ or S (Stats)

...


74

GUM-S1 – Step 2 : Probability distributions • Probability distributions commonly used in metrology: o

Normal or Gaussian: calibration certificates

o

Rectangular or Uniform: device resolution

o

Triangular

o

Student: repeated number of measurements

o

U-shaped: oscillating phenomenon

• Other probability distributions : o

Beta, gamma, weibull, Poisson, exponential, log-normal,…

20/05/2015


75

GUM-S1 – Step 2 : Associating a Distribution to an Input Quantity Maximum entropy principle, advocated in Supplement 1 to the GUM Choice of expert, difficulty to quantify prior information available. Statistical goodness-of-fit tests between the experimental and theoretical distributions: Tests: Anderson-Darling, Kolmogorov-Smirnov, Chi-squared test,… Use of statistical software

20/05/2015


76

GUM-S1 – Step 2 : Maximum entropy principle

Objective: Selection of a probability density function that is consistent with our knowledge and introduces no unwarranted information1

∫

Formula for a density probability g:

h(g ) = − g (t ) log(g (t ))dt

Maximum Entropy: used to determine a prior probability distribution The principle consists of finding the distribution g which maximizes the quantity h(g) (representing misreadings of the phenomenon) given the only available information Important property in Bayesian methods

20/05/2015


77

GUM-S1 – Step 2 : Assignment of probability distributions Available information on quantity X

Distribution to be assigned to X according to maximum entropy

Lower limit: a Higher limit: b

R (a,b) Rectangular

Best estimate x with associated uncertainty u(x)

N (x, u²(x))

X varies sinusoidally between limits a and b (a
U-shaped(a,b)

Best estimation λ of a positive quantity

graph

Gaussian

Arcsine Derivative

1 e( ) λ

Exponential

Normal distribution corresponds to maximum entropy taken from all the distributions which have an identical mean x and a identical standard deviation u(x) 20/05/2015


78

GUM-S1 : Assignment of probability distributions (Type A) • Using the Student distribution : Approximation to the Gaussian distribution for n ≤ 30 Example: SO2 Concentration Readings (in ppb) 112.9 114.1 112.9 113.4 113.4

113.6 113.2 115.4 113.1 113.6

113.5 114.4 112.7 113.6 113.5

Applying the Student distribution with n – 1 degrees of freedom

Mean = 113.55 Scale (experimental std deviation) = 0.68 Number of df = 14 Student Standard Deviation :

20/05/2015


79

GUM-S1 – Step 3 : Generating random numbers • The pseudo random number generator must be tested prior to any calculation: o

Resistence to adequation tests

o

Period

o

independence of generated values

• Sampling of the distribution function o

Generation of a random number ψ in the interval [0 ; 1]

o

Calculation of value x1 verifying G(x1) = ψ

(G-1(ψ)= x1)

x

G( x ) =

∫ g (t )dt

−∞ 20/05/2015


80

GUM-S1 – Step 3 : Generating random numbers Example : Sampling from a Gaussian distribution N(0,1) o Generation of a random number ψ in the interval [0 ; 1] o Calculation of value x1 verifying G(x1) = ψ (G-1(ψ)= x1)

R = 0.71

0.5534 20/05/2015


81

GUM-S1 – Step 3

Random trials of M input variable samples

Step N°1

Step N°2

Step 3 Step N°M

X1

X2

 x11   x12   x1M

x21 x22 x2M

X3

Y

Calculation of M values resulting from the empirical measurand distribution Y

Step 4

 y1  x31     x32  f  y 2   ⇒      x3M  y M 

Number of M Monte Carlo simulations? GUM-S1 : 106

20/05/2015


82

GUM-S1 – Step 3 : Choice of Simulation Software • Many software have pseudo random number generation algorithms : Excel, Crystal Ball, Matlab, R, C++,… • Decision criteria : o

Software validation,

o

Ergonomy, ease of use,

o

The duration of the calculations (from a few seconds to several hours) also depends on : The complexity of the model, The number M of draws.

o

Costs: purchase, maintenance, training or computation development

20/05/2015


83

GUM-S1 – Step 3 : Experience and validation • Use of Matlab : Mersenne-Twister algorithm o

recognized as a good pseudo random number generator

o

Period T = 219937-1

o

Quoted in ISO 28640:2010: « Pseudo Random Number Generating Methods»

• Validation of another tool (ex : CRYSTAL BALL V.5.2.2 ) o

Identification and properties of pseudo random number generator used

o

Adequation test of simulated distributions (rectangle, normal,etc.)

o

Comparison and statistical validation with Matlab (or any tool validated beforehand)

20/05/2015


84

GUM-S1 – Step 4 Propagation of Uncertainty (GUM) Most frequently k = 2 x 1, u(x 1) x 2, u(x 2)

Y = f(X1, X2, X3)

y, u(y)

y ±U

x 3, u(x 3)


Interval at p% Coverage interval

Y = f(X1, X2, X3)

20/05/2015


[ylow;yhigh]

85

GUM-S1 – Step 4 : Distribution of the measurand The value attributed to the measurand is obtained by calculating the empirical mean of the output distribution: V

1 TU = F

T

The standard uncertainty of the measurand is obtained from the experimental standard deviation of the output distribution: 1 2 TU = F−1

V

T − TU

The expanded uncertainty corresponds to the interval at p% which contains proportion p chosen among the possible values of the measurand: TWKX ; TY ZY 20/05/2015


86

GUM-S1 – Step 4 : Choice of the coverage interval Quantile function (= Cumulative distribution function) [\ < = T such that = P $ ≤ T

p = 0.95 1

^

( 2

0.975

0.025 [\ <

( 2

Symmetrical distribution ( < \ [ ; [\ < 1 2 20/05/2015

[\ <

1.96

( 2

1

avec (

( 2

1.96

1


87

GUM-S1 – Step 4 : Choice of the coverage interval • Asymmetrical distribution o o

Different candidates coverage intervals GUM-S1 recommends to choose the shortest interval among all possible intervals : Determine ( ∈ 0; 1 −

such that [\ <

1−

^

− [\ <

^

is minimum

Determine ylow and yhigh such that %a TWKX = %a TY ZY Example of Triangular distribution between 1 and 9 with Mode 3

Iy = 8.13-1.74 = 6.36 20/05/2015

Iy = 7.48-1.14 = 6.34

Iy = 7.68-1.47 = 6.21


88

GUM-S1 – Step 3’ : Uncertainty budget / sensitivity analysis • Not straightforward : it requires extra calculations • Proposal of the GUM-S1 : Perfoms the Monte Carlo simulation holding all quantities but one fixed at their best estimate o

N additional Monte Carlo simulations for N input quantities in the measurement model

o

Does not evaluate interaction effects

o

Correlation/ Regression coefficients :

● No additional Monte Carlo simulations ● Does not evaluate interaction effects •

Variance based sensitivity indices (ex: Sobol’ indices) o

Requires additional Monte Carlo simulations

o

Gives an evaluation of significant interaction effects

A.Saltelli, Sensitivity analysis, Wiley, 2000 A.Allard & N.Fischer, Recommended tools for sensitivity analysis associated to the evaluation of measurement uncertainty, World Sc. Publ. Comp., 2012, pp 1-12 20/05/2015


89

Different methods for the evaluation of measurement uncertainty Measurement Process analysis


no

?

yes

Does the model exist ? Analytical Estimation of standard method uncertainties hypotheses ?

Analytical Method

Reference documents

yes

?

ISO 5725

ISO GUM


Variance Analysis

Law of Propagation

Monte Carlo Method



90

Inter-Laboratory Comparison (ILC) • Standard ISO 5725-2 : Accuracy (trueness and precision) of measurement results and methods o

Alternative uncertainty estimation method when one does not know how to write the model

o

estimation of the trueness and the dispersion of a measurement method through interlaboratory tests

• Trueness (VIM 2012,2.14) : closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value. • Precision (VIM 2012, 2.15) : closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions. 20/05/2015


91

Inter-Laboratory Comparison (ILC) Step 1 no y=f(x1,x2,...xn)

yes

Step 2 quantify

Standardized Method + Interlaboratory tests ISO 5725 : sr et sR

no

Estimate One intermediate uncertainty

yes u(xi) participation ?

no

Proof Normalized Method

yes Step 3 uncertainty propagation uc(y)

uc(y) = sR

Step 4 Expanded uncertainty U = kuc(y)

20/05/2015


92

ILC : Expression of the precision

Minimum

5725-2

Maximum

Reproducibility conditions (R)

repeatability (r) conditions Intermediate Precision 5725-3 same method identical individuals same laboratory same operator same equipment short time interval

20/05/2015

same method identical individuals different laboratories different equipment


93

ILC : Calculation for one level (same ni ) Results raw Lab 1

y11

…

Mean T

y1n

… Lab i

Descriptive Statistics

s1

… yi1

Tb

yin

… Lab p

Standard deviation

si

… yi1

T

ypn Grubbs

sp Homogeneity Tests (filter p⇒ p*)

Cochran

Hypothesis : ni = n constant

Tc =

e

∑ Tb ∗

r

20/05/2015

=

1 ∗−1

=

1

repeatability standard deviation

∗

Tb − Tc

+

−1

e

reproducibility standard deviation


94



•

Reference documents

•


•



Linear example

•


• Conclusion •


•

Bayesian methods

•



95

Example 1 : Linear model – Step 1 Determination of the thickness of a sample •

Step 1 : analysis of the measurement process Measurand : “Thickness e of the sample expressed in mm, using a sliding caliper, at (20 ± 2) °C” Uncertainty sources :

● Trueness of the sliding caliper ● Quantification of the sliding caliper ● Repeatability Mathematical model :

t = tu +

20/05/2015

+ t̅ Workshop measurement uncertainty Jakarta

96

Example 1 : Linear model – Step 2

•

Step 2 : Quantification of the uncertainty sources Type B :

● Information available for the trueness : a calibration certificate with an expanded uncertainty U = 0.1 mm (k=2)

● Information available for the quantification : The sliding caliper has a quantification step of q = 0.1 mm Type A :

● Repeatability : The operator performs n = 10 measurements

20/05/2015


97

Example 1 : Linear model – Step 2 GUM

GUM-S1

• Trueness o

2 tu =

• Trueness

v w

=

.

= 0.05 mm

o Gaussian distribution o Mean : 0, std deviation : v . 2 tu = w = = 0.05 mm

• Quantification o

Rectangular distribution

o

2

=

x

=

.

• Quantification

o Rectangular distribution x

= 0.029 mm

o Lower bound : − = −0.05 mm

3.1 3.0

• Repeatability

3.1 2.9 •

o

n = 10 repetitions

3.2 3.2

o

t̅ = 3.05 mm

3.0 3.1

o

2 t̅ =

= 0.11 mm

20/05/2015

2.9 3.0

x

o Upper bound : = 0.05 mm

Repeatability

o Student distribution o t̅ = 3.05 mm o I5•t = = 0.11 mm


98

Example 1 : Linear model (Step 3 and 4) GUM

GUM-S1

2 t = 2 tu + 2

+ 2 t̅

2 t =

+ 2 t̅

2 t =

2 tu + 2

M = 106 Monte Carlo iterations

0.05 + 0.029 + 0.11 2 t = 0.123 mm

D = 2 ∗ 2 t = 0.25 mm Coverage interval : T − D; T + D = 2.80; 3.30 20/05/2015

t̃ = 3.05 2 t̃ = 0.14 Coverage interval : tWKX ; tY ZY = 2.78; 3.32 Workshop measurement uncertainty Jakarta

99

Example 1 : Uncertainty budget GUM

GUM-S1

Input quantity

Coontribution to the variance (%)

Input quantity


Trueness eT

16%

Trueness eT

14%

Repeatability t̅

79%

Repeatability t̅

81%

Quantification q

5%

Quantification q

5%

20/05/2015


100

Example 1 : Conclusion

• Good agreement between GUM and GUM-S1 • Linear measurement model • The repeatability is the most influential input quantity •

Importance of the number of repeated measurements performed

20/05/2015


101

Example 2 : Mass calibration (GUM-S1) • Mass calibration o

Step 1 : analysis of the measurement process Measurand : Mass difference δm between the conventional mass mW,c to calibrate and the nominal mass mnom, expressed in mg. Uncertainty sources :

● Conventional mass mR,c(mg) of the reference weight, ● Conventional mass δmR,c(mg) of a small weight added to R to balance it with W,

● Mass density ρa(kg/m3) of air, ● Mass density ρW(kg/m3) of the weight W to calibrate. ● Mass density ρR(kg/m3) of the reference weight R.

20/05/2015


102

Example 2 : Mass calibration – Step 2 GUM-S1

GUM •

o o

•

o

ρa o o

•

ρW o o

•

•r,‚ = 100 000 mg 2 •r,‚ = 0.05 mg

δmR,c o

•

•

mR,c

ρR o o

•

δ•r,‚ = 1.234 mg

2 δ•r,‚ = 0.02 mg

•

0ƒ = 1.20 kg/m†

2 0ƒ = 0.058 kg/m†

•

0‡ = 8000 kg/m†

2 0‡ = 577 kg/m† 0r = 8000 kg/m†

2 0r = 29 kg/m† 20/05/2015

•

mR,c o Gaussian distribution o Mean : •r,‚ = 100 000 mg, Standard deviation : 2 •r,‚ = 0.05 mg δmR,c o Gaussian distribution o Mean :δ•r,‚ = 1.234 mg, Standard deviation : 2 δ•r,‚ = 0.02 mg ρa o Rectangular distribution o Lower bound : 1.10 kg/m†, Upper bound : 1.30 kg/m† ρW o Rectangular distribution o Lower bound : 7000 kg/m†, Upper bound : 9000 kg/m† ρR o Rectangular distribution o Lower bound : 7950 kg/m†, Upper bound : 8050 kg/m†


103

Example 2 : Mass calibration – Steps 3 and 4 GUM-S1

GUM 2 ‹• = 7•r,‚ 2 •r,‚ + 7••r,‚ 2 /•r,‚

M = 106 Monte Carlo iterations

+7‘ƒ 2 0ƒ +7‘‡ 2 0‡ +7‘r 2 0r

2 ‹• =

2 •r,‚ + 2 /•r,‚

2 ‹• = 0.05 + 0.02 2 ‹• = 5.4 × 10<Ž kg Œ = 1.234 × 10<• kg ‹• Œ = 7.5 × 10<Ž kg 2 ‹•

D = 2 ∗ 2 ‹• = 11 × 10<Ž kg Coverage interval : T − D; T + D = 1.124 × 10<• ; 1.343 × 10<• 20/05/2015

Coverage interval : ‹•WKX ; ‹•Y ZY = 1.083 × 10<• ; 1.383 × 10<•


104

Example 2 : Mass calibration (uncertainty budget) GUM-S1

GUM Input quantity


Input quantity


Convent. Mass of the reference weight mR,c

86%

Convent. Mass of the reference weight mR,c

43%

Convent. Mass of the added weight δmR,c

14%

Convent. Mass of the added weight δmR,c

8%

Interaction between the mass densities ρa and ρW

Not available

Interaction between the mass densities ρa and ρW

49%

20/05/2015


105



•

Reference documents

•


•


• GUM methodology • Supplement 1 to the GUM (Monte Carlo) • Examples •

Linear example

•


• Conclusion •


•

Bayesian methods

•



106

Conclusion : Scope of the GUM/GUM-S1

GUM Measurement model

GUM-S1

Explicit mathematical expression

Linearity

Linear or approximately linear

No assumptions

Distribution associated to the measurand

Often Gaussian

Any distribution

Software implementation

Excel sheet

Software with a validated random generation number algorithm

Sensitivity analysis

Directly availabe (partial derivatives)

Requires additional calculations or simulations

Indirect measurements

20/05/2015

Not addressed


107

Conclusion : Bayesian methods • The Bayesian methods offer a framework that combines both experimental data X and prior knowledge of a measurand θ. Prior probability distribution

Bayes theorem

’ |” =

Likelihood function ’ • ’; ” ”|’

Posterior probability distribution

• The combination of prior information with information carried out by experimental data is summarized in the posterior probability distribution o Useful for inverse and regression problems (see JRP NEW04 « Novel mathematical and statistical approaches to uncertainty evaluation », http://www.ptb.de/emrp/new04-home.html ) 20/05/2015


108

General conclusion (1/4) • The evaluation of measurement uncertainty is a requirement : a measurement result without any associated has no meaning and is useless. • Harmonization of the methodology to evaluate uncertainty : o The Guide to the expression of Uncertainty in Measurement (GUM), o The propagation of distributions using a Monte Carlo method (GUM Supplement 1)

• Harmonization of the vocabulary used in metrology : o The International Vocabulary of Metrology (VIM) 20/05/2015


109

General conclusion (2/4)

• Before any uncertainty quantification, a deep analysis of the measurement process is required. • GUM steps : •

Step 1 : Analysis the measurement process : $ = #

,…,

•

Step 2 : Quantification the standard uncertainty u(Xi) associated with the input quantities

•

Step 3 : Propagation of uncertainty using the Law of Propagation of Uncertainty.

•

Step 4 : Reporting of the measurement result

• The GUM has been the reference method since 1994 (more than 20 years) ! 20/05/2015


110

General conclusion (3/4)

• However, in some applications, GUM methodology may not be suitable : •

Non linear measurement models

•

Models having one predominant non-Gaussian input quantity

•

Computational codes

•

…

• GUM-S1 steps : •

Step 1 : Analysis the measurement process : $ = #

,…,

•

Step 2 : Attribution of a probability density function to each of the input quantities

•

Step 3 : Propagation of distributions using Monte Carlo simulation

•

Step 4 : Reporting of the measurement result

20/05/2015


111

General conclusion (4/4) • In most cases, both methods are consistent because in metrology, we often deal with : •

Small deviations from the best estimates (the linear approximation of the GUM is then often adequate).

•

Many different uncertainty sources that contribute signifiicantly to the measurand (the measurand is then often Gaussian).

• Despite its wide use in the world, the GUM needs revisions •

Harmonization with its supplements and the VIM

•

Examples in a separate document

•

A revised GUM has been reviewed by NMIs in early 2015…

•

… and comments are under investigation by JCGM

•

But the methodology still relies on the LPU

20/05/2015


112

Questions ?

Thank you ! Merci ! terima kasih !

20/05/2015


113

Materi Sesi i General

Recommend Documents