Software Quality Requirements & Evaluation - IsO 25000

Pittsburgh, PA 15213-3890

Software Quality Requirements and Evaluation, the ISO 25000 Series PSM Technical Working Group February 2004 Dave Zubrow Sponsored by the U.S. Department of Defense © 2004 by Carnegie Mellon University

Carnegie Mellon Software Engineering Institute

Background This presentation reviews the latest developments with the ISO 25000 SQuaR SQuaRE E serie series s of stand standards. ards. This serie series s on Software Quality Requirements and Evaluation (SQuaRE) is an effort to harmonize ISO 9126 and ISO 14598. The information presented here is based on the current state of the standards/

Note: Many of the slides in this presentation are taken from WG 6 documents and materials, especially slides from Professor Azuma, WG convener.

page 2 © 2004 by Carnegie Mellon University


Background This presentation reviews the latest developments with the ISO 25000 SQuaR SQuaRE E serie series s of stand standards. ards. This serie series s on Software Quality Requirements and Evaluation (SQuaRE) is an effort to harmonize ISO 9126 and ISO 14598. The information presented here is based on the current state of the standards/

Note: Many of the slides in this presentation are taken from WG 6 documents and materials, especially slides from Professor Azuma, WG convener.



Outline Background and Overview Concepts and Models Software Product Quality Measurement Software Product Quality Requirements and Evaluation



Relationship between ISO/IEC 9126 and ISO/IEC 14598 R e s o u rc e s a n d e n v ir o n m e n t

E v a lu a tio n support

E v a lu a tio n process

E v a lu a tio n process

S o f tw a r e product

Internal m e t ric s

E x te r n a l m e t ric s

E ffe c t o f th e s o ftw a r e product

Q u a lity i n u s e m e t r ic s

14598-1 1 4 5 9 8 - 2 14598-6

14598-3

9126-1

1 4 5 9 8 - 4 -

9126-3

9126-2

9126-4



Relationship and transition process between ISO/IEC 9126, ISO/IEC 14598 and SQuaRE series of standards C U R R E N T

S Q u a R E

9 1 2 6 : P r o d u c t q u a l i ty

2 5 0 0 0 : Q u a l i t y M a n a g e m e n t D i v is i o n

- 1 : Q u a l it y m o d e l

2 5 0 0 0 : G u id e to S Q u a R E (N P )

- 2 : E x t e r n a l m e t ri cs

2 5 0 0 1 : P l a n n in g a n d m a n a g e m e n t

- 3 : In t e r n a l m e t r i c s

2 5 0 1 0 : Q u a l it y M o d e l D i v i s io n

- 4 : Q u a l i ty i n u s e m e t ri c s

2 5 0 1 0 : Q u a l it y m o d e l a n d g u i d e ( R e v )

2 5 0 2 0 : Q u a l it y M e a s u r e m e n t D i v i s i o n N e w P r o p o s al G u id e s to u s e 9 1 2 6 &

1 4 5 9 8

2 5 0 2 0 : M e a s u r e m e n t re fe r e n c e m o d e l a n d g u i d e ( N P ) 2 5 0 2 1 : M e a s u r e m e n t p r im i t i v e s ( N P )

B a s e m e t r ic s

2 5 0 2 2 : M e a s u r e m e n t o f in t e rn a l q u a l it y

Q u a l i ty r e q u i r e m e n t s

2 5 0 2 3 : M e a s u r e m e n t o f e x te r n a l q u a l ity 2 5 0 2 4 : M e a s u r e m e n t o f q u a l it y in u s e

1 4 5 9 8 : P r o d u c t e v a lu a t io n

2 5 0 3 0 : Q u a li t y R e q u i r e m e n t s D iv is io n

-1: G eneral overview

2 5 0 3 0 : Q u a l it y r e q u i re m e n t s a n d g u i d e (N P )

- 2 : P l a n n in g a n d m a n a g e m e n t

2 5 0 4 0 : Q u a l i t y E v a l u a t io n D i v i s i o n

- 3 : P r o c fo r d e v e l o p e r s -4: Proc for acquirers

2 5 0 4 0 : Q u a l it y e v a lu a t io n o v e r v ie w

and guide

2 5 0 4 1 : E v a l u a t io n m o d u l e s

-5: Proc for evaluators

2 5 0 4 2 : P r o c e s s fo r d e v e l o p e r s

- 6 : D o c o f e v a l u a t io n m o d u l e s

2 5 0 4 3 : P r o c e s s fo r a c q u i r e rs 2 5 0 4 4 : P r o c e s s fo r e v a l u a t o r s



State of the Standards 25000, Guide to the SQuaRE series – Ballot out for FCD 25020, Software Product Quality Measurement Reference Model – Ballot out for 2nd CD 25021, Measurement Primitives – Ballot out for 1st CD 25030, Quality Requirements – Ballot out for 2nd CD All Ballots closing Mid-April to Early May Editors assigned but no drafts out 25010, Quality Model 25023, External Quality Measures



SQuaRE:

Architecture ISO/IEC ISO/IEC2501n 2501n Quality QualityModel Model Division Division

ISO/IEC ISO/IEC2503n 2503n Quality Quality Requirement Requirement Division Division

ISO/IEC ISO/IEC2500n 2500n Product ProductQuality Quality General GeneralDivision Division General General Overview Overview and and Guide Guide to to the the SQuaRE SQuaRE Planning Planning and and Management Management

ISO/IEC ISO/IEC2502n 2502n Quality QualityMetrics Metrics Division Division © 2004 by Carnegie Mellon University

ISO/IEC ISO/IEC2504n 2504n Quality Quality Evaluation Evaluation Division Division

page 7


SQuaRE general reference model Software Product

Target of process

Process

User Task

(Effect)

Requirements Specification

Requirements Specification

Planning

Planning

Measurement

Measurement

Evaluatioin

Evaluation

Execution

25030 (QR)

25001 (PM)

25021( MP) 25022 (MfIQ) 25023 (MfEQ)

25041(EM) 25042 (PD) 25043 (PA)

25030 (QR)

25001 (PM)

25021( MP) 25022 (MfIQ) 25024 (MfQU) 25025(DEM)

25044 (PE)

25020

(Measurement Reference Model and Guide)

25041(EM) 25042 (PD) 25043 (PA) 25044 (PE)

25030 (Quality Requirements and Guide) 25040 (Evaluation Process Overview and Guide)

Particular Guidance

Legend : applied to 25010 (Quality Model and Guide) used by MP – Measurement Primitives MfIQ – Metrics for Internal Quality MfEQ – Metrics for External Quality MfQU – Metrics for Quality in Use EM – EvaluationModules PM – Planning and Management PA – Process for Acquirers PE – Process for Evaluators PD – Process for Developers QR – Quality Requirements

General Guidance 25000 (Guide to SQuaRE)






The Product Quality Measurement Reference Model



Software Product Quality Life-Cycle and Quality Measures Requirements

Products Needs

Quality In Use Measures

Users UsersQuality Quality Needs Needs

Quality Quality In InUse Use Validation

External Quality Measures Internal Quality Measures

External ExternalQuality Quality Requirement Requirement

External External Quality Quality

Verification And Validation Internal Internal InternalQuality Quality Internal Requirement Quality Requirement Quality Verification

© 2004 by Carnegie Mellon University

Implementation

page 11


Quality Model Quality model Characteristic 1

Subcharacteristic 1

Characteristic 2

Subcharacteristic 2

Characteristic n

Subcharacteristic m

Quality Attributes



ISO/IEC 9126-1 Quality In Use

Quality QualityIn InUse Use

Effectiveness Effectiveness

Productivity Productivity

Safety Safety

Satisfaction Satisfaction page 13 © 2004 by Carnegie Mellon University


ISO/IEC 9126-1 - Quality Model Quality Characteristics

Subcharacteristics

•Functionality Suitability

Accuracy

Interoperability

Security

Compliance

•Reliability Maturity

Fault tolerance

Recoverability

Compliance

•Usability Understandability Learnability Operability Comp

Attractiveness

•Efficiency Time behavior

Resource utilization

Compliance

•Maintainability Analyzability

Changeability

Stability

Testability

Compliance

•Portability Adaptability

Installability

Co-existence

Replaceability

Comp page 14

© 2004 by Carnegie Mellon University


Process and Product Measurement External External Quality Quality Measure Measure Internal Internal Quality Quality Measure Measure

Quality Quality in in Use Use Measure Measure

Software

Computer System Information System Business System Computer Centered System page 15 © 2004 by Carnegie Mellon University


Software product quality measurement reference model (SPQM-RM)



Measurement concept of the SQuaRE model



Relationship between the SQuaRE SPQM-RM and the ISO/IEC 15939 Information Model Q u a lity N e e d s

/ Q u a lity E v a l u a tio n R e p o r t

A s s e ss m e n t A n a ly s is R a t in g

In f o r m a t io n P r o d u c t

In t e r p r e t a t io n Indicator

Q u a l it y M e a s u re s A n a ly s is Function (F o r m u la )

M e a s u re m e n t P r im i t iv e s

D e r iv e d M e a s u r e M e a s u re m e n t Function B a s e

M e a s u re m e n t M e th o d

A t tr ib u t e s f o r Q u a l it y M e a s u re

S Q u a R E

S P Q M -R M

M o d e l

M e a s u re

M e a s u re m e n t M e th o d A t tr ib u t e s

IS O /IE C

1 5 9 3 9






Titles of the Measurement Standards: 2502n Common: Software Engineering Software product Quality Requirements and Evaluation (SQuaRE) Division: Software Quality Measurement (For Explanation Only) 25020 Measurement Reference Model and Guide 25021 Measurement Primitives 25022 Measurement of Internal Quality 25023 Measurement of External Quality 25024 Measurement of Quality In Use



Quality Measure and Measurement Method (Concept) Quality Measures: A variable, which shows satisfactory levels of a quality characteristic, subcharacteristic or sub-sub-characteristic, to which a value is assigned as the result of measurement Quality Measure Set: A set of a Quality Measure, Measurement Primitives that are used for deriving the quality measure, associated Scales and Measurement Methods, a Formula to combine them to generate the value of Quality Measure, and Guide to use them and analyze the results. for a Quality Characteristic or Subcharacteristic. (A line of Quality Measure Table) Quality Measurement Table: A set of Quality Measure Set for each Quality Subcharacteristic



Application of SQuaRE Activity 1

Activity 2

Activity 3

Activity 4

Activity 5

Activity 6

Activity 7

Activity 8

Phase

Requirement analysis (Software and systems)

Architectural design (Software and systems)

Software detailed design

Software coding and testing

Software integration and software qualification testing

System integration and system qualification testing

Software installation

Software acceptance support

25000 (SQuaRE) series model reference

Required quality in use, Required internal quality, Required external quality

Predicted quality in use, Predicted external quality, Measured internal quality

Predicted quality in use, Predicted external quality, Measured internal quality

Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality

Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality

Predicted quality in use, Measured external quality, Measured internal quality

Predicted quality in use, Measured external quality, Measured internal quality

Measured quality in use, Measured external quality, Measured internal quality

Key deliverables of activity

Quality in use requirements (specified), External quality requirements (specified), Internal quality requirements (specified)

Architecture design of Software / system

Software detailed design

Software code, Test results

Software product, Test results

Integrated system, Test results

Installed system

Delivered software product

Applicable measures

Internal measures (External measures may be applied to validate specifications)

Internal measures

Internal measures

Internal measures External measures




Quality in use measures Internal measures External measures



User needs, characteristics & weights Q u a l i t y in u s e W E IG H T

C H A R A C T E R I S T IC E f f e c t iv e n e s s

H

P r o d u c t iv i t y

H

Safety

L

S a t is f a c t io n

M

E x t e r n a l & i n t e r n a l q u a l it y C H A R A C T E R I S T IC

W E IG H T

S U B C H A R A C T E R IS T IC i

F u n c t i o n a l it y

U s a b i l i ty

Efficiency

H

I n t e r o p e r a b i l it y

L

S e c u r i ty

L

M a t u r i ty

L L

R e c o v e r a b i li ty

H

C o m p l ia n c e

H

U n d e r s t a n d a b i l i ty

M

L e a r n a b i li ty

L

O p e r a b i l i ty

H

A t t r a c t iv e n e s s

M

C o m p l ia n c e

H

Time

H

b e h a vio u r u t i li z a t io n

C o m p l ia n c e

P o r t a b i l it y

/

M

Fa ult tolerance

R eso urce

M a i n t a i n a b i li t y

i H

A c c u r a c y

C o m p l ia n c e R e l i a b i li t y

h /

S u i t a b i l it y

H H

A n a l y z a b i l it y

H

C h a n g e a b i li t y

M

S t a b i l it y

L

T e s t a b i li t y

M

C o m p l ia n c e

H

A d a p t a b i li ty

H

I n s t a l l a b i l it y

L

C o - e x is t e n c e

H

R e p l a c e a b i l it y

M

C o m p l ia n c e

H



Quality measurement tables Q u a l it y i n u s e m e a s u r e m e n t c a te g o r y

C H A R A C T E R IS T IC

R E Q U I R E D

M E A S U R E S

A S S E S S M E N T A C T U A L

L E V E L

R E S U L T

Effectiveness Productivity Safety Satisfaction

Internal quality m easu rem ent

E x t e r n a l q u a lit y m e a s u r e m e n t c a t e g o r y

C H A R A C T E R I S T IC

S U B C H A R A C T E R I S T IC

M E A S U R E S

R E Q U I R E D L E V E L

A S S E S S M E N T

C H A R A C T E R IS T IC

M E A S U R E S

R E Q U IR E D L E V E L

A S S E S S M E N T A C T U A L R E S U L T

R E S U L T S u i t a b i l it y A c c u r a c y

Su itability

I n t e r o p e r a b i l i ty

Accu ra cy

Security

I n t e r o p e r a b i li t y Sec urity

S U B C H A R A C T E R I S T IC

A C T U A L F u n c t i o n a l i ty

Functionality

c a te g o r y

C o m p l ia n c e Re liability

Ma turity

C o m p l ia n c e R e l i a b i l it y

Fault tolerance

M aturity

R e c o v e r a b i l it y ( d a t a , p r o c e s s ,

F a u l t t o le r a n c e R e c o v e r a b i li ty ( d a t a , p r o c e s s ,

Us ability

C o m p l ia n c e U s a b i l it y

U n d e r s t a n d a b i l it y

C o m p l ia n c e

Learnability

U n d e r s t a n d a b i l i ty

O p e r a b i l it y A t t r a c ti v e n e s s

L e a r n a b i l it y

C o m p l ia n c e

O p e r a b i l i ty Attractiveness

Efficiency

T i m e b e h a v i ou r

C o m p l ia n c e R e s o u r c e u t i li s a t io n

E f f i c ie n c y

Time

behaviour C o m p l ia n c e

R e s o u r c e u t il is a t i o n M a i n t a i n a b i l it y

C o m p l ia n c e

A n a l y z a b i l it y C h a n g e a b i l i ty S t a b i li t y

M a i n t a in a b i l i ty

A n a l y z a b i l i ty

T e s t a b i l i ty

C h a n g e a b i l i ty S t a b i l it y

Porta bility

C o m p l ia n c e Portability

A d a p t a b i l it y

T e s t a b i li t y

Instability

C o m p l ia n c e

Co-existence

A d a p t a b i l i ty

R e p l a c e a b i li ty

I n s t a b i l it y

C o m p l ia n c e

Co-existence R e p l a c e a b i li t y C o m p l ia n c e



Measurement plan implications S U B C H A R A C T E R I

D E L IV E R A B L E S

S T I C

T O

B E

IN T E R N A L M E A S U R E S T O B E

E V A L U A T E D 1 . S u i t a b i li t y

2. S atisfaction

E X T E R N A L M E A S U R E S

A P P L IE D

T O

Q U A L IT Y IN B E

A P P L IE D

1 .

1 .

1 .

2 .

2 .

2 .

3 .

3 .

3 .

1 .

( N o t A p p l ic a b l e )

( N o t A p p l ic a b l e )

U S E

M E A S U R E S T O B E A P P L IE D ( N o t A p p lic a b l e )

1.

2 .

2 .

3 .

3 .

3 . 4 . 5 . 6 .



Measurement Primitive A measure, either a base measure or a derived measure, that is commonly used for deriving internal quality measures, external quality measures and quality in use measures. No definition (25000) measure collected during Software Product Lifecycle from which Internal, External and Quality in Use Measures are derived. (25020 and 25021) Single value of measurement primitive generally does not indicate the quality of the measured entity. NOTE The Quality is measured afterwards by calculating the Quality Measures. (25020 and 25021)



Measurement Primitives in SPQM-RM



Consideration views

Considerations Measurement Concept Base Derived

Measurement Scale Type

Measurement Focus

Nominal Ordinal Interval Ratio Absolute

Internal External

Subjec tive Objec tive

Q in use

Measurement

Measurement Method Type

primitive



Measurement Primitives Hyperlink Format example C o n s id e r a tio n

V ie w s

M e a s u re m e n t co n c e p t B a s e D e r iv e d M e a s u r e m e n t s c a le N o m in a l : R a t io A b s o lu t e M e a s u re m e n t fo c u s Internal E x te r n a l Q u a lit y in U s e M e a s u re m e n t m e th o d

M e a s u r e m e n t P r im it i v e s

Q u a l ity M e a s u r e s

: N u m b e r o f f u n c ti o n s : : T a s k t im e : : T im e o r n u m b e r o f t a s k s :

: P h y s i c a l a c c e s s i b i li t y : : W a i tin g t im e : : T a s k e ff ic i e n c y :

O b j e c t iv e S u b je c tiv e






Relationship between Needs and Requirements Stakeholders’ Needs in their Minds

Collected and Identified Stakeholders’

Stated, Implied or Unaware Needs

(Business) Needs

Selected and Specified s Requirements

Functional Requirements

External Quality Requirements

Functional Design & Internal Quality Requirements

Non Functional Design & Internal Quality Requirements



ISO/IEC 15288 System Life Cycle Processes to appear in 25030 ISO/IEC 25030 Quality requirements

Elicit and define quality requirements Formalise identified quality requirements Identify & formalise internal quality requirements

Stakeholder requirements definition process Requirements analysis process Architectural design process Implementation process Integration process Verification process Transition process Validation process Operation process Maintenance process

SQuaRE

ISO/IEC 15288 (Technical Processes)



Evaluation process view according to ISO/IEC 14598-1 E s ta b l i s h evaluation re q u i re m e n t s

E s t a b l i s h p u r p o s e o f e v a lu a ti o n I d e n tif y ty p e s o f p r o d u c t( s ) S p e c if y q u a li t y m o d e l

S e l e c t m e t r ic s S p e c if y t h e evaluation

E s t a b l i s h r a t i n g l e v e l s f o r m e t r ic s E s t a b lis h c r i te ria f o r a s s e s s m e n t

D es ig n th e evalu ation

9 1 2 6 -1

Q u a li t y C h a r a c t e r is t ic s

9 11 42 56 9 -6 2- 6E Ex tvear lnu aa lt i o n ri MM e otrrdi u c sl e s 19 1 2 6 8 - 3- 6I nEt ve ar nl ua al t i o M M eo tdr u i c lse s 9 1 2 6 - 4 Q u a l it y in u s e M e t r ic s 1 4 5 9 8 - 6 E v a l u a t io n M o d u le s

P r o d u c e e v a lu a t i o n p l a n

M e a s u r e c h a r a c t e r i s t ic s E x e cu te th e evalu ation

C o m p a r e w it h c r ite r i a A s s e s s r e s u l ts



Summary - 1 SQuaRE series addresses requirements and evaluation of software product quality • Internal, External, and Quality in Use Measures are the link between requirements and evaluation Software Product Quality Measurement Reference Model is a specific instance of 15939 information model Additional standards to create catalogues of measures for quality attributes and measurement primitives Need for validated measures to populate catalogues page 34 © 2004 by Carnegie Mellon University

Software Quality Requirements & Evaluation - IsO 25000

Recommend Documents