Deloitte - KPI and Measuring Security

“You Can’t Manage It If You Can’t Measure It ISACA March 2006

Agenda • Do you y ou know how w ell your y our information security program is working?

• Key Performance Performa nce Indicator Indicato r (KP I) Perfor mance Index (KP X) • Key Performance

• I nformation Collection Collect ion • Examples • Summary

© Deloitt Deloitte e & Touc Touche he LLP and affiliated affiliated entities.

W hat do w e have hav e to be w orri orried ed about? abo ut?

The time between the discovery of a vulnerability and the potential exploit is diminishing from months to days if not hours


I T Security Security Governance Governance Maturi ty Model

• The Maturity Model is sponsored by the I T Governance Gov ernance Institute organizati on’s practices practice s • I t is used to rank the maturity of an organization’s and standards against industry best practices and standards

• I t can be used to help guide an organization on the areas that w ill improve their overall information security posture post ure © Deloitt Deloitte e & Touc Touche he LLP and affiliated affiliated entities.

How do you know if you have ha ve an a n information securit s ecurity y progr am that effectively effectiv ely manages risks? • Obtain a high score on an I SO 17799 assessment? • Comple te regular, active penetration tests w ith no discovered vulnerabilities? reporte d using • Have an acceptably low # of security incidents reported the I ncident Response process?

• Have an effective effec tive virus v irus program (few or no infections and any infections are managed effectively effectiv ely w ith little interruption)? Service Level Expectations (SLE) that are • Have Measu rable Service consisten tly being achieved?

• Have an effective I DS program ( # and type of alerts are being managed effec tively, little impact on the busi ness, in line o r better than industry benchmarks)?

• Obtain certification against an information security reference standard standar d ( I SO 27001)? © Deloitt Deloitte e & Touc Touche he LLP and affiliated affiliated entities.

There are several sev eral problems to avoid w hen establishing establishi ng an information security measurement program • Lack of management commitment t oo soon • Measuring too much, too

• Measu ring too little, too late • Measuring the w rong things definit ions • I mprecise metrics definitions

• Using metrics data to evaluate individuals • Using metric s to motivate, rather than to understand • Collecting data that is no t used • Lack of commu nication and training • Misinterpreting metrics data © Deloitt Deloitte e & Touc Touche he LLP and affiliated affiliated entities.

Key Performance Performanc e I ndic ndicato ators rs (KP I s) can help determine the current status of the inform ation security program • A key performan ce indicator is a measure of a particular organization al performance activity, or an importan t indicator of a precise health condition o f an organization

• Used as an indication of the current state of a component of the busines b usiness s to take t ake the “surprise” “surpr ise” out of risk effectiv e, the KP I must be defined as succinctly as • To be effective, possible me asured ed as an “improvement “improv ement”” from a know n state sta te or • Can be measur a reference standard


A Key P erform ance I ndicator . . . • Must be som ething that can be measured and continued to be measured

• Mu st be precise, meaning ful and understandable relevant ant to the business • Must be relev

• May be required require d by legislation legi slation and/ or Regulations • Must have a m easurement index that has meaning • Mu st have an appropriate life (Stickiness) • Should be tied to the organ ization’s vision and strategy


Types of Key P erformance I ndicators (KP I s) Threshold shold – w hen an index i ndex rea reaches ches set tar targets gets or fal falls ls into • Thre set ranges – e. e.g. g.,, ETS scores on defin ed ris ks

• Milest Milestone one – w hen a speci s pecific fic condit c ondition ion is reac r eached hed – e.g. e.g.,, certi fica tio n

Quantitativ ative e – measure of value (num ber, time, $, etc. etc.)) • Quantit e.g.,, num ber of repo rted securi ty inciden ts, lost tim e due to – e.g. viruses

• Qualit Qualitativ ative e – measure of accept acceptability ability or health – e. e.g., g., survey r ating s, ratin g of r isks


Examples of Key P erformance Indicators • Awareness standards s and procedures (surveys and • Know ledge of policies, standard tests)

• Risk Assessment • Depth and breadth of regular risk assessments across the enterpri se (W hen w as the last assessment? Qualit enterprise Qualitativ ative e measure of the ris ks, risk index)

• Risk Management • Number of incidents reported, amount of loss incurred, number of situations managed

• Audit • Noted deficiencies against the policy and standar standards ds ( measured year over year) Certification • Benchm arks and Certification • Mainta Maintaining ining// foll follow ow ing I T security secur ity certi ce rtifica fications tions such suc h as FI P S 140-1, I SO 27001, ISO 15408 (Comm on Criteria)


P ossible Non-Risk Key P erformance I ndicators (KP I s) • People – Training & Certifications – Competence Turnover

• Technology – Currency – Cost management – Compliance / licensi licensing ng

• Investment – Trends per area

Effective tiveness ness & Return on I nvestment • Effec – Key Risk I ndicator experience vs. cost

• Productivity – Missed Deadlines


KP I s can be used to measure the Effect Ef fectivene iveness ss of I nvest nvestment ment (EOI (EOI ) securi ty is • A Return on I nvestment (ROI) for information security difficult to measu re since risk, and especially risk redu ction, is challenging to quantify in terms of dollars.

• The Effectivene Effe ctiveness ss of I nvestment (EOI) could be the comparison of the effectiveness effectiv eness of the security measures w ith the value of the investment.

• For example, the number and impact of viruses and w orms can be compared w ith the investment in virus vir us detection technology and support programs. E OI for • A collection of KPI s could be used to measure the EOI information security


A Key P erf erformanc ormance e I ndex (KP X) is a summary or or correlation correla tion of one or more KP I s that provides prov ides an indication of the overall perform ance of a defined area of the security program directi on in • May prompt the organization to change strategic direction information security

• Levels may be triggered by a variety of factors understanda ble • Must be m eaningful and understandable relevant ant to the business • Must be relev

• Must have a m easurement index that has meaning • Mu st have an appropriate life (Stickiness) and • Should be tied to the organ ization’s vision and strategy


Example Exa mple KPI Forma Formatt KPI Name

Short name or title for the KPI

Description

Description Descr iption of of the KPI – what does does it address? address?

Objective

What are are the objecti objectives ves of the KPI KPI – what is it measur measuring? ing? Why is it important?

Stakeholder

Who is this KPI relevant to?

Type

__ Quantitative ___ Qualitative ___ Milestone ___ Threshold

Effort

__ Low

__ Medium

__ High

Unit// De Unit Dept pt

What does it apply to?

Method

Method used to measure the KPI

Tools

Any potential tools used to support the measurement and reporting process?

Frequency Comments

___ Day ___ Week ___ Month ___ Quarter ___ Year ___ Year+ Any additional information or comments? Is this a requirement from legislation or regulations?


Example Exa mple Key P erfo erformanc rmance e I ndic ndicat ator or (KP I ) KPI Name

Weekly Reported Security Incidents

Description

Provides a relative index on the current number of reported security incidents/events at differing security levels for the recent reporting week

Objective

A measure of the relative size and effectiveness of the organizations risk management processes

Stakeholder

CSIO, CIO, Operations Management, Technology Management

Type

_X_ Quantitative

Effort

__ Low

___ Qualitative

_X_ Medium

___ Milestone ___ Threshold

__ High


Information Security

Method

Count number of reported security incidents/events at low, medium and high severity over the past week

Tools

IDS and/or security management/reporting software

Frequency Comments

___ Day

_X_ Week _X_ Month

_X_ Quarter

_X_ Year

___ Year+

Need to have confidence in the detection and reporting mechanisms to be able to measure measure changes to the index over time. A lower index will then mean less risk © Deloitt Deloitte e & Touc Touche he LLP and affiliated affiliated entities.

Example Exa mple Key Performance Performanc e I ndex (KP X) KPI Name

Information Security Risk Management Index

Description

Provides a relative index on the current number of reported security incidents/event incidents /events s at differing security security levels within within a specified time frame

Objective

A measure of the relative size and effectiveness of the organizations risk management processes

Stakeholder

CSIO,CIO

Type

_X_ Quantitative

Effort

__ Low

___ Qualitative

_X_ Medium

___ Milestone ___ Threshold

__ High


Core Systems

Method

Count number of reported security incidents/events at low, medium and high severity over a defined time frame

Tools

IDS and/or security management/reporting software

Frequency Comments

___ Day

_X_ Week _X_ Month

_X_ Quarter

_X_ Year

___ Year+

Need to have confidence in the detection and reporting mechanisms to be able to measure measure changes to the index over time. A lower index will then mean less risk © Deloitt Deloitte e & Touc Touche he LLP and affiliated affiliated entities.

Several automated tools can provide a view v iew of security incidents and trends


Secur ecurity ity Incidents Incident s - Adva Advanced nced Forensi For ensic c Tools


The I nformation Securit S ecurity y P rogram should include a reporting m echanism that provides a single point of reference for concise, executiveexecutive-level level inform ation for business and technology t echnology ow ners. Sample Security Dashboard Operator Event View

Geographic Threat View

Incident Tracking (Ticketing System)

Trend View

Advanced Forensic Tools

Geographical Dashboard View Reports

The dashboard aims to transform data from operations to actionable informatio n for decision makers © Deloitt Deloitte e & Touc Touche he LLP and affiliated affiliated entities.

An analy analysis sis of security incidents incide nts w ill contribute to the current status of the I nform ation Security Security Pr ogram


Keep track of each area of concern that is the ob ject of a KP I or KP X defi definiti nition on Topic Topi c - > Vision/ Visi on/ Missi Mission on Objective

What is the Vision and Mission statement that directs IT security? What is the the main objec objective tive – how is it it measured? measured? – Why is it import important? ant?

Key Control Objectives and Controls

What are the key control objectives and controls that should be in place for the organization? The controls should be based on international reference standards

Measurements

What are the measurements that may be available to report on this area?

KPI(s)

What Key Performance Indicators(s) should be defined for this objective?

KPX(s)

What summary index(s) can be defined that is a high-level representation of one or more KPIs that are vitally important to the organization?

Map KPI(s) to Performance Goals

How does the KPI(s) map to the individual performance goals?

Reporting

Any required acknowledgement or reporting for this KPI?

Comments

Any additional information or comments?


An example KPI for I nappr nappropria opriate te Use KPI KPI - 1 Number of verified instances of inappropriate use over a set time period. (weekly or by reporting period)

Inappropi Inappropirate rate Use - KPX The impact of recorded inappropriate use events compared to the amount of IT security awareness training per person.

Measurem Measurement ent - 1 Number of inappropriate use cases opened and verified

KPI KPI - 2 Impact of inappropriate use events to the business in terms of resources and or loss over time (weekly or by reporting time)

Measurem Measurement ent - 2 Amount of service lost to inappropriate use

KPI KPI - 3 Number of verified inappropriate use events compared with the number of IT security awareness training days per person compared over time

Presentation Name (View / Header and Footer)

Measurement -3 Number of IT security awareness training days


An example KP X for In appropriat appropriate e Use

KPX


An example KP I for In trusion Detec D etection tion Measuremen Measurementt - 1 Number of incidents of intrusions detected and reported KPI KPI - 1 Average amount of Loss (productivity time) per intrusion within a set time period (weekly or per reporting period).

IDS KPX The measureable amount of productivity loss attributed to intrusions in relation to the the number of events and the cost of the IDS program.

KPI KPI - 2 Number of events caught and prevented by the IDS within a set time period

KPI - 3 Number of IDS program failures

KPI - 4 Cost of the IDS program in relation to the number and impact of detected events

Measuremen Measurementt - 3 Amount of downtime or productivity loss caused by intrusion incidents.

Measureme Measurement nt - 2 Number of incidents of intrusions impacting the organization that were not reported

Measureme Measurement nt - 4 The number of systems with active monitoring capabilities

Measureme Measurement nt - 5 Number of Sensors per network segment

Measureme Measurement nt - 6 Cost of the hardware and/or software to implement intrusion detection sensors


An example KP X for Threat Management– I ntrusion Detection System System (I DS DS)) 1

Number of Resolved Major and Catastrophic Incidents Over Time

# of Resolved Major and Catastrophic Incidents

Number of Major and Catastrophic Incidents Over Tim T ime e

High Risk Incidents

# of Major and Catastrophic Incidents

Time/ Reporti Reporti ng Period

3

2

Critical Incidents

Time/ Reporting Period

Average Time to Resolve a Number of Major and and Catastrophic Catastrophic Incid Incidents ents

>4<10 hrs/month/ system productivity loss

# of Resolved Major and Catastrophic Incidents

>10hrs/month/ system productivity loss Major Incidents Catastrophic Incidents

Average Aver age Time to Resolve Major a nd Catastrophic Incidents

Number of Resolved Major and Catastrophic Incidents I ncidents


Summary prov ide • A good collection of Key Performance I ndicators w ill provide an overview overview of the current status st atus of risk management w ithin the organizat organization ion – Use the collection collection of KP I s as an information security dashboard

legislat ive or • The KP I s can be used to help comply w ith legislative regulatory requirements – P rovide the information that can be used for reporting purposes

sele cted and defined to be useful • The KP I s must be carefully selected – Must be meaningful and measurable

Effective ctive KPI s can be used to demonstrate good management • Effe of risk – For example, KPI s may provide a financial institution the ability to reduce the percentage of reserve required to offset operational risk defined define d by the Basel II Accord


Questions? Glen Bruce, [email protected]

© Deloit Deloitte te & Touch Touche e LLP and affiliated entities. entities. Deloitte, one of Canada's leading professional services firms, provides audit, tax, consulting, and financial advisory services through more than 6,100 people in 47 offices. Deloitte operates in Québec as Samson Bélair/Deloitte & Touche s.e.n.c.r.l. s.e.n.c.r.l. The firm is dedicated to helping i ts clients and its people excel. Deloitte is the Canadian member firm of Deloitte Touche Tohmatsu. Deloitte refers to one or more o f Deloitte Touche Tohmatsu, a Swiss Verein, its member firms, and their respective subsidiaries and affiliates. As a Swiss Verein (association), neither Deloitte Touche Tohmatsu nor any of its member member firms has any liability for each other's acts or omissions. Each of the member firms is a separate and independent legal entity operating under the names "Deloitte," "Deloitte & Touche," "Deloitte Touche Tohmatsu," Tohmatsu," or other related names. Services are provided by the member firms or their subsidiaries or affiliates and not by the Deloitte Touche Tohmatsu Verein.

Member of Deloitte Touche Tohmatsu

Deloitte - KPI and Measuring Security

Recommend Documents