Innovation in Performance Measurement Trends and Research Implications

JMAR Volume Ten 1998

Innovations in Performance Measurement: Trends and Research Implications Christopher D. Ittner David F. Larcker University of Pennsylvania Abstract: The objective of this paper is to foster research on recent innovations in performance measurement by providing a rich description of emerging measurement practices and suggesting directions for future research. Using survey data coiiected by consuiting firms and government organizations, we examine three measurement trends: (1) economic vaiue measures, (2) nonfinancial performance measures and the balanced scorecard, and (3) performance measurement initiatives in government agencies. Existing research on these topics is reviewed and research opportunities are highlighted.

The choice of performance measures is one of the most critical challenges facing organizations. Performance measurement systems play a key role in developing strategic plans, evaluating the achievement of organizational objectives, and compensating managers. Yet many managers feel that traditional accounting-based measurement systems no longer adequately fulfill these functions. A 1996 survey by the Institute of Management Accounting (IMA) found that only 15 percent of the respondents' measurement systems supported top management's business objectives well, while 43 percent were less than adequate or poor. In response, firms increasingly are Implementing new performance measurement systems to overcome these limitations. Sixty percent of the IMA respondents, for example, reported they were undertaking a major overhaul or planning to replace their performance measurement systems. The perceived inadequacies in traditional accounting-based performance measures have motivated a variety of performance measurement innovations ranging from "improved" financial metrics such as "economic value" measures to "balanced scorecards" of integrated financial and nonfinancial measures. However, despite increasing adoption of these performance measurement innovations, relatively few studies have examined the new measures' economic relevance, the implementation issues arising from their adoption, or the performance consequences from their use. The objective of this paper is to foster research on these topics by: (1) providing a rich description of emerging performance measurement practices, (2) synthesizing current research on the use and performance consequences of the new measures, and (3) suggesting directions for future research. Financial support was provided by Ernst & Young LLP and KPMG Peat Marwick LLP. The convnents o/Madhav RaJan and the research assistance oJTim Meyer are greatly appreciated. We are especially indebted to AT&T, Ernst & Young, Sibson & Co., Towers Perrin, and the United States General Accounting Office Jor providing data reported in this paper.

206

Journal of Management Accounting Research, 1998

The remainder of the paper is organized into five sections. The next section reviews emerging trends in performance measurement practices. The following sections discuss three topics that have dominated recent performance measurement discussions: (1) economic value measures, (2) nonfinancial measures and the balanced scorecard, and (3) performance measurement initiatives in government agencies. The final section offers our conclusions.

TRENDS IN PERFORMANCE MEASUREMENT Most economic theories analyzing the choice of performance measures indicate that performance measurement and reward systems should incorporate any financial or nonfinancial measure that provides incremental information on managerial effort (subject to its cost).^ Despite these models, firms traditionally have relied almost exclusively on financial measures such as budgets, profits, accounting returns and stock returns for measuring performance (Balkcom et al. 1997). Many firms now believe that the heavy emphasis placed on financial measures is inconsistent with their relative importance. Wm. Schiemann and Associates surveyed 203 executives in 1996 on the quality, uses and perceived importance of various financial and nonfinancial performance measures (Lingle and Schiemann 1996). Their results are presented in table 1. While 82 percent of the respondents valued financial information highly, more than 90 percent clearly defined financial measures in each performance area, included these measures in regular management reviews, and linked compensation to financial performance. In contrast, 85 percent valued customer satisfaction information highly, but only 76 percent included satisfaction measures in management reviews, just 48 percent clearly defined customer satisfaction for each performance area or used these measures for driving organizational change, and only 37 percent linked compensation to customer satisfaction. Similar disparities exist for measures of operating efficiency, employee performance, community and environment, and innovation and change. More importantly, most executives had little confidence in any of their measures, with only 61 percent willing to bet their jobs on the quality of their financial performance information and only 41 percent on the quality of operating efficiency indicators, the highest rated nonfinancial measure. Perceived inadequacies in traditional performance measurement systems have led many organizations to place greater emphasis not only on nonfinancial measures, but also on "improved" financial measures. The increased emphasis on performance measures of all kinds is refiected in data from Ernst & Young's 1991 International Quality Study (IQS 1991) of 584 businesses in four countries (Canada, Germany, Japan and the United States) and four industries (automobile, banking, computer and health care). Table 2 lists the importance of various financial and nonfinancial process improvement, strategic planning, and compensation measures in 1988 and 1991, and their expected importance in 1994. The process improvement responses indicate that each of the measures, including reduced costs, increased in importance over time. Whereas reduced cost ' See Holmstrom (1979). Banker and Datar (1989) and Feltham and Xle (1994).

Ittner and Larcker

CO
O) CD

u C

V

CO

ID

* j

Is

•—•

O

ill CJ

0 ttc

e

o -S CO

3 O

4H

-M

o o

4H

-M

C C o4i (U V 3

F^

li

(U U

Ul

u u o

CO

.D

,9 cu a, C/}

207

208


TABLE 2 Changes in Performance Measures Used for Evaluating Process Improvements, Strategic Planning, and Compensation 1991 Ernst & Young Survey of Manufacturing and Service Firms iu Four Countries* 1988

1991

1994 (Expected)

ANOVA F-statistic

Importance of the following measures for evaluating process improvements:" Reduced costs Reduced errors Reduced cycle time Less process variation Fewer customer complaints Importance of the following Customer satisfaction

3.01 2.77 2.49 2.18

3.31 3.22 2.91 2.74

3.52 3.55 3.32 3.31

61.07*** 144.35*** 126.74*** 208.03***

2.92 3.32 3.64 112.11*** measures in the strategic-planning process:** 2.59

3.13

3.67

233.21***

Importance of the following assessment criteria for compensation:** Senior management Quality Stock appreciation Profit or cash flow Market share or other positional measure

2.22 1.86 2.91

2.62 2.00 3.15

3.14 2.15 3.27

137.99*** 5.91*** 19.29***

2.31

2.53

2.81

33.64***

Quality Stock appreciation Proflt or cash flow Market share or other posltionai measure

2.19 1.45 2.48

2.55 1.49 2.68

3.13 1.69 2.93

133.16*** 7.63*** 26.73***

1.97

2.14

2.49

37.71***

Quality Stock appreciation Proflt or cash flow Market share or other posltionai measure

2.03 1.18 1.84

2.29 1.20 1.96

2.85 1.35 2.27

86.69*** 6.84*** 23.78***

1.54

1.64

1.96

27.82***

Middle management

*** Responses across years are statistically different at the 1 percent level. ''The International Quality Study (IQS) was a joint survey of 584 organizations by Ernst & Young and the American Quality Foundation. The survey was conducted In the automobile, computer, banking and health care industries In Canada, Germany, Japan and the United States. •"Deflnitlon of scales: 1 = slight or not at all (trivial or no concern at all), 2 = secondary (less important, but not trivial), 3 = major (Important, together with others), 4 = primary (dominant).

Ittner arvi Larcker

209

was the most important process measure in 1988, it ranked lower than the number of customer complaints in 1991 and was expected to rank below customer complaints and reduced errors by 1994. Similarly, the Importance of customer satisfaction measures in strategic planning increased significantly from 1988 to 1991, and was expected to increase even further by 1994. Compensation assessment criteria exhibit similar patterns. The use of nonfinancial measures such as customer satisfaction and market share became significantly more important in compensation decisions at all organizational levels. However, financial measures such as stock appreciation, profits and cash flows also became more important, refiecting an overall increase in pay-for-performance as well as greater use of nonfinancial measures. Additional analysis of the IQS data (not reported) Indicates that these trends are not limited to manufacturing or North American firms. In each industry and country, customer satisfaction measures became increasingly important for strategic planning, and nonfinancial measures such as reductions in customer complaints and process variability played a greater role in assessing process improvements. With the exception of stock appreciation in German businesses, western organizations reported greater use of each of the compensation criteria at all organizational levels. Industry-specific tests also found significant increases in the importance placed on nearly all of the compensation measures. Japanese firms, however, reported few changes in compensation criteria; the only significant difference over time (p < 0.10, two-tail) was greater use of quality measures for senior management compensation. The increased emphasis on both financial and nonfinancial measures is consistent with two trends that have dominated recent performance measurement discussions: (1) the addition of "new" financial measures that are claimed to overcome some of the limitations of traditional financial performance measures, and (2) greater emphasis on "forward-looking" nonfinancial measures such as customer satisfaction, employee satisfaction and defect rates. We review these trends in the following sections. "ECONOMIC VALUE" MEASURES While traditional accounting measures such as earnings per share and return on investment are the most common performance measures, they have been criticized for not taking into consideration the cost of capital and for being unduly infiuenced by external reporting rules. Consulting firms are promoting a variety of "economic value" measures to overcome these limitations. The foundations for these "new" performance measures are residual income and internal rate of return concepts developed in the 1950s and 1960s. Stern Stewart & Co.'s (hereafter Stern Stewart) trademarked "Economic Value Added" or EVA® measure, for example, is the firm's proprietary adaptation of residual income. EVA® is defined as adjusted operating income minus a capital charge, and assumes that a manager's actions only add economic value when the resulting profits exceed the cost of capital. To eliminate perceived distortions created by external accounting rules. Stern Stewart recommends up to 160 adjustments that firms can make to their accounting systems to more closely approximate

210


"economic" profits. Common adjustments to compute EVA® include modifications to the deferred income t£ix reserve, the LIFO reserve, the treatment of intangible assets such as research and development and advertising, and goodwill amortization (see Stewart [1991, 113-117], for other recommended adjustments). A second economic value measure that has received considerable attention is "Cash Flow Return on Investment" (CFROI) and its variants. CFROI essentially is the long-term internal rate of return, calculated by dividing infiation-adjusted cash fiow by the infiation-adjusted cash investment (Snyder 1995). Advocates of CFROI argue that this metric is vastly superior to traditional accounting measures and EVA® as a performance measure. In an article on the "metric wars" between consulting firms pushing various economic value measures, a partner at HOLT Value Associates claimed, "CFROIs are ideally suited to displaying long-term track records, whereas a Stern Stewart-tj^ie EVA is in millions of dollars, heavily infiuenced by asset size, and unadjusted for infiation-induced biases" (Myers 1996, 41). Responded Stern Stewart co-founder G. Bennet Stewart III, "CFROI is literally a consultant's concoction. It was quite an imaginative development by a consulting firm, but it is not well grounded in the basic elements of corporate finance theory. CFROI attempts to measure shareholder wealth—which is not clearly related to maximizing shareholder wealth" (Myers 1996, 42). A number of impressive claims have been made for each of the economic value measures. Stern Stewart, for example, cites in-house research indicating that "EVA® stands well out from the crowd as the single best measure of wealth creation on a contemporaneous basis" (Stewart 1991, 75), while Dixon and Hedley (1993) of Braxton Associates cite an internal study showing their CFROI measure explains 91 percent of the variation in market capitalization ratios. Claims such as these have prompted a growing number of firms to adopt various forms of economic value measures. A 1996 survey by the Institute of Management Accountants (IMA 1996) found that 35 percent of the respondents used EVA® or similar measures (up from 18 percent in 1995) and 45 percent expected to use them in the future (up from 27 percent in 1995). Yet, despite the increasing emphasis on these measures, research on the extent to which they are superior to traditional accounting measures is limited and mixed. The Association Between Economic Value Measures and Stock Returns Most studies to date have examined claims that EVA® is a better predictor of stock returns than traditional accounting measures. Milunovich and Tseui's (1996) examination of the computer server industry found market-value added between 1990 and 1995 more highly correlated with EVA® than with earnings per share, earnings per share growth, return on equity, free cash fiow, or free cash growth. Lehn and Makhija (1997) also found that stock returns over a ten-year period were more highly correlated with average EVA® over the period than with average ROA, ROS, or ROE. In addition, EVA® performed somewhat better than accounting profits in predicting CEO turnover. O'Byrne (1996) examined the association between market value and two performance measures: EVA® and net operating profit after tax (NOPAT). He found that both measures had similar

Ittner and Larcker

211

explanatory power when no control variables were included in the regression models, but that a modified EVA® model had greater explanatory power when indicator variables for 57 industries and the log of capital for each firm were included as additional explanatory variables. However, O'Byrne (1996) did not make similar adjustments to the NOPAT model, making it impossible to compare results using the different measures. Other studies suggest that EVA® is predictive of stock returns, but is not the only performance measure that ties directly to a stock's intrinsic value, one of the primary claims of EVA® advocates (e.g., Stewart 1991). Bacidore et al. (1997) compared EVA® to "Refined Economic Value Added" (REVA), which applies the cost of capital to the opening market (rather than book) value of the firm's equity plus debt. Although both measures were statistically related to abnormal stock returns, REVA outperformed EVA® in both regression and portfolio tests. Chen and Dodd (1997) examined the explanatory power of accounting measures (earnings per share, ROA and ROE), residual income, and various EVA®-related measures. They found that EVA® measures outperformed accounting earnings in explaining stock returns, but the associations were not as strong as suggested by EVA® proponents (maximum R^ = 41.5 percent). In addition, accounting earnings provided significant incremental explanatory power above EVA®, leading the authors to conclude that firms should not follow EVA® advocates' prescription to replace traditional accounting measures completely with EVA®. Finally, residual income provided nearly identical results to EVA®, without the need for the accounting adjustments advocated by Stern Stewart. Biddle et al. (1998) provide the most comprehensive study of EVA®'s value relevance to date. Their analyses examined the power of accounting measures (earnings and operating profits) to explain stock market returns relative to EVA® and five components of EVA® (cash fiow from operations, operating accruals, after-tcix interest expense, capital charge, and accounting adjustments). They found that traditional accounting measures generally outperformed EVA® in explaining stock prices. While capital charges and Stern Stewart's adjustments for accounting "distortions" had some incremental explanatory power over traditional accounting measures, the contribution from these variables was not economically significant. Sensitivity analyses indicated that these results were robust to Stern Stewart's grouping of firms into five "types" based on their past operating returns and growth rates, the time period examined, and the dependent variable used in the tests (i.e., stock returns or levels or the time frame used to compute the market measures). Managerial Implications of Economic Value Measures From a managerial accounting standpoint, the key question is not whether economic value measures are more highly correlated with stock returns than traditional accounting measures, but whether the use of economic value measures for internal decision-making, performance measurement, and compensation purposes improves organizational performance. Wallace's (1998a) examination of relative performance changes in 40 adopters of residual income-based measures such as EVA® and a matched sample of non-users supports claims that these measures change managerial behavior. Compared to the control firms, the residual

212


income firms decreased new investments, increased payouts to shareholders through share repurchases, eind utilized assets more intensively, leading to significantly greater change in residual income.^ Wallace (1998a) also found weak evidence that stock market participants responded favorably to the adoption of residual income-based compensation plans. A related issue is whether the performance implications of economic value measures depend upon how the measures are used within the organization. Stern Stewart argues that effective implementation of EVA® requires firms to make this measure the cornerstone of a total financial management system that focuses on EVA® for capital budgeting, goal setting, investor communication, and compensation (Stern et al. 1995). Stewart (1995) asserts that the poor results from many EVA® implementations are attributable to the fact that EVA® use has not become pervasive throughout the organization, especially for compensation decisions. A survey by Sibson & Co. supports claims that many users of economic value measures do not base compensation on these measures. As shown in table 3, 41.2 percent of respondents used economic value measures for business planning and financial management purposes. However, only 16.7 percent used these measures in incentive plans, of which only 26.3 percent made economic value the sole performance measure in these plans. In addition, many of the respondents used economic value measures only in annual incentive plans and not in long-term plans, and relatively few used them at all organizational levels. Biddle et al. (1998) provide some evidence that the use of economic value measures in compensation plans is associated with the measures' effectiveness. The only subsample in which EVA® outperformed traditional accounting measures in predicting stock returns was firms using EVA® in compensation plans. Similarly, the Sibson & Co. survey in table 3 indicates that 26.3 percent of firms using economic value measures in incentive plans reported that these measures were "very successful" and 36.8 percent reported they were "marginally successful." None of the respondents stated that the measures were "not successful." The 31.5 percent of respondents who were "not sure" of the measures' effectiveness were all recent adopters. Wallace's (1998b) survey of EVA® users found that firms including EVA® in their incentive compensation plans also implemented the measure to a significantly greater degree for capital budgeting and dividend decisions, but not for asset disposal, working capital management, share repurchase, or financing decisions. Firms using EVA® in incentive plans also reported significantly greater awareness of the cost of capital, reduced average accounts receivable age, increased use of debt, increased sales revenues and a longer accounts payable cycle. However, changes in the Wallace (1998a) found no significant difference between users of EVA® and users of other residual income-based measures. Wallace's (1998a) tests did not examine whether the residual Income firms achieved higher accounting and stock performance levels than the control sample after adopting the new measures. Although the change in performance for firms adopting residual Income measures was greater than the control sample, the study did not examine whether the performance of the residual income firms remained below the average performance levels of the control sample after the improvements.

Ittner and Larcker

213

TABLE 3 Uses of Economic Value Measures in Business Planning and Incentive Plans: 1995 Survey of 114 Firms by Sibson & Co. Economic value used in business planning/financial management* Economic value used as a performance measure in incentive plans If economic value is used in incentive plans: Used in annual incentive plans only Used in long-term incentive plans only Used in annual and long-term incentive plans Used in corporate and business unit management plans only Used in corporate management plans only Used in business unit management plans only Used in corporate, business unit, and small group plans Economic value is the sole performance measure in incentive plans Effectiveness of economic value as a measure in the incentive plan: Very successful Moderately successful Not successful Not sure No response

41.2% 16.7 42.1 10.5 47.4 52.6 5.3 31.6 10.5 26.3 26.3 36.8 0.0 31.5 5.3

survey defined economic value as cash flow or earnings above the cost of capital or discounted cash fiow, including measures such as economic value added, cash fiow return-on-investment, economic profit, or residual profit.

degree of selectivity in the choice of new investment projects, inventory turnover, share repurchases and debt repayment were not statistically different in the two EVA® user groups. Research Topics Claims regarding the superiority of economic value measures, as well as the limited academic research on the topic, suggest a number of avenues for future studies. Perhaps the most important question is the longterm performance benefits from the adoption of economic value measures. Consistent with Wallace's (1998a) event study results, some stock market analysts are now taking the implementation of internal EVA® systems into consideration when recommending companies. BT Alex. Brown, for example, has put a "strong buy" rating on JC Penney, in part because the firm is installing an EVA® system {Director's Alert 1998).^ Future research can determine whether the market's expectation of higher stock returns from EVA® adopters is accurate. Second, considerable debate exists on the relative value relevance of the alternative economic value measures. Consulting firms battle over the superiority of their economic value measures, charging that competitors' measures have fiaws that compromise their predictive ability (Myers 1996; Although many analysts In the United States and United Kingdom are using EVA® to evaluate firms. Its use Is not universally embraced. Merrill Lynch, for example, has attacked EVA®, claiming that they have found little evidence that the measure relates to enhanced shareholder value (Director's Alert 1998).

214


The Economist 1996). However, the relative ability of different economic value measures (EVA®, CFROI, or variants of these measures) to predict stock returns is unknown.* If, as might be expected, one measure does not consistently exhibit superior predictability, researchers can attempt to determine the factors explaining cross-sectional differences in the predictive ability of alternative economic value measures. Structural and environmental variables such as firm strategy, competitive environment, and product or industry life cycle, for example, are likely to be important determinants of the relative explanatory power of different economic value measures, as well as the explanatory power of traditional accounting measures. Third, no evidence exists on the factors infiuencing the adoption and performance consequences of economic value measures for internal purposes. Advocates of economic value measures such as EVA® suggest that these measures may not be applicable in industries such as financial services (which are required to set aside capital for regulatory reasons), or in very young companies where revenue calculations frequently are subject to guesswork [The Economist 1996). Similarly, our discussions with EVA®. adopters indicate that economic value measures can be problematic for firms in highly cyclical industries, where exogenous factors may cause EVA® to be negative in many periods, even though managers have taken the appropriate actions. Stern Stewart argues that effective implementation of EVA® also requires their measure to become the cornerstone of a total financial management system (Stern et al. 1995). The firm attributes the lack of success in many EVA® implementations to four factors: (1) EVA® is not made a way of life; (2) EVA® is implemented too fast, (3) lack of conviction by the CEO or division head, and (4) inadequate training (Stewart 1995). Braxton Associates, in turn, states that a value-based system using CFROI will be unsuccessful without a value driver analysis that unbundles CFROI into discrete, controllable financial and operational variables (Snyder 1995). These discussions suggest a number of testable hypotheses regarding determinants of the measures' effectiveness. The internal benefits from economic value measures may also vary with the chosen metric. Critics of CFROI, for example, argue that this measure is too complex for managers to understand and act upon, even if it is conceptually superior to traditional accounting measures and EVA® (Birchard 1994; Myers 1996). EVA® has also been criticized for being too complex for front-line managers to use, for motivating managers to reduce beneficial capital expenditures to improve short-term EVA®, and for ignoring the firm's "core competencies" and providing little actionable information on the long-term drivers of firm value (Birchard 1994; Hamel 1997). Research examining (1) key implementation issues infiuencing the success or failure of various economic value measures, and (2) the extent •* An mternal study by Monsanto using data from competitors, suppliers, customers and other companies in the St. Louis area found that the correlation with stock market performance over a 20-year period was substantially higher using CFROI than using EVA® (Myers 1996). However, a comprehensive analysis of this issue using contemporary capital market research methods has not been conducted.

Ittner and Larcker

215

to which the alternative metrics produce distinctly different functional and dysfunctional managerial behavior can shed light on the factors affecting the effectiveness of economic value measures. A growing number of firms have tried and abandoned one or more of the economic value measures (Birchard 1994; Myers 1996), providing a natural opportunity to study differences between successful and unsuccessful adopters of these innovations. For example, AT&T was once touted in the business press as a leading proponent of EVA® (Tully 1993), but has since abandoned this measure. Table 4 describes the evolution in AT&T's use of EVA® in its senior management performance measurement system. Prior to an internal reorganization in the early 1990s, performance was evaluated based on measured operating income and measured operating units (a service volume indicator), and no variable compensation was awarded. In 1992, the firm adopted EVA® for decision-making and compensation purposes, and implemented an EVA®-based bonus plan covering approximately 110,000 employees. However, EVA® was supplemented by two new nonfinancial measures ("customer value added" [CVA] and "people value added" [PVAp) within two years, and was abandoned altogether by 1997 in favor of traditional accounting measures. Our interviews with EVA® implementers and users in AT&T and its offspring (Lucent Technologies and NCR) identified three primary reasons for the measure's demise. 1. The measure was too complex for most employees to understand, even though AT&T made relatively few of the accounting adjustments recommended by Stern Stewart. Despite extensive training in the computation and use of EVA®, employees outside of corporate headquarters did not understand how their actions affected EVA® results, and felt they had limited ability to impact corporate or business unit EVA® targets. In 1995, NCR was the first AT&T unit to abandon EVA® due to its perceived complexity, choosing to focus on ROA and contribution margin. Lucent's new management team also chose not to continue EVA® measurement after the unit's spin-off, feeling that EVA® was similar but more complex than traditional accounting measures such as ROA. 2. The company came to recognize that EVA® was an historical measure that provided incomplete information on key drivers of future performance, such as employees and customers. In addition, the company's efforts to win the Malcolm Baldrige National Quality Award required customer-related measures to become a major component of their performance measurement system. As a result, EVA® was supplemented by the CVA and PVA measures, which continue to be included in managers' individual performance goals. 3. Although internal EVA® results were positive, total shareholder return between December 1992 and December 1996 (the period covered by "People value added" (PVA) was based on employee satisfaction, work force diversity, employee turnover / retention, work force health, and leadership visibility. "Customer value added" (CVA) was based on customers' satisfaction with price and with product, service and contact quality.


216

00

a

0)

o

o

u

l •

in

I

•

S

a

cs cS

in

W CU O • • •

•

•

n tii

(£1

oo

o

o o

o

g

in

I 2

0)

s^

I

o o o

c

S< cs cs "^

in

I

u (0

oa

•c

CO

u

cs

a

®I

I

Ittner and Larcker

217

the EVA® measure) was -6.46 percent.^ The inconsistencies between reported EVA® and stock returns, combined with the hiring of a new CEO who had not championed the EVA® system, led the firm to drop EVA® in favor of traditional accounting performance measures they believe to be more closely aligned with analysts' forecast models and shareholder value. Although far from definitive, the AT&T example suggests several explanations for EVA® implementation results. More detailed comparative studies using matched samples of successful and unsuccessful implementations can help determine the extent to which these and other explanations account for the effectiveness of economic value measures. NONFINANCIAL PERFORMANCE MEASURES While some firms are attempting to overcome perceived limitations in traditional accounting-based performance measures using economicvalue metrics, others are embracing the use of nonfinancial measures for decisionmaking and performance evaluation. In particular, many firms are implementing "balanced scorecard" systems that supplement traditional accounting measures with nonfinancial measures focused on at least three other perspectives —customers, internal business processes, and learning and growth (Kaplan and Norton 1992, 1996). Proponents of the balanced scorecard contend that this approach provides a powerful means for translating a firm's vision and strategy into a tool that effectively communicates strategic intent and motivates performance against established strategic goals. Case studies by Fisher (1995) and Brancato (1995) have identified three principal reasons firms are adopting nonfinancial measures. 1. Perceived Limitations in Traditional Accounting-Based Measures. Companies believed that, relative to key nonfinancial indicators, traditional accounting measures (1) are too historical and "backward-looking," (2) lack predictive ability to explain future performance, (3) reward shortterm or incorrect behavior, (4) are not actionable, providing little information on root causes or solutions to problems, (5) do not capture key business changes until it is too late, (6) are too aggregated and summarized to guide managerial action, (7) refiect functions, not cross-functional processes, within a company, and (8) give inadequate consideration to difficult to quantify "intangible" assets such as intellectual capital.'' By incorporating nonfinancial Indicators into their measurement systems, many firms sought to create a wider set of measures that capture not only firm value, but also the factors leading to the creation of value in the business. 2. Competitive Pressure. Many firms experienced a perceived shock to their operating environments that motivated meinagement to find new ways of managing, measuring and controlling operations. The substantial changes in the nature and intensity of competition forced ^ Over the same period, total shareholder returns were 16.44 percent for MCI and 112.46 percent for Sprint, AT&Ts two primary competitors for long-distance telephone services. ^ For additional discussions on nonfinandeil measures of Intangible assets and intellectual capital, see Edvinsson and Malone (1997) and Stewart (1997).

218


firms to determine and measure the nonfinancial "value drivers" leading to success in the new competitive environment. The greater emphasis placed on nonfinancial measures in firms facing competitive pressure is consistent with research finding positive associations between perceived environmental uncertainty and the demand for broadbased information systems incorporating nonfinancial indicators (e.g., Chenhall and Morris 1986). 3. Outgrowth of Other Initiatives. Other firms adopted nonfinancial measures as an outgrowth of improvement initiatives that required new performance indicators, especially the adoption of total quality management (TQM) programs. Many management accounting researchers argue that effective TQM requires timely, detailed process information for identifjang the sources of defects and monitoring the consequences of subsequent improvement activities — information that typically is not available from aggregate accounting data (e.g., Kaplan 1983; Johnson 1992). The quality management literature also maintains that TQM requires greater emphasis on customer requirements and customers' satisfaction with the firm's products or services, leading to greater emphasis on nonfinancial customer measures such as complaints, satisfaction and retention (e.g., U.S. Department of Commerce 1997). While some case study sites limited the adoption of nonfinancial measures to quality-related issues, others looked upon the need for new quality measures as an opportunity for a more extensive overhaul of their measurement processes (Brancato 1995). Current Research Several related research streams have examined various issues raised by these case studies. One stream focuses on claims that nonfinancial measures are "leading" indicators that provide information on future performance that is not contained in current accounting measures. Although the performance measurement literature claims that predictive ability is one of the primary benefits of nonfinancial measures, studies indicate that firms experience considerable difficulty linking these measures to future accounting or stock price performance. Brancato (1995), for example, reported that none of her case study participants could precisely quantify the link between key nonfinancial performance measures and the bottom line. Our survey of vice presidents of quality for major U.S. firms found similar problems relating quality and customer satisfaction measures to accounting and stock returns. As shown in table 5, 75 percent of the senior quality executives felt pressure to demonstrate the financial consequences of their quality initiatives, but fewer than 55 percent could directly relate their quality measures to operational, productivity, or revenue improvements, only 29 percent to accounting returns, and just 12 percent to stock returns. Similarly, only 28 percent could link customer satisfaction measures to accounting returns and 27 percent to stock returns. As a result, 52 percent of the executives found it difficult to identify the quality improvement opportunities offering the highest economic returns, and none found this to be an easy task. Consistent with the survey evidence, studies investigating the link between nonfinancial measures and future financial performance have produced mixed results. The majority of these studies have examined the

Ittner and Larcker

219 If) CS ;.;

CS

00

5 cs

o

rt

cs

M

cs

CO

cs CO

00

CO

05

_o in

cr XI

c o c s i n c ' j h ^ c o ^ ^ o o f ^ t ^ - ^ ( N C S C O - H C O ' - i C S C S - H

CJ

cs u ,o

- H C S O c s t ^ o o i n c N o o N M C S C O C O - H

c

g

^

r

t

c

S

I CO

ber

i n c s i n i n o o o c j ) C D o ) c s cs M -H cs r-< cs

CD

a

I )wevei

05

•a

cS

cr o -J2 "c

(N ai

il

220


association between customer satisfaction measures and subsequent accounting or stock returns. Banker et al. (1998), for example, found positive associations between customer satisfaction measures and future accounting performance in 18 hotels managed by a hospitality firm, while Ittner and Larcker (1996) provided evidence that hedge portfolios formed on the basis of customer satisfaction measures outperformed the stock market in subsequent periods. Anderson et al. (1994, 1997) supported the hypothesis that, on average, customer satisfaction in 77 Swedish firms was positively related to contemporaneous accounting return on investment, but found weaker or negative relations in service firms. Similarly, Ittner and Larcker's (1998) investigation of customer, business unit and firmlevel data supported claims that customer satisfaction measures are leading indicators of future customer purchase behavior (retention, revenue and revenue growth), growth in customers, changes in business unit accounting performance, cind current market values. However, the firm-level results varied by industry, with positive relations in some industries and negative or insignificant relations in others. Foster and Gupta's (1997) investigation of the association between satisfaction measures for individual customers of a wholesale beverage distributor and current or future customer profitability also found positive, negative, or insignificant relations depending upon the questions included in the satisfaction measures or the model specification (levels or percentage changes). A second research stream has emphasized the use and performance consequences of nonfinancial measures in organizations adopting TQM or other advanced manufacturing practices. Nearly all of these studies have found positive associations between the emphasis placed on TQM, justin-time (JIT) production practices, or manufacturing fiexibility and the provision of nonfinancial measures such defect rates, on-time delivery, and machine utilization (e.g., Daniel and Reitsperger 1991a, 1991b; Banker et al. 1993; Abernethy and Lillis 1995; Perera et al. 1997). Positive associations have also been found between TQM and the use of nonfinancial measures in reward systems (Ittner and Larcker 1995, 1997; Daniel et al. 1995). However, empirical support for the hypothesized performance benefits from these measurement practices is marginal at best. Young and Selto (1993) found little evidence that the provision of nonfinancial operational measures to workers in a JIT facility was associated with differences in manufacturing performance or workgroup performance ratings. Ittner and Larcker (1995) found that information and reward systems that placed greater emphasis on nonfinancial Information were associated with higher ROA in organizations making relatively little use of formal TQM practices, but not in organizations with extensive TQM programs. Furthermore, the performance consequences of quality-oriented performance measures varied somewhat across the automotive and computer industries, suggesting that the use of these measures must be adjusted to refiect the firm's production and competitive environment (Ittner and Larcker 1997). Symons and Jacobs' (1995) examination of a TQM-based reward system indicated that the introduction of the system was associated with higher production output, lower scrap, and reduced product variability, but did not investigate the incentive plan's effect on costs or profitability. Abernethy and Lillis' (1995) study of management control systems in fiexible manufacturing plants Implied that greater reliance on nonfinancial

Ittner and Larcker

221

manufacturing measures had a greater positive effect on perceived performance in fiexible firms than in nonfiexible firms. However, a similar study in plants following a "customer-focused" manufacturing strategy found no association between the use of nonfinancial measures and perceived performance (Perera et al. 1997). A third research stream has examined the use of nonfinancial measures in compensation plans. Thirty-six percent of firms placing explicit weights on performance measures in their CEO bonus plans in 1993 and 1994 included nonfinancial measures (Ittner, Larcker, and Rajan 1997). The mean weight on nonfinancial performance in these plans was 37.1 percent of the bonus award, with the average firm including three performance measures in the bonus formula (two nonfinancial and one financial).^ Significant determinants of the weight placed on nonfinancial measures included the extent to which the firm followed an innovationoriented or "prospector" strategy, the adoption of strategic quality initiatives, the length of product development and product life cycles, regulation, and "noise" in traditional financial measures (Bushman et al. 1996; Ittner, Larcker, and Rajan 1997). Although these studies suggest that the desirability of including nonfinancial measures in compensation contracts is contingent on a variety of factors, little evidence exists on the relation between the "match" between incentive plan performance measures and firm characteristics and the performance benefits arising from the use of nonfinancial measures. In one of the few studies examining this issue, Govindarajan and Gupta (1985) found that the benefits from nonfinancial compensation criteria are contingent on a business unit's strategy, with greater reliance on long-run nonfinancial criteria (sales growth, market share, new product and market development, research and development, personal development, and political/public affairs) having a stronger positive impact in units following a "build" strategy than in those following a "harvest" strategy. Surprisingly little research has been conducted on the implementation or performance consequences of the balanced scorecard concept, despite widespread practitioner interest in the subject. Table 6 provides descriptive statistics from a survey of balanced scorecard implementations by the consulting firm Towers Perrin. Balanced scorecard adopters continue to place the majority of weight on financial measures (mean = 56 percent), followed by customer measures (19 percent) and internal process measures (12 percent). Corporate and division-level measures are most common, with relatively little weight placed on subsidiary or other (e.g., department or team) measures. Although Kaplan and Norton (1996) argue that the proper role of the balanced scorecard in determining compensation is not yet clear, 70 percent of respondents already base compensation on the balanced scorecard or some variant that incorporates financial and nonfinancial measures, and 17 percent are actively considering using their scorecards for this purpose. An additional 15 percent use their scorecards for evaluating performance but not for compensation purposes. For example, Chrysler's 1994 Incentive contract determined executives' annual bonuses based 40 percent on vehicle quality Improvements (warranty repairs per 100 vehicles sold), 20 percent on customer satisfaction, 20 percent on market share and 20 percent on financial performance (pre-tcix earnings).

222


TABLES Balaneed Scorecard Implementations in 60 Firms Responding to a 1996 Survey by Towers Perrin Relative weight placed on the foUowing perspectives (mean response): Financial 56% Customer 19 Internal business 12 Innovation and learning 5 Other 9 Types of performance measures under the balanced scorecard approach (check all that apply): Corporate 57 Division/group 65 Subsidiary 22 Other (e.g., department, team, etc.) 22 Uses of the balanced scorecard approach in compensation: Used in incentive compensation awards 37 Used for evaluating performance, but not for determining 15 incentive compensation awards Not used for Incentive compensation, but use a variation that 33 considers both flnancial and operational measures The same scorecard is used for everyone in the plan 23 Top executives are Included In the balanced scorecard plan 37 Problems experienced implementing the balanced scorecard (n = 57): Not a Problem 1 Difficult to evaluate relative importance of measures Time and expense involved Requires quantification of quaiitatlve data Large number of measures dilute overall impact Dlfflcult to decompose goals for lower levels in organization Requires a highly-developed Information system

2

3

4

Major Problem S

2% 7 7

25% 35% 29% 25 43 20 18 30 36

9% 5 9

9

23

25

36

7

12

18

36

25

9

13

18

25

35

9

Effectiveness of the balanced scorecard compared to performance measurement approaches used in the past:

Employee understanding of performance measures and goals (n = 40) Satisfaction or value received (n = 39)

Significantly Lower 1

0% 0

2

As Effective 3

4

18%

45%

32%

31

54

5

Significantly Higher 5

5% 10

Ittner and Larcker

223

Scorecards generally are modified for individual managers, with top executives included in 37 percent of the scorecard incentive plans. Kaplan and Norton (1996) contend that the balanced scorecard provides a number of mechanisms for linking long-term strategic objectives with short-term actions. According to these authors, development of the balanced scorecard forces managers to develop a consensus around the firm's vision and strategy, and allows managers to communicate the firm's strategy throughout the organization. This communication ensures that employees understand the long-term strategy, the relations among the various strategic objectives, and the association between the employees' actions and the chosen strategic goals. The balanced scorecard is also expected to help firms allocate resources and set priorities based on the initiatives' contribution to long-term strategic objectives, and to provide strategic feedback and promote learning through the monitoring of shortterm strategic results. Empirical support for these claims is limited. Although 64 percent of the Towers Perrin respondents reported that the satisfaction or value received from their balanced scorecard systems was higher or significantly higher than that received from other performance measurement approaches, only 37 percent felt that employees' understanding of performance measures and goals was higher under the scorecard than under other approaches and 18 percent thought it was lower. Similarly, Ittner, Larcker, and Meyer's (1997) study of a balanced scorecard compensation system in retail branch banks found no evidence that the scorecard approach enhanced branch managers' understanding of business goals, plans for meeting these goals, or connections between the managers' job and business objectives. Moreover, the perceived adequacy of information about progress against the multiple business goals was statistically lower. Research Topics The primary research question arising from the use of nonfinancial measures and the balanced scorecard is the net economic benefits from these measurement practices. Despite increasing use of nonfinancial measures, many firms believe that performance measures should be purely financial in order to focus efforts on the ultimate goals of the firm (Newman 1991; Kurtzman 1997). Similarly, some EVA® advocates claim that a balanced scorecard of financial and nonfinancial measures hinders performance because there is no single overall measure of performance on which managers can concentrate their efforts to improve [Journal qf Applied Corporate Finance 1997, 65). The implementation of more complex measurement systems can also be quite costly. As shown In table 6, 25 percent of the respondents to the Towers Perrin survey experienced problems or major problems with the extra time and expense required to implement and operate the balanced scorecard, and 44 percent encountered problems developing the extensive information systems needed to support the scorecard approach. Combined with the weak performance results in prior studies of nonfinancial performance measures, these Issues raise important questions about the net benefits from incorporating nonfinancial metrics into performance measurement systems. If nonfinancial performance measures are not beneficial in all settings, an important research topic is identifying the circumstances under which

224


these systems do improve performance. As highlighted in our literature review, prior studies suggest that the link between nonfinancial measures such as customer satisfaction and subsequent accounting and stock market performance vary across industries. Similarly, the use and performance consequences of these measures appear to be affected by organizational strategies and the structural and environmental factors confronting the organization. Future research can make a significant contribution by providing evidence on the contingency variables affecting the predictive ability, adoption and performance consequences of various nonfinancial measures and balanced scorecards. Another key issue is defining precisely what "balance" is and the mechanisms through which "balance" promotes performance. A common view, perpetuated by early writings on the balanced scorecard concept (e.g., Kaplan and Norton 1992), is that "balance" is achieved by diverse measurement in the domains of financial performance, operational performance, performance for the customer, and learning and innovation. According to this view, multiple measures in each of several domains minimize the risk that information germane to business results will be lost. More recently, Kaplan and Norton (1996) argue that a balanced scorecard is not merely a collection of financial and nonfinancial measures in various categories, but an integrated set of measures developed from a "theory of the business" that explicitly links the scorecard metrics in a causal chain of performance drivers and outcomes. This view, originally outlined by Eccles (1991), contends that a firm's "business model" must be understood before a "balanced" scorecard of performance measures can be chosen and implemented. Figure 1 illustrates the business model developed by Sears Roebuck and Company (Rucci et al. 1998). The basic model consists of employee, customer and shareholder components. After considerable data collection and statistical refinement. Sears identified several key value drivers (e.g., increases in employee attitude have a direct impact on customer impression, and customer impression has a direct impact on the future accounting performance of individual stores). Sears claims that customer satisfaction increased 4 percent after incorporating the results from this model into the choice of quality/customer initiatives and the design of their longterm performance plan.^ The increase in customer satisfaction led to an estimated $200 million increase in revenues, and ultimately an estimated $250 million increase in market capitalization (based on their current after-tax margins and price-earnings ratio). The business model approach to the selection of performance measures raises a number of potential research questions. Although establishing the firm's business model prior to selecting measures has the advantage of sharpening strategic focus and organizational priorities, it can One Interesting question is whether statistical relations between the measures and desired outcomes (e.g., stock price or accounting performance) should be used when designing performance evaluation and compensation plans. Analytical research by GJesdal (1981), Paul (1992) and Feltham and Xle (1994) shows that an information system that is useful for valuing the firm need not be useful in assessing a manager's performance. Consequently, just because one economic value measure or nonfinancial Indicator predicts stock returns or accounting performance better than an alternative measure does not necessarily imply that the same measures or weights should be used to evaluate and reward managers.

Ittner and Larcker

225

ro ro T3

ro c

CD CO

o

E «

V

\

u (Q / /

JO

O

cf o ro

E <"

0)

"o a. o

o

loyee ntion

X

\ \

o 3

£.. S^

s o->

^£ 55 o

lUOC

1I r1 J

/

/

c a> TJ

0) (0 (0 0) (0

o

>.

.5 5

o

a

E c

c a.

3 F i n 0)

E

rvey 998).

u S

u

CO

ro

3 CO

•—-

"c "TO CD CO

o

Q. O

ucc

0>

01 3 (0 >

incr yee tti

idl

E

loyee avior

O

(0

elp

rvi

u

o

I

M 0) C

epn

m

C/}

QC

o c

E

o

ro CD

©

1-

CD

ro

T3

226


be difficult to establish the reliability and predictive validity of the multiple measures in the business model without having done a great deal of measurement and analysis in the first place.'" Moreover, there is no guarantee that a business model based on current measures and competitive environments will be relevant to the choice of performance measures if there are major shifts in the firm's environment. Future research can provide guidance as to which conception of "balance"—variety in measurement or selection of a smaller set of measures based on their current reliability and predictive validity (which may not hold in the future)—best promotes desired business outcomes. It would also be instructive to study the development and use of business models in a diverse set of organizations to determine whether the claimed success at Sears can be replicated in other settings, and to investigate how business models vary across organizational life cycles, competitive environments, corporate strategies, and other contextual factors. Nagar's (1998) study of retail banks, for example, found that the nonfinancial drivers of future accounting performance vary significantly with the bank's competitive strategy. It may also be possible to develop more general business models for estimating the economic impact of quality, customer, or other improvement initiatives. For example, research by Sterman et al. (1997) on the performance consequences of TQM at Analog Devices is a novel attempt to use a detailed business model to understand complex interrelations among a broad set of financial and nonfinancial measures. Although this research requires extensive data collection and detailed institutional knowledge, the resulting insights can challenge existing assumptions regarding the relations between performance measures. The use of multiple financial and nonfinancial measures also leads to questions on the value of including a broad set of metrics in performance measurement systems. Studies on "information overload" suggest that a large number of measures can reduce performance by exceeding managers' processing capabilities when making judgements. ^^ A diverse set of performance measures may also cause managers to spread their efforts over too many objectives, reducing the effectiveness of the performance measurement system. More than 40 percent of the Towers Perrin respondents stated that the large number of measures in the balanced scorecard diluted the overall impact of the new measurement systems. Similarly, a study by the Consortium for Alternative Reward Strategies Research found that the performance benefits from reward plans for nonmanagement employees peaked when the plans used 3-5 performance measures and declined thereafter (McAdams and Hawk 1994). Holmstrom and Milgrom's '" Reliability refers to the extent of noise or measurement error in a measure, and validity to the extent to which the measure does what it is intended to do. Predictive validity is one of the key attributes of interest when selecting performance measures. From an accounting standpoint, a crucial test is whether a broad set of nonfinancial measures such as employee satisfaction, employee turnover, product development cycle time, and supplier relations possess incremental ability to predict future financial performance, after controlling for the predictability of past financial performance. " See Schlck et al. (1990) for a review of accounting studies on information overload. In contrast to the Information overload hypothesis, an experiment by Lipe and Salterio (1998) found that performance evaluations were not affected by increasing the number of measures when these measures were organized into the four balanced scorecard categories.

Ittner and Larcker

227

(1991) analytical model adds that multi-criteria incentive contracts may direct agents' effort to tasks that are easily measured at the expense of tasks that are harder to measure, even if this allocation of effort is detrimental to the firm. Their model indicates that the advantages of adding new performance measures to an incentive contract decrease with the difficulty of measuring performance in any other activities that make competing demands on the agent's time and attention. Thus, the net benefit from including a larger number of metrics in performance measurement systems is unclear. It is also unclear whether decisions using multi-criteria performance measurement systems should be computed using explicit, objective formulae that prescribe the weights to be attached to each measure, or should be based on subjective evaluations where the weight attached to each measure is implicitly or explicitly chosen by the decision maker. For example, many firms rank quality improvement opportunities using some quality measures denominated in financial terms (e.g., scrap dollars or warranty expenses), some in percentages or counts (e.g., defect rates or customer complaints), and some in arbitrary survey scale points (e.g., customer satisfaction indices).'^ As a result, managers must combine the various measures using pre-determined formulae or subjective weighting of the various measures' importance when selecting improvement projects. ^^ Similarly, performance evaluation and compensation decisions in multi-criteria systems can be based on formulaic or subjective weightings of the various measures. Kaplan and Norton (1996) highlight three potential difficulties in integrating the balanced scorecard measures into formula-based compensation plans. First, the firm must determine the appropriate weights to place on the multiple performance measures. The Towers Perrin survey suggests this is a difficult task in many organizations, with 38 percent of the respondents experiencing problems in evaluating the relative importance of the scorecard measures. Second, formulaic compensation plans may be susceptible to the game-plajring associated with explicit, formula-based rules. Finally, formula-based plans may allow bonuses to be paid even when performance is "unbalanced" (i.e., over-achievement on some objectives but under-achievement on others).** Some firms using multiple financial and nonfinancial performance measures for performance evaluations have abandoned formula-based '^ See Hemmer (1996) for a model providing insight into the conditions under which different tjTses of nonflnancial measures (e.g., levels vs. ratios) are appropriate. '^ For example, Xerox rates quality improvement projects using subjective assessments of four factors: (1) cost of poor quality reduction, (2) external customer impact, (3) ability to control the solution, and (4) degree of difficulty in resolving the problem. Each factor is rated on a scale ranging from 1 = little to 5 = great, and the average of the four subjective assessments is used to rank improvement alternatives. '•* Strategy researchers point out that the use of formal, pre-set goals and milestones in strategic control systems such as the balanced scorecard may also prevent the adaptability andflexibilitythat is the essence of good strategy (e.g., Quinn 1980; Mintzberg 1987). In addition, case studies by Lorange and Murphy (1984) and Goold and Quinn (1993) indicate that formal strategic control systems may reduce performance by focusing attention on incomplete or incorrect goals and performance measures, and fostering behavioral and political barriers that adversely affect the utility of the strategic controls.

228


plans in favor of subjective appraisals in order to minimize these difficulties. Indeed, Kaplan and Norton (1996, 220) argue that the balanced scorecard renders subjective compensation systems "easier and more defensible to administer..;and also less susceptible to game playing." Analytical studies indicate that subjective compensation plans Ccin be superior to objective, formula-based plans because they allow the firm to exploit noncontractable information that might otherwise be ignored in formulabased contracts. For example, Baiman and Rajan (1995) show that discretionary bonus schemes, in which an objectively-determined bonus pool is allocated to managers based on subjective evaluations of the managers' performance, enable the owner to use noncontractible information such as nonquantifiable or "soft" measures (e.g., the principal's personal observations of the manager's ability or effort level) to achieve an optimal improvement in managerial effort. Similarly, Baker et al.'s (1994) theoretical analysis indicates that the use of subjective weights on objective performance measures allows the employer to mitigate distortions in performance measures by "backing out" unintended dysfunctional behavior or gaming induced by the incomplete objective performance measures. Despite these advantages, subjective performance evaluations have a number of potential drawbacks. Prendegast and Topel's (1993) review article identifies several reasons why subjective performance evaluations may be inferior to objective, formula-based evaluations. These include greater possibility of reneging on promises to reward superior performance since the subjective measures are not verifiable, increased favoritism and bias in performance evaluations, the tendency to compress subjective evaluations and rewards (to avoid giving poor ratings), and greater perceived "unfairness" in performance evaluations. Should these possibilities materialize, workers will exert less effort under a subjective compensation system than under a more objective incentive plan.'^ Since the net benefits of formulaic vs. subjective performance evaluations are unclear, this issue offers an exciting topic for future research. Another important question is whether the same measures or scorecard used to develop strategic priorities and monitor strategic actions should be used to evaluate managerial performance. Although a large number of scorecard measures may be desirable for decision-making and performance monitoring purposes, a smaller number of selected performance measures may be more appropriate for managerial performance evaluation and compensation purposes. The balanced scorecard literature also suggests that performance measures should be tailored for each business unit. Thus, corporate-level measures may not be applicable to lowerlevel employees. More than a third of the respondents to the Towers Perrin survey, for example, found it difficult to decompose scorecard goals for lower-levels in the organization. Experiments by Schiff and Hoffmann (1996) provide some evidence that firms use different measures for assessing organizational and managerial performance. When presented with In addition, studies have found relatively low correlations between objective and subjective performance ratings, raising questions about which ratings are more accurate or appropriate. See Bommer et al. (1995) for a review.

Ittner and Larcker

229

a scorecard of financial measures and generally "softer" nonfinancial measures, participants tended to place greater emphasis on the financial measures for evaluating the performance of the business unit and on nonfinancial measures for evaluating managerial performance. Lipe and Salterio's (1998) experiments, on the other hand, found that when scorecards for two divisions contained some common and some unique measures, performance evaluations were affected only by the common measures. Thus, a potential avenue for research is examining the relative value of different types of scorecards and performance measures for different purposes. A final topic is the issue of trade-offs among multiple financial and nonfinancial performance measures. Although "balance" may require a manager to perform well on multiple dimensions, actions taken to improve one measure may lead to short-term declines in other performance measures. A key question is how to retain "balance" in managerial actions and performance evaluations in the presence of trade-offs. Kaplan and Norton (1996) suggest the use of hurdles to ensure that managers do not receive bonuses when they over-perform on some dimensions but under-perform on others. However, a potential problem with hurdles is that they may focus undue attention on dimensions requiring minimum performance levels and may prompt managers to avoid investments that reduce shortterm performance on the hurdle dimensions, even if these investments are beneficial in the long-term. Additional research is needed on the treatment of the inevitable trade-offs that managers will need to make among various financial and nonfinancial performance measures. PERFORMANCE MEASURMENT INITIATIVES IN GOVERNMENT While most academic discussions of performance measurement issues focus on the private sector, recent efforts to "reinvent" the government have emphasized the important role performance measurement systems can play in improving the efficiency and effectiveness of government operations. The Governmental Accounting Standards Board (GASB), for example, has promoted the reporting of "Service Efforts and Accomplishments" (SEA) by state and local governments. The objective of SEA reporting is to provide more complete Information about a government entity's performance than can be provided by traditional financial statements. According to GASB's Concept Statement No. 2, SEA information should focus on results-oriented measures of service accomplishments (outputs and outcomes) and measures of the relationships between service efforts and service accomplishments (i.e., efficiency), thereby assisting users in assessing the economy, efficiency and effectiveness of services. At the federal level, the Government Performance and Results Act (GPRA) seeks to hold federal agencies accountable for program results by requiring agencies to clarify their missions, set program goals, and measure performance towards those goals. Beglrming in 1999, agencies must (1) establish goals that define the level of performance to be achieved by a program activity; (2) express goals in an objective, quantifiable and measurable form; (3) describe the operational processes and resources required to achieve goals; (4) establish performance indicators to be used in

230


measuring or assessing the relevant outputs, service levels and outcomes of each activity; (5) provide a basis for comparing actual program results with established goals; and (6) describe the means for verifying and validating measured values (General Accounting Office 1998). Annual reporting of program performance begins the following year. The extent to which government organizations have adopted these performance measures varies widely. Table 7 reports descriptive statistics from a survey of 900 state and local government entities conducted by the Governmental Accounting Standards Board and the National Academy of Public Administration (1997). The survey examined the measurement and use of performance measures related to service efforts and accomplishments. Slightly more than half (53.2 percent) of the respondents have TABLE 7 Use and Reporting of Performance Measures in State and Local Government Entities (n = 900)' Percentage of entities that have developed: Performance measures Output or outcome performance measures

53.2 39.3

Percentage of entities that say they use: Performance measures Output or outcome performance measures

46.8 32.9

Percentage of entities that use measures for:

Strategic planning Resource ailocatlon Program management and monitoring

Output Measures

Outcome Measures

24.7 27.7 26.2

23.9 25.2 28.0

Output Measures

Outcome Measures

25.0 24.0 21.2

24.8 24.3 21.0

Percentage of entities that report measures to:

Internal management Elected officials Citizens and media 'ercentage of entities that plan to use measures for: Strategic planning Resource allocation Program management and monitoring Reporting

le to use) performance 46.3 58.9 62.9 41.9

Source: Adapted from Governmentai Accounting Standards Board and National Academy of Public Administration (1997). ''Entities refer to municipalities, counties, school districts, coileges and universities, pubiic authorities, state departments, pubilc employee retirement systems, and special districts.

Ittner and Larcker

231

developed these performance measures, with 46.8 percent using the measures internally. However, many of these measures are not output- or outcome-oriented. Only 39.3 percent have developed output or outcome measures, and just 32.9 percent use them for decision-making or performance evaluation. Less than 29 percent of the entities use output or outcome measures for strategic planning, resource allocation, or program planning and monitoring, and 25 percent or fewer use them for internal or external reporting. Although these percentages are relatively low, the use of performance measures for these purposes is expected to increase dramatically in the future, with 62.9 percent planning to use performance measures for program evaluation, 58.9 percent for resource allocation, 46.3 percent for strategic planning, and 41.9 percent for reporting. Financial World's assessments of municipal performance measurement and evaluation practices provide additional evidence on the use of results-oriented performance measures in local government. Since 1991, Financial World has issued report cards on the financial management practices of the 30 largest U.S. cities. Performance measurement systems are graded based on a number of factors, including the availability of clear, measurable goals, the extent to which performance measures relate to outcome rather than output measures, the measures' impact on decisions, the testing and benchmarking of performance measurement information, and the use of customer surveys. In 1991, the mean grade was C-I-, with two cities receiving As and one (Philadelphia) receiving an F (Barrett and Greene 1992). By 1995 (the latest year covered by the assessment), the mean grade improved slightly to B - , and no city received an F (Barrett and Greene 1995). More importantly, several cities made substantial improvements, with Philadelphia moving from F to C-I-, Memphis from D-lto B - , and Dallas from B to A-. In contrast, Seattle's grade fell from A to C-I-. Like the GASB survey results, the Financial World report cards suggest that state and local governments may provide a unique setting for examining major differences in performance measurement systems, as well as the behavioral and performance changes associated with dramatic changes in measurement practices. The development and use of results-oriented performance measures have also increased in the federal government, but remain relatively limited. Despite the approaching deadline for the implementation of resultsoriented measurement systems, the General Accounting Office (GAO) survey results in table 8 indicate that 38 percent or fewer federal managers had the tj^es of measures required by the GPRA to a "great" or "very great" extent in 1997. Moreover, only 21 percent used the measures for developing budgets, 20 percent for funding decisions, 13 percent as the basis for legislative changes, and 16 percent as the basis for program changes. Although these percentages are significantly greater than three years prior (p < 0.10, two-tail), they are far lower than the GAO expected (General Accounting Office 1998). In addition, relatively few of the managers believed that their agencies' efforts to implement the GPRA have had much effect on their programs, operations, or projects, and more than half thought that the GPRA's impact in the future will be modest. Research Topics One research topic prompted by the wide variations in governmental measurement practices is identification of factors infiuencing the adoption

232


TABLE 8 Performance Measurement Practices in the U.S. Government 1997 General Accounting Office Survey of Government Managers Percent of federal managers having these types of measures to a great or very great extent: Measures that: 3 Tears Ago Currently demonstrate to someone outside their agencies whether they are achieving their Intended results 19 32** tell whether they are satisfying their customers 11 32** teil about the quality of their product or services 19 31** tell how many things they produce or services they provide 27 38** tell If they are operating efficiently 17 26** Percent of federal managers using results-oriented performance measures for these purposes to a great or very great extent: Performance measures are: 3 Years Ago Currently used to develop tbe agency's budget 16 21** used as the basis for funding decisions 14 20** used as the basis for legislative changes 9 13* used as the basis for program changes 12 16** Percent of federal managers believing that their agency's efforts to implement the Government Performance and Results Act have improved its programs, operations, or projects to date (of those expressing an opinion): To a very great extent 1.2 To a great extent 9.1 To a moderate extent 31.5 To a small extent 36.3 To no extent 21.9 Percent of federal managers believing that their agency's efforts to implement the Government Performance and Results Act will improve its programs, operations, or projects iu the future (of those expressing an opinion): To a very great extent 1.6 To a great extent 25.3 To a moderate extent 34.1 To a small extent 14.4 To no extent 5.6 **, * statistically different from three years prior at the 5 percent and 10 percent leveis (two-taii), respectively.

of results-oriented performance measures. For example, nearly half of the GASB survey respondents with performance measures were not legally required to implement the measures. The survey also suggests that the adoption of performance measures varies across different types of government organizations (e.g., municipalities, counties, state departments, school districts, etc.). An interesting question is why some entities implemented these systems while others did not. Similarly, what factors have

Ittner and Larcker

233

motivated some of the cities in the Financial World assessment to make substantial improvements their performance evaluation systems? Some indication of the relevant factors may already be available from research on government accounting practices. Studies of financial accounting and disclosure in government agencies, for example, suggest that variables such as legislative power, the power of the governor or mayor, interestgroup strength, political competition, financing requirements, and socioeconomic development and diversity are associated with financial reporting practices (e.g., Evans and Patton 1983; Ingram 1984; Cheng 1992). Management accounting studies, in turn, suggest that the adoption and use of cost accounting systems in government agencies is related to the extent of internal and external competition and legislative requirements to be self-funding (e.g., Geiger and Ittner 1996). Performance measurement research can build on these studies to identify the determinants of governmental performance measurement practices. A second research question is whether the new performance measurement systems will actually improve governmental performance. There is a long history of unsuccessful management control initiatives in the U.S. government, ranging from management-by-objectives to zero-based budgeting. One avenue for research is examining how (or if) the implementation of the new performance measurement systems differs from earlier efforts, and how these differences affect performance outcomes. Another avenue is investigating how other management practices infiuence the new measures' use and performance benefits. The U.S. Comptroller General, for example, testified before Congress that greater flexibility and incentives for managers to act creatively are critical to the achievement of fundamental improvements in agency performance (U.S. Senate 1993). This claim suggests that efforts to improve government efficiency and effectiveness through improved performance measurement will be unsuccessful without complementary changes in other organizational practices. Perhaps the most fundamental question is whether private sector notions of performance measurement and accountability are applicable in the public sector. Many recent performance measurement initiatives are based on the idea that legal requirements to measure and report performance indicators will improve governmental performance by increasing the accountability of government managers. However, institutional theories argue that in organizations such as government agencies, whose survival depends primarily on the support of external constituents and only secondarily on actual performance, managers will implement the mandated systems in order to appear modern, rational and efficient, but will not actually use the systems for Improving performance (e.g., Scott 1987; Gupta et al. 1994). The large-scale implementation of the GPRA's requirements provides an ideal setting for examining whether mandated performance measurement systems are actually used for internal decision making and performance evaluation or are simply implemented to legitimate the agencies with Congress and other stakeholders. CONCLUSIONS The objective of this paper is to foster research on recent innovations in performance measurement by providing a rich description of emerging performance measurement practices and identifying fruitful avenues for

234


future research. Rather than providing a comprehensive review of performance measurement papers, we discuss a targeted selection of related papers that provide a broad overview of potential research opportunities related to economic value measures, nonfinancial performance measures and the balanced scorecard, and performance measurement initiatives in government agencies. Although the paper is limited to these three topics, many of the research questions are equally relevant to other important performance measurement issues. Including the measurement of interorganizational relations with suppliers and customers, the evaluation of joint ventures, and the choice of performance measures in "global" organizations. A final issue is the role of consultants in the adoption of new measurement practices. It would be useful to know whether economic value measures or the balanced scorecard represent solutions to real problems facing firms or are simply fads that consultants have packaged into "products" that can be easily sold to corporate management. The ultimate question is whether there is anything intrinsically superior about these measurement practices that produces desirable changes in managerial behavior. Perhaps the same (or better) results could be obtained with other performance metrics, as long as the performance measurement choice is championed by senior-level managers. In fact, it is possible that any performance consequences are simply due to a "Hawthorne Effect," with the specific measures having minimal importance. Additional insight into the role of consultants in the adoption of new measurement practices would contribute to our understanding of trends in performance measurement.

REFERENCES Abernethy, M. A., and A. M. Lillis. 1995. Tbe impact of manufacturing flexibility on management control system design. Accounting, Organizations and Society 20(4): 241-258. Anderson, E. W., C. Fornell, and D. R. Lehmann. 1994. Customer satisfaction, market share, and profitability: Findings from Sweden. Journal of Marketing Research 58 (July): 53-66. , , and R. T. Rust. 1997. Customer satisfaction, productivity, and profitability: Differences between goods and services. Marketing Science 16 (2): 129-145. Bacldore, J. M., J. A. Boqulst, T. T. Mllbourn, and A. V. Thakor. 1997. The search for tbe best financial performance measure. Financial Analysts Journal 53 (May/June): 11-20. Baiman, S., and M. V. Rajan. 1995. The Informationai advantages of discretionary bonus schemes. The Accounting Review 70: 557-579. Baker, G., R. Gibbons, and K. J. Murphy. 1994. Subjective performance measures In optimal incentive contracts. Quarterly Journal of Economics 109: 11251156. Balkcom, J. E., C. D. Ittner, and D. F. Larcker. 1997. Strategic performance measurement: Lessons learned and future directions. Journal of Strategic Performance Measurement 1 (2): 22-32. Banker, R., and S. Datar. 1989. Sensitivity, precision and linear aggregation of signals for performance evaluation. Journal of Accounting Research 27: 21-39.

Ittner and Larcker

235

, G. Potter, and R. G. Schroeder. 1993. Reporting manufacturing performance measures to workers: An empirical study. Journal of Management Accounting Research 5 Fall): 33-55. . . and D. Srinivasan. 1998. An empirical investigation of an incentive plan based on nonfinancial performance measures. Working paper. University of Texas at Dallas, Cornell University, and University of Pittsburgh. Barrett, K., and R. Greene. 1992. Tales of 30 cities. Financial World 161 (February 18): 28-47. , and . 1995. The state of the cities: 1995. Financial World 164 (March 14): 52-54. Biddle, G. C , R. M. Bowen, and J. S. Wallace. 1998. Evidence on the relative and incremental information content of EVA®, residual income, earnings and operating cash flow. Journal of Accounting and Economics (forthcoming). Birchard, B. 1994. Mastering the new metrics. CFO: The Magazine for Chief Financial Officers 10 (10): 30-38. Bommer, W. H., J. L. Johnson, G. A. Rich, P. M. Podsakoff, and S. B. MacKenzie. 1995. On the Interchangeability of objective and subjective measures of employee performance: A meta-analysis. Personnel Psychology 48: 587-605. Brancato, C. K. 1995. New performance measures—A research report. Report Number 1118-95-RR. New York, NY: The Conference Board. Bushman, R. M., R. J. Indejejikian, and A. Smith. 1996. CEO compensation: The role of individual performance evaluation. Journal of Accounting and Economics 21: 161-193. Chen, S., and J. L. Dodd. 1997. Economic value added (EVA®): An empirical examination of a new corporate performance measure. Journal of Managerial Issues 9 (3): 319-333. Cheng, R. H., 1992. An empirical analysis of theories on factors influencing state government accounting disclosure. Journal of Accounting and Public Policy 11: 5-42. Chenhall, R. H., and D. Morris. 1986. The impact of structure, environment, and interdependence on the perceived usefulness of management accounting systems. The Accounting Review 61 (1): 16-35. Daniel, S. J., and W. D. Reitsperger. 1991a. Linking quality strategy with management control systems: Empirical evidence from Japanese industry. Accounting, Organizations and Society 16 (7): 601-618. , and . 1991b. Management control systems for J.I.T.: An empirical comparison of Japan and the U.S. Journal of International Business Studies 22 (4): 603-617. . , and T. Gregson. 1995. Quality consciousness in Japanese and U.S. electronics manufacturers: An examination of the impact of quality strategy and management control systems on perceptions of the importance of quality to expected rewards. Management Accounting Research 6 (4): 367-382. Director's Alert. 1998. Merrill Lynch Attacks Economic Value Added®. (March 8). Dixon, P., and B. Hedley. 1993. Managing for Value. Boston, MA: Braxton Associates. Eccles, R. G. 1991. The performance measurement manifesto. Harvard Business Review 69 (2): 131-137. Edvinsson, L., and M. S. Malone. 1997. Intellectual Capital: Realizing Your Company's True Value by Finding its Hidden Brainpower. New York, NY: HarperBusiness. Ekonomist. 1996. A star to sail by? (August 2): 53-55. Evcins, J., and J. Patton. 1983. An economic analysis of participation in the municipal finance officers association certificate of conformance program. Journal qf Accounting and Economics 5 (2): 151-175.

236


Feltbam, G., and J. Xie. 1994. Performance measure congruity and diversity in multi-task principal / agent relations. The Accounting Review 69: 429-453. Fisher. 1995. Use of nonfinancial performance measures. In Readings in Management Accounting, edited by S.M. Young, 329-335. Englewood Cliffs, NJ: Prentice Hail. Foster, G., and M. Gupta. 1997. Tbe customer profitability Implications of customer satisfaction. Working paper, Stanford University and Washington University. Gelger, D. R., and C. D. Ittner. 1996. The Influence of funding source and legislative requirements on government cost accounting practices. Accounting, Organizations and Society 21 (6): 549-567. General Accounting Office. 1998. Managing for Results: The Statutory Framework for Performance-based Management and Accountability. Gaithersburg, MD: U.S. Generai Accounting Office. Gjesdai, F. 1981. Accounting for stewardship. Journal of Accounting Research 19: 208-231. Goold, M., and J. J. Qulnn. 1993. Strategic Control: Milestones for Long-term Performance. London, U.K.: Pitman Publishing. Governmental Accounting Standards Board and Nationai Academy of Public Administration. 1997. Report on Survey of State and Local Government Use and Reporting of Performance Measures—First Questionnaire Results. (September 30). Govlndarajan, V., and A. Gupta. 1985. Linking control systems to business unit strategy: Impact on performance. Accountiry, Organizations and Society 10 (1): 51-66. Gupta, P. P., M. W. Dlrsmith, and T. J. Fogarty. 1994. Coordination and control in a government agency: Contingency and Institutional perspectives on GAO audits. Administrative Science Quarterly 39 (2): 264-284. Hamel, G. 1997. How killers count. Fortune (June 23): 74. Hemmer, T. 1996. On the design and choice of "modern" management accounting measures. Journal of Management Accounting Research 8 (Fall): 87-116. Holmstrom, B. 1979. Moral hazard and observability. BeU Journal of Ek:onomics 10: 74-91. , and P. Mllgrom. 1991. Multitask principal-agent analyses: Incentive contracts, asset ownership, and job design. Journal of Law, Economics, & Organization 7: 24-52. Institute of Management Accountants (IMA). 1996. Are Corporate America's Financial Measurements Outdated? Montvale, NJ: IMA. Ingram, R. 1984. Economic Incentives and the choice of state government accounting practices. Journal of Accounting Research 22 (1): 126-134. IQS. 1991. International Quality Study: The definitive study of the best internationai quality management practices. Cleveland, OH: Ernst & Young and American Quality Foundation. Ittner, C D . , and D. F. Larcker. 1995. Total quality management and the choice of Information and reward systems. Journal of Accounting Research 33 (Supplement): 1-34. , and . 1996. Measuring the Impact of quality Initiatives onfirmfineindal performance. Advances in the Management of Organizational Quality 1:137. , and . 1997. Quality strategy, strategic control systems, Eind organizational performance. Accounting, Organizations and Society 22: 293-314. , , and M. Meyer. 1997. Performance, compensation, and the balanced scorecard. Working paper. University of Pennsylvania. , , and M. V. Rajan. 1997. Tbe choice of performance measures in annuai bonus contracts. The Accounting Review 72 (April): 231-255. , and . 1998. Are non-financiai measures leading indicators of financial performance? An analysis of customer satisfaction. Journal of Accounting Research (forthcoming).

Ittner and Larcker

237

Johnson, H. T. 1992. Relevance Regained: From Top-Down Control to Bottom-Up Empowerment. New York, NY: The Free Press. Journal of Applied Corporate Finance. 1997. The link between capital structure and shareholder value. Journal of Applied Corporate Finance 10 (2): 40-67. Kaplan, R. S. 1983. Measuring manufacturing performance: A new challenge for managerial accounting research. The Accounting Review (October): 686-705. , and D. P. Norton. 1992. The balanced scorecard—Measures that drive performance. Harvard Business Review 70 (1): 71-79. , cind . 1996. The Balanced Scorecard: Translating Strategy into Action. Boston, MA: Harvard Business School Press. Kurtzman, J. 1997. Is your company off course? Now you can find out why. Fortune 135 (February 17): 128-130. Lehn, K., and A. K. Makhija. 1997. EVA®, accounting profits, and CEO turnover: An empirical examination, 1985-1994. Journal qf Applied Corporate Finance 10 (Summer): 90-97. Lingle J. H., and W. A. Schiemann. 1996. From balanced scorecard to strategic gauges: Is measurement worth it? Management Review (March): 56-61. Lipe, M. G., and S. Salterio. 1998. The balanced scorecard: Judgmental effects of information volume, diversity, and organization. Working paper. University of Oklahoma and University of Alberta. Lorange, P., and D. Murphy. 1984. Considerations in implementing strategic control. Journal of Business Strategy (Fall): 27-35. McAdams, J. L., and E. J. Hawk. 1994. Organizational Performance & Rewards: 663 Experiences in Making the Link. New York, NY: American Compensation Association. Milunovich, S., and A. Tsuei. 1996. EVA® in the computer industry. Journal qf Applied Corporate Finance 9 (Spring): 104-115. Mintzberg, H. (1987). Crafting strategy. Harvard Business Review 65 (4): 66-75. Myers, R. 1996. The metric wars. CFO: The Magazine for Chief Financial Officers 12 (October): 41-50. Nagar, V. 1998. The information content of balanced scorecard measures: Evidence from the retail banking industry. Working paper. University of Michigan. Newman, G. 1991. The absolute measure of corporate excellence. Across the Board 28(10): 10-12. O'Byrne, S. F. 1996. EVA® and market value. Journal of Applied Corporate Finance 9 (Spring): 116-125. Paul, J. 1992. On the efficiency of stock-based compensation. Review qf Financial Studies 5: 471-502. Perera, S., G. Harrison, and M. Poole. 1997. Customer-focused manufacturing strategy and the use of operations-based non-financial performance measures: A research note. Accounting, Organizations arui Soc^ty 22 (6): 557-572. Prendergast, C, and R. Topel. 1993. Discretion and bias in performance evaluation. European Economic Review 37: 355-365. Quinn, J. B. 1980. Strategies for Change. Homewood, IL: Richard D. Irwin. Rucci, A. J., S. P. Kirn, and R. T. Quinn. 1998. The employee-customer-profit chain at Sears. Harvard Business Review 76 (January-February): 82-97. Schick, A., L. Gordon, and S. Haka. 1990. Information overload: A temporal approach. Accounting, Organizations and Society 15 (3): 199-220. Schiff, A. D. and L. R. Hoffman. 1996. An exploration of the use of financial and nonfinancial measures of performcince in a service orgcinization. Behavioral Research in Accounting 8: 134-153.

238


Scott, W. R. 1987. The adolescence of institutional theory. Administrative Science Quarterly 32 (4) 493-511. Snyder, A. V. 1995. Value-Based Management: Highlighting the Resource Allocation Challenge. Boston, MA: Braxton Associates. Sterman, J. D., N. P. Repennlng, and F. Kofman. 1997. Unanticipated side effects of successful quality programs: Exploring a paradox of organizational improvements. Management Science 43 (4): 503-521. Stern, J. M., G. B. Stewart III, and D. H. Chew, Jr. 1995. Tbe EVA® financial management system. Journal of Applied Corporate Finance 8 (Summer): 3246. Stewart, G. B. III. 1991. The Questfor Value. New York, NY: Harper Business. . 1995. EVA® works —But not If you make these common mistakes. Fortune 131(8): 117-118. Stewart, T. A. 1997. Intellectual Capital: The New Wealth of Organizations. New York, NY: Doubleday / Currency. Symons, R. T., and R. A. Jacobs. 1995. A total quality management-based Incentive system supporting total quality management impiementation. Production and Operations Management 4 (3): 228-241. Tully, S. 1993. The real key to creating wealth. Fortune (September 20): 30-50. U.S. Department of Commerce. 1997. Malcolm Baldrige National Quality Award 1997 Award Criteria. Gaithersburg, MD: U.S. Department of Commerce. U.S. Senate. 1993. Government Performance and Results Act of 1993. Washington, D.C.: Committee on Governmental Affairs, Report 103-58. Wallace, J. S. 1998a. Adopting residual income-based compensation plans: Do you get what you pay for? Journal of Accounting and Economics: forthcoming. . 1998b. EVA® Financial systems: Management perspectives. Advances in Management Accounting: forthcoming. Young, S. M. and F. H. Selto. 1993. Explaining cross-sectional workgroup performance differences In a JIT facility: A critical appraisal of a field-based study. Journal of Management Accounting Research (Fall): 300-326.

Innovation in Performance Measurement Trends and Research Implications

Recommend Documents