Relevance Assumed: A Case Study of Balanced Scorecard Development Using System Dynamics Author(s): H. A. Akkermans and K. E.

van Oorschot Source: The Journal of the Operational Research Society, Vol. 56, No. 8 (Aug., 2005), pp. 931941 Published by: Palgrave Macmillan Journals on behalf of the Operational Research Society Stable URL: http://www.jstor.org/stable/4102064 . Accessed: 18/02/2011 23:20
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at . http://www.jstor.org/action/showPublisher?publisherCode=pal. . Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Palgrave Macmillan Journals and Operational Research Society are collaborating with JSTOR to digitize, preserve and extend access to The Journal of the Operational Research Society.

http://www.jstor.org

Journal of the Operational Research Society (2005) 56, 931-941

C 2005 OperationalResearchSociety Ltd.All rightsreserved. 0160-5682/05 $30.00 www.palgrave-journals.com/jors

Relevance
scorecard

assumed:

case

study
system

of

balanced

development using

dynamics

HA Akkermans* and KE van Oorschot

Minase, Tilburg, The Netherlands The balanced scorecard(BSC) has become a popular concept for performancemeasurement.It focuses attention of measuresand bridgesdifferentfunctionalareasas it includesboth financialand managementon only a few performance non-financialmeasures. However, doubts frequentlyarise regardingthe quality of the BSCs developed as well as the quality of the process in which this development takes place. This article describes a case study in which system dynamics (SD) modelling and simulation was used to overcome both kinds of problems. In a two-stage modelling process (qualitative causal loop diagramming followed by quantitative simulation), a BSC was developed for management of one organizational unit of a leading Dutch insurer. This research illustrates how, through their involvementin this developmentprocess, managementcame to understandthat seeminglycontradictorygoals such as customer satisfaction, employee satisfaction and employee productivity were, in fact, better seen as mutually would reinforcing.Also, analysisof the SD model showed how, contraryto ex antemanagementintuition,performance first have to drop furtherbefore significantimprovementscould be realized.Finally, the quantitativemodellingprocess also helped to evaluate severalimprovementinitiativesthat were underconsiderationat the time, proving some of them to have unclear benefits, others to be very promisingindeed. Journalof the Operational ResearchSociety (2005) 56, 931-941. doi:10.1057/palgrave.jors.2601923 Published online 22 December2004 balanced scorecard;insurance;system dynamics;simulation Keywords: performancemeasurement;

his Trusting images,he assumestheirrelevance,; Mistrusting images,I questiontheirrelevance. my he Assumingtheirrelevance, assumesthefact,; theirrelevance, questionthefact. I Questioning Whenthefact fails him, he questions his senses,; Whenthefact fails me, I approve senses. my Robert Graves,from In BrokenImages

Introduction Some 2 decades ago, Harvard Professors Johnson and Kaplan renounced conventional financial measures as the right way to control company performancein their book
Relevance lost: The Rise and Fall of Management Account-

ing.' Instead, they introducedan integratedset of financial and non-financial measures,whichhas becomeknown under the label of the balanced scorecard.2-4 Since then, performance managementin general,and the balanced scorecard (BSC) approachin particular,have risen to prominencein both the business world and in academia. Balanced scorecardconcepts are now also studied in severalareas of managementresearch,such as organizationstudies,5operations management6,7 informationsystems.8 and
HA Akkermans,Minase BV, PO Box 278, 5000 AG *Correspondence. Tilburg, The Netherlands and Eindhoven University of Technology, Eindhoven,The Netherlands

E-mail: henk(cominase.nl

The reasons for this sudden rise to prevalence appear obviousin retrospect.On the one hand, thereis the appealof simplicity:no longer do managershave to work their way throughpiles of statistics,but can focus on monitoringsome 15-20 key indicatorsinstead.On the other hand, thereis the In strength of interdisciplinarity. the past, all the relevant inputs from differentfunctional areas had to be translated into financialdata. Some functions,such as marketing,may have been significantlybetter suited to do this than others, such as R&D or operations. But, regardless of how successfulthis translationwas, it remaineda conversation in a 'foreignlanguage'9 for non-financialmanagers.With a BSC, managers have now a more acceptable common languageto discussissues in. Despite or perhapsin responseto its popularity,the BSC concept has had its share of criticism. Broadly speaking, these criticismscan be seen as the other side of the coin for the before-mentioned advantagesof the BSC. Yes, the idea of only a few process indicatorsthat point at key leverage points of the systemis very attractive.10,11But, how can one be sure that the few ones selectedare indeed the right ones? Should there be more?Or less?And, do they all work in the same direction or might they counteract each other? Moreover, if they are the right variables, what are the correct values to target for? And, within what time frame should these be achieved?From a theoretical perspective, these are not trivial questions. Nor are they, from a

8 of Research Vol. 932 JournaltheOperational Society 56, No.

perspective,trivial to managersimplementing practitioner's a BSC, as we will observe furtheron. Rather than assume their scorecardwas relevant,the managersin the case study we will be describingwere eager to go through a rigorous its processof questioning relevance,as in the poem by Robert Graves this articlederivesits title from. A comparable weakness is inherent to the apparent of advantage of interdisciplinarity BSCs. If a BSC is to reflectall the differentrelevantperspectiveson the business, then all the stakeholdersthat representthese perspectives should be actively involved in its development.How else can one be sure that all the relevant viewpoints are representedin the BSC? But then, how does one organize a process in which a group of people with inherently different perspectives,goals and constraints and, indeed, languages,can find agreementupon just a few numbersas the basis for a joint strategy? In this paper, we suggest that system dynamics(SD) can be an effectiveway of overcomingthese limitationsto BSC development.After a brief recapitulationin the next two sections of what BSC and SD entail, we describe the developmentof a BSC with system dynamicsfor one of the businessunits of Interpolis,a leading insurancecompany in the Netherlands.We introducethe case settingnext and then describehow a 'strategymap'12,13of the BSC was developed in close collaborationwith the managementteam. We then outline the SD simulationmodel that was developed from this map and look at a number of policy experimentsthat were conducted with this model, which led to further in refinements the final BSC. In the concludingsections we consider limitationsof this researchand pros and cons of differentOR-basedapproachesfor BSC development.

The 'balance' of the scorecardis reflectedin its mix of lagging (outcome measures) and leading (performance drivers) indicators, and of financial and non-financial measures. Recently,Kaplan and Norton have developedthe notion of a strategymap as a complementaryconcept next to the BSC. A strategymap links measuresof processperformance, or key performanceindicators (KPIs), together in a causal chain that leads through all four perspectives:measuresof organizationallearning and growth influence measures of internal business processes, which, in turn, act upon measures of the customer perspective, which ultimately drive financial measures.12,13 Causal chains or causal diagramsprovidea mediumby which people can externalize mental models and assumptionsand enrichthese by sharing them.'4-l7In fact, one of the hidden strengthsof a balanced in measurement framework, particularof the BSC, may well teams to explorethe beliefsand be that it forcesmanagement their strategy.8 assumptionsunderpinning The BSC concept originatesfrom the USA. There it has been applied successfullyacross many industriesand within the public sector. It has also been delivered to an front.19 internationalaudience and on a multi-disciplinary found that the logic of the BSC was For example,Malmi20 appealing to many companies in Finland. Wisniewskiand Dickson2' describe its application to a police force in Scotland. From a functional perspective,researchersfrom differentmanagementfields have made contributions.The management accountancy aspect of the BSC has been Also consideredby, for example,Newing22and Norreklit.23 in the operations management field the BSC is wellFrom a general strategy perspective, the known.6'724'25 BSC has been described by for example Mooraj et al,26 Hudson et al.27 Partly, the success of the scorecardcan be explained by Current BSC theory and practice good timing and marketing. Managers were clearly fruThe Business scorecard is a performance measurement strated with traditionalmeasurementsystems at the time system introducedby Kaplan and Norton.2'3According to when the BSC was promotedin the RelevanceLost book by these authors, a BSC addressesshortcomingsof traditional Johnson and Kaplan and in articlesin the HarvardBusiness performance measurement systems that relied solely on Review.However,apartfrom timing,one key strengthof the financialmeasures.To overcome this, Kaplan and Norton is BSC is that its appearance so agreeablysimple.It suggests introduced three additional measurement categories that numbersone can monitor that with only a few well-balanced cover non-financialaspects. The result is a scorecardthat of the performance an entirecompany.Anotherkey strength translates the vision and strategy of a business unit into of the BSC concept is that it can serve as a bridge between
objectives and measures in four different areas: 1. The financial perspective: how the company wishes to be viewed by its shareholders; 2. The customer perspective: how the company wishes to be viewed by its customers; 3. The internal business process perspective: in which processes the company must excel in order to satisfy it shareholders and customers; 4. The organizational learning and growth perspective: which changes and improvements the company must achieve to implement its vision. different fields, both financial and non-financial ones. Next to the well-published successes of BSC, a number of inherent weaknesses have been reported in the literature. Interestingly, the advantages of the BSC mentioned in the previous paragraph can also be interpreted as disadvantages. If all one has is a small set of indicators, how can one be sure that these are the right ones; can one be assured of their relevance? And, if the BSC development process offers opportunities for bridging different fields, how does one organize this effectively? Regarding the first weakness, it has been stated repeatedly that the BSC concept provides no

933 HA and van AkkermansKE Oorschot-System for development dynamics BSC

mechanism for maintaining the relevance of defined measures.23Neely et al6 found that the problem for managersis usuallynot identifyingwhat could be measured, but reducingthe list of possible measuresto a manageable (and relevant) set. An additional concern here is that the concept of causality is not in all implementationsof BSCs equally well developed. In their more recent work on strategy maps, Kaplan and Norton emphasize the importance of showing how improvementsin one area lead to improvements in others.12,13 Norreklit23 questions the existence of such causality in most BSCs currently being used. Moreover, instead of unidirectionalcausal relationis ships, he believesthat the relationship much ratherone of of bi-directional interdependence, causality.Tan et a!28agree and stress that simply looking at different measures simultaneouslyis not enough. The linkages between them must also be understood. Regarding the second advantage, that of bridging different fields, this is in practice easier said than done. For instance, what to do when performanceindicators of differentfieldscounteractor thwarteach other?The bridging to be performedcan be both hierarchicaland horizontal. Regarding the former, Mooraj et a!26 state that the BSC measurement a twoas concept fails to identifyperformance way process, as it focuses primarilyon top-down performance measurement. Hudson et a!27 agree to this and write that BSCs have a lack of integrationbetween the top-level, measures.Regardstrategicscorecard,and operational-level ing the latter, Mooraj et a!26 discuss that BSCs do not consider the extended value chain in which employee and supplier contributions are also highlighted. Neely et al6 argue that the BSC is not able to answer one of the most fundamentalquestions for managers:what are the competitors doing? So, to summarize,the advantagesof checking just a few numbersrelatedto differentfields may become a disadvantage when there are many relevant fields to be looked at.

* Stage 2: Translate the causal loop diagram into a quantifiedsimulation model. Calibrate the model using key company data. Test the implicit assumptionsabout BSC on the basis of dynamicbehaviourin the preliminary this simulationmodel with managersand discussimplications for mental models and BSC. Not only is the distinction between qualitative and quantitativemodelling a common one in the SD literature, but a two-stage approach as suggested here to system dynamicsmodelling is also the 'normal'way of conducting in In system dynamicsinterventions organizations.14-16,29-33 this article, we show how this generic approach can be tailoredto supportthe processof developingBSCs. We will illustratehow this approachyields advantagescomparedto more 'conventionalapproaches',advantagesthat stem from the systematic approach to 'strategy mapping' that SD offers, as well as advantagesthat originatefrom the smooth transitionfrom qualitativemodels to quantifiedsimulation. Obviously, SD is not the only method available for improving the conventionalway in which most scorecards have been developed so far. There are several 'soft OR' methods that offer similar improvementsin rigour in the In mapping process.33'34 particular, cognitive mapping come to mind. (SODA)34and soft systems methodology35 Both approachesplace great emphasison identifyingcausal relations between key variablesand constructing'maps' of these. Both these qualitativeapproachescan be followed up by a more focusedquantitative modellingstage.In particular of SODA there have been reports in the literature of modelling efforts where this approach was combined with SD modelling and simulation.36-38 Also, both approaches typically employ a process of close collaborationwith key problem owners in which maps and models are developed interactivelyto assure maximal model relevanceand client The other way round, qualitative SD ownership.34,35 followed by a quantitativestage on the basis of another OR method, also occurs. One publishedexample describes the use of multicriteria analysis,precededby a first stage of causal loop diagramming.39 Of course, if SD simulationis deemedmost appropriate the Stage 2 process,a strength for of the SD approachbecomesthat the transitionbetweenthe causal maps that are developed first and the simulation model to be developed next can be a very smooth and
natural one. Problem owners see the same diagrams in the same software package that they used first but now get a 'back-end' to it, equations and graphs which build upon their intuitive grasp of the model after the first stage. There is one school of thought in operations and strategic management that sets system dynamics apart from all other OR methods for BSC development. Authors such as stress the importance of distinguishing explicitly Warren40'41 time delays and accumulations in BSCs. Regarding time delays, some indicators are leading, others are lagging. Changes can be made to inputs and changes can be observed

System dynamics as a modelling method for BSC development In this article,we suggestthe use of systemdynamics(SD) as an approach to overcome the limitations to current BSC theory and usage mentionedin the literature.In particular, we suggest a two-stage process of SD modelling for BSC development.We call these two stages the qualitativeand the quantitativestage in SD modelling.This distinctionis a very common one in SD, see for example, Wolstenholme's overviewin the specialissue on SD in JORS in 1999.29 * Stage 1: Elicit qualitativemental models from management of perceived interrelationshipsusing causal loop diagramming(CLDs), resultingin a refinedversion of a From this map, distil key performance 'strategymap'.12"13 indicators(KPIs) and assign preliminary targets.

934 JournaltheOperational Society 56, No. of Research Vol. 8

in outputs.28 'conventional'BSCs this distinctionin time In is not made as explicitly and rigorously as in SD delays models. Regardingaccumulations,in the strategicmanagement literature on the resource-basedview of the firm, the concept of accumulation as a driving force for firm Such reperformance has been stressed repeatedly.4243 sources, both tangible and intangible, grow or decline that really graduallyover time, and it is these accumulations driveorganizational performance accordingto this literature. Therefore,one should make it clear what those accumulations really are and what drives their behaviourover time. The stocks-and-flowsconcept in SD addresses both time in delays and accumulations a rigorousmanner,while most other BSC and OR methods do so considerablyless.

The case study The case study describedin this articleconcerneda business unit of Interpolis.Interpolis(www.interpolis.nl) one of the is insurers the Netherlands.Since 1990,the company in leading as a whole formspart of the Rabobankorganization,one of the three large bankingand insuranceconglomeratesin the Netherlands. Interpolis employs some 6000 people. 2003 revenueswere 5 Billion. The businessunit concernedis called StichtingRechtsbijstand ('Foundationfor LegalAid', abbreviated from here on as 'SRB'). Some 600.000consumershave insuredtheirneeds for judicialaid with this formallyindependentorganization. Some 300 SRB employees provide this aid. Overall, the demand for legal aid continues to increaseconsiderably.In 2003, 68.000 requestsfor legal aid were noted, which is up 16% from 2002. In the years prior to our involvementwith this company, SRB had gone through a period of considerableupheaval. First, there had been several changes in top management. Secondly, there was the noted and continued significant increasein demandfor its services,as a resultof changes in the marketand new insurancesales policies with its parent, Rabobank. In response, staff hiring had increasedsignificantly, aftera long periodof little to no growth.Thirdly,the organizationhad recentlyundergonea major restructuring, shiftingfrom a regionalstructureto a structureaccordingto area of judicialexpertise.Fourthly and finally,the management team memberswere almost all less than a year in their currentjobs, includingthe CEO. All in all, the time seemed right for a seriousreorientationon key goals for the future. This was the backgroundagainst which our involvement with this company should be situated. We, the authors, formed part of a small group of external consultantswho facilitatedthe managementteam (MT) of this organization in the developmentof a BSC. This developmentprocesswas set up in two stages. During the first stage, preparatory interviews were conducted with MT members, the results of which were

discussedin a half-dayworkshopwherethe groupengagedin a numberof causalloop diagramming exercises.The findings from this workshop were summarizedin a so-called workwhich was distributedto the MT members,studied, book,5," filledin and sent back by for analysis,leadingto input for the next workshop.A simplifiedand stylizedversion of the end resultof this CLD mappingexerciseis shown in Figure 1 and describedin the next section. The intermediatecausal loop diagramswerediscussedin a final full-dayworkshop.On the basis of these discussions,an intermediate version of a BSC was generated in a fairly straightforwardmanner. MT memberschose those key indicatorsfrom the diagramthat they felt would enablemanagementto keep a good graspof key drivers of performance.The resultinglist of measures and objectiveswas discussed,refined, simplifiedand finally versionof the agreedupon. This resultedin the intermediate BSC that is shown in Table 1. Perhaps the most important managerial insight that emerged from this stage was that management came to realize how goals that they had first believed to be at odds with each other were, in fact, not mutually exclusive but mutuallyreinforcing.It was not: choose eitherfor customer satisfaction, or for employee satisfaction or for cost effectiveness, but rather: either we will achieve all three goals or none at all. The key linking concept was employee productivity. Higher productivity does mean greater cost effectiveness,but also greatercustomersatisfaction,as cases are handled sooner, and greater employee satisfaction, as work pressureis less severe. At the end of stage 1 therewas agreementon the content of the BSC. Equallyimportant,therewas also agreementon the approachforward.Especiallyrelevantin the context of the currentarticleis that the team felt pleasedwith theirfirst BSC, but at the same time was very uncertain about its Had they been quality.Werethese reallythe rightindicators? complete? And would these indeed all work towards the same goal? To what extent would the chosen indicators really be sufficientto achieve the overall company mission? On the other hand, could this list of KPI's not be shortened further?After all, the fewer dials to watch the easier it becomes to monitor performance effectively. To address these uncertainties, team decidedthat a systemdynamics the simulation model was to be developed to investigate these questions more thoroughly. In the second stage of the project we, the authors, developed a quantified simulation model for SRB. We startedfrom the causal loop diagramsand the intermediate BSC that had been developed in the first stage. These were sufficientto develop a first skeleton of an SD model. This skeletonwas then filledwith key company data, which were deliveredby two managersfrom the MT. These two were more closely involved than the others in the subsequent developmentof the model, critiquingintermediateversions and providingvaluablefeedback,fulfillingwhat Richardson and Andersenhave called a 'gatekeeperrole'.

HA AkkermansKE Oorschot-System for development and van 935 dynamics BSC

case processing
experience +

+ +

productivity

employee

practice breeds perfect loop

case processing+ + rate

work pressuremotivation loop +/

+throughput
time customer

D"
case outsourcing /e loopurcing . /

crpe work pressure

(B2

ase workload
+ c rr

B3 customer
satisfaction loop

satisfaction s policies e n
staff available

#insurance

employee motivationsai

Outsourcing
P1
(

rate

+ rate R3 employee.hiring
retention loop \ \

OUTSOURCING TARGE

time available for processing cases

\ \\

quality of case intake \+

+ \+

workpressure ~hiringoo ir g loop

>

experienced . +ne t.ffstaff new staff P2

--,.+

-STAFF TRAINING

+/
employee satisfaction
.

eexperienced employee ~quit rate

staff available
f

HI-QUALITY T CASE INTAKE P experienced staff allocated to intake

Figure 1

Causal loop diagram of key interdependencies.

Table 1 Objectives Financial perspective Customer perspective Process perspective

Intermediate BSC as developed during the first stage Measures Output per employee Percentage outsourcing of cases Customer satisfaction Throughput time per case Throughput time per case % of small and easy cases Number of successful projects Working at home

Be able to meet continued demand growth Keep cost levels in line with agreements Deliver a good service for a reasonable price Improve company agility Reduce work pressure and employee stress in general

Learning and growth perspective

Attract and retain good people Increase collaboration between employees

Employee turnover rates Employee satisfaction Training on the job/coaching Hiring of new staff through referral by colleagues

With the calibratedsimulationmodel that was developed in this manner, we conducted a number of analyses that addressedthe questionsthe MT was still grapplingwith after the first stage. This process led to significant additional managerial insights.Firstly,it was comfortingfor the team to

see that most of the KPIs selected for the intermediate version of the BSC were confirmed by the simulation exercises as key in driving performance in the simulation model. Secondly, what was less comforting for management to notice was the model's prediction that performance would

936 JournaltheOperational Society 56, No. of Research Vol. 8

first deterioratefurtherbefore things would get better. The increasein workload that had been buildingup the past 2 yearswould stabilizein the year ahead beforeit would drop significantly. As a result, setting ambitious targets for especially the first half of the coming year was not appropriate.This was a message that none of the actionoriented managers really liked, but one that was key in managingexpectationsadequately. A thirdmanagerialinsight from the simulationeffort was that it focused attention on the real leveragepoints in the company, on those policies that really mattered to performance and away from the ones that were less likely to be high-impact.Among these were, as we will also see further on, the two-edged sword of outsourcing of cases to thirdparties and the mixed blessing of additional staff training. Unexpectedly high benefits were found to result from anotherpolicy initiative,which was to install a high-quality case intake process staffed with experiencedemployees. All these findingshave been discussedwith the management team, have been challenged by them, have in some cases been mitigated but neverthelesshave broadly been accepted.They have been used to guide the implementation of the BSC approach for the organizationas a whole and well as for the various sub-unitsinvolved.

practice, a quick and straightforwardsettlement may be preferable.Judgmentssuch as these are best learnedon the the job. Interestingly, more cases one has handled,the more proficientand productiveone becomes. So, the better one gets, the better one becomes: a clear virtuous cycle of mutuallyreinforcingexperienceand productivity.
R3. Employee retention loop: Employees that feel pressur-

ized by high workload are less motivated. Unmotivated employees are more likely to leave than motivated ones. A higher quit rate furtherreduces the organization'scapacity to handle cases, which increases work pressureeven more and leads to even more staff turnover. Obviously, this reinforcing loop works the same way as well: happy employeeswill be more inclinedto stay, making work loads better manageableand everybodyeven more happy. In the former setting, this structureis called a vicious cycle, in the latter the same structurebecomes a virtuous one.
Bl. Work pressure-hiring loop: Apart from simply

Qualitative model structure A simplifiedand stylized version of the qualitativemodel that was the end resultof the first modellingphase is shown in Figure 1. In this causal loop diagram,six interconnected key feedback loops are shown that together determinethe dynamicbehaviourof the model. These are labelledRl, R2, R3 with the 'R' standing for 'reinforcing' or positive feedback loop, and Bl, B2 and B3, the 'B' standing for 'balancing'or negativefeedbackloop.16 observationthe MT made was that, in recent years, work pressurehad gone up as a result of increasesin requestsfor legal aid. More work had to be performed by the same people, so work pressure went up. As a result, employee motivation had dropped. Lower motivation leads to more more sick leaves,and, in general,to a reduction absenteeism, in productivity(defined as # cases handled per person per
period). Productivity changes affect again the workload, and hence, the work pressure, leading to a vicious cycle of low employee motivation and high work pressure. R2. Practice breeds perfect-loop: How do employees become more proficient in processing cases? Obviously, by training, but mostly by learning-on-the-job. A law degree does not automatically make one an ideal staff member at SRB. For instance, a solution that is legally quite sophisticated may not be the most desirable option from a client or managerial perspective, as it may require long handling times and a great deal of personal attention. In R1. The work pressure-motivation loop: A pertinent

waitinguntil staff becomessufficientlyproficientin handling highervolumes of work, what else can one do? One obvious solution is to hiremore people. This is visualizedby loop BI, which goes fromwork pressureto hiringrate, and from there to new staff, to staff available, from there on to case processingrate, to case work load and back again to work pressure. Obviously, this adjustment process takes some time. In the case of SRB, it was estimated that more staff could effectivelybe hired after some 3 to 6 months. Then, would gettingthem up to speed,that is, become experienced, take 1-3 years.
B2. Case outsourcing loop: A more short-term fix to high

workload levels was to outsource part of the caseload to outside firms.This reduceswork pressureimmediately.How much was to be outsourced was a point of debate. One obvious reason was that this was expensive.Another, more subtle one, was that every case outsourced was a learning opportunitymissed, in the logic of loop R2.
B3. Customer satisfaction loop: Perhaps in theory most

powerfulbalancingloop was the one involvingthe customer. If, as a result of high workloads, processingtimes become very long and other aspects of performance deteriorate likewise, consumers may decide that they are better off switchingto anotherinsurer.However,in the realityof SRB at that time, this was definitelynot an immediateconcern. and Most people will ask for legal aid veryinfrequently, even
then handling speed would only be one of many drivers of customer satisfaction. These six feedback loops describe most, but not all the variables and links in the diagram. In particular, they do not fully address the three specific managerial policies that were being considered by the MT at that time, and that found their way into the intermediate BSC. The first one (labelled as P1 in the diagram) was to strongly increase the target for outsourced cases, so as to relieve workloads immediately, and help reverse the employee retention loop from a vicious into a virtuous cycle. The second proposed policy was to

HA Akkermans KE Oorschot-System and van for dynamics BSC development937

train new staff better and hence to boost the company's abilityto handlehigherworkloadsof cases. Obviously,when employees are on trainingthey cannot be handlingcases at the same time, so this increasein trainingwould come at the expense of staff availability. Finally,therewas a plan to improvethe qualityof the case intakeprocessdramatically, that everynew case would be so allocatedto preciselythe rightpersonin the organization,as these differed widely in skill levels, fields of experience, hobby's, work style and the like. This was a job that would requirethe refinedjudgmentof experiencedstaff, who were at the same time badly neededfor handlingcases. So, every policy had its potential downside,and from the conceptual discussions during Stage 1 of the modelling process it becameclear that quantification would be neededto provide final answers. Quantitative model structure In the second stage of the BSC definition,a quantifiedSD simulationmodel of the insurancecompany was developed. In this section, the structureof this model is described.As a results of reasons of confidentialityand size, we will not describethe model in full detail. The quantitativerelationships betweenvariablesare not givenin this paper.However, the full model (with fictitiousnumbers)is availablefrom the authors upon request. The simulation model is based on the causal loop diagrams described above. Quantified system dynamics modelling asks for specification of the main flows and accumulations,or stocks, in the system.In this case, the two main flows are the processing of cases and the flow of employees.The most importantstocks are, for the firstflow, cases being processedand, for the second flow, employees, both new and experienced. These two main flows are drawn in Figures2 and 3. The causalloops connect these two main flows in variousways. In the next paragraphs contentsof the the model are explained. Flow of cases The legal aid that SRB providescan only be suppliedafter a so-calledintakeprocess.In this processit is decidedwhether the requestfor legal aid of a customercan be given by the company. When the intake is accepted, a specificcustomer file (a 'case' in legal terms)is made and this case is allocated to an employee for furtherprocessing.Occasionally,when

New '3HireRateof 0""

New Experienced Z ? Em ployees IC3 ployees Assimilation IEm Rate HireRate Experienced Zo , ,, Employees QuitRateof New QuitRate Experienced Employees
7-7

23 23 for Figure3 Stocks-and-flows diagram employees. employeesare too busy, a case can be outsourced.When the intake is rejected,the request of the customer is discarded without furtherprocessingtakingplace. This flow of cases is in notationl6 in Figure 3. represented stocks-and-flows
Flow of employees

Figure 3 shows the employee flow for new and experienced employees. This distinction is necessary because both turnoverrates and productivitydiffer for new and experienced employees.New employeesbecome experiencedafter a certain assimilationtime. More experiencemeans higher productivity.So, the productivityof experiencedemployees is considerably higherthan that of new employees.However, experienced employees are also needed to train new employees. The time that experienced employees 'lose' decreasestheir productivity,definedas the number of cases that they can handle per period. These two main flows, cases and employees, interact in many ways. Most of the feedbackloops that were described in the precedingsection link these two parts of the model. For instance, loop R3 states that higherworkload (flow of cases) leads to more work pressure and hence to more employee turnover(flow of employees).Loop BI says that highercase load (flow of cases)and more work pressurelead to more aggressive hiring(flowof employees).Quantification of most of these loops is straightforward a trained SD for model builder.One exceptionis loop R2, the 'practicebreeds perfect'loop. In more specificterms,this part of the model calculatesproductivity.It is based on learningcurve theory, in which experience is linked with productivity. For our calculations, we referredto the model provided by Sterman,16 p 507). His key definitionis: Productivity= Reference Productivity

Average Experience ( Reference

ExperienceJ

23

for New Requests LegalAid -.Requests

Inke Intake

Cases in Process

2 hd FinishedCases

or AverageExperienceof eitherexperienced new employees is the total experience(expressedin working years) divided or by the numberof experienced new employees.Experience can be gained through processingof cases, but experience
can also be lost. People forget relevant knowledge and new developments in the insurance sector cause experience to become obsolete. This is expressed in the model by an experience decay rate. Furthermore, experience is also lost when employees leave the company.

RejectedRequests

Outsourced Cases

for Figure2 Stocks-and-flows diagram cases.

938 JournaltheOperational Society 56, No. of Research Vol. 8

is ReferenceProductivity the productivityattained at the ReferenceExperiencelevel. For example, in the simulation model ReferenceExperienceis about 7 working years for experienced employees and 0.2 working years for new employees. ReferenceProductivityfor experiencedemployees is almost double the referencevalue for new employees. The exponent c in the computation of the productivity determinesthe strengthof the learningcurve and is equal to c = In(1 + fp)/lIn(2) where is the fractionalchangein productivity doubling per fp of effectiveexperience.12 Model analysis The simulationeffort was conducted because the team was insecureabout the qualityof the balancedscorecardthat had been developedon a qualitativebasis. After quantification, what were the lessons learned?One overall conclusion was that the performance indicatorsthat had been selectedwere broadly speaking the right ones, but that the chosen performanceimprovementindicators were not straightforwardly positive. So, aspects such as employee productivity, throughputtime, customer and employee productivityand employee turnoverrates were indeed confirmedto be key driversof performance,also in the quantitativeversion of the model. Figure4 shows the behaviourover time for some of the key performanceindicators identified. This graph historyand partlypredictsfuturebehaviour, partlyreplicates as the modellingeffort took place towardsthe end of Year 2. Although precise time series data were missing for the two precedingyears, behaviourwas found to be broadly in line The gradual with availableknowledgeof past performance. of work pressureover time is clearlyvisible,and the build-up increasesin throughputtime that are the logicalconsequence but of that as well. Less straightforward quite explainableis the gradualdecreaseof employeeproductivity(measuredin Cases per Employee per Year, so C/(E/Y)). As there had staff been a considerableincreaseof new and inexperienced

and productivityof new staff is only around half of that of experiencedstaff, averageproductivitywould have to drop. Overall performance:gradual progress, but first worse-before-better What Figure 4 also shows is that progress would not be made immediately.For the first half of Year 3, the model predicted that performancewould still become somewhat worse, and that only around the start of Year 4 work pressure and throughput times would have gone down significantly. So, the good news was that a turnaround appearedto be in the making;the bad news was that SRB was not quite there yet. This finding resulted for management in the uncomfortable recommendation of setting performancetargetsfor the first quartersof the coming year not higher but lower than last year's performance,which is not a naturalinclinationof most managers. Existing policies insufficientfor swift progress Figure 4 shows performancewith existing policies. These includedpolicies for hiringnew staff. So far, those had been mainly reactive: when work pressure went up, more employees were hired. This also becomes apparent from Figure 5, where some key performanceindicators for the flow of employeesare visualized.From this graph it is clear that thereare severaldelaysinvolved.When it is finallynoted that work pressure is structurallyincreasing,it still takes some time beforeemployeesare actuallyhiredand then even longerbeforethey becomeproductiveand the effectof hiring them becomes visible. For example, in Year 0 the total number of employeeswas 207. In Year 3 this numberwas estimatedat 330. BetweenYear 0 and 3 throughputtime and work pressure(see Figure 4) were still increasing. Unclear benefitsfrom proposed policies The rate of improvementwith existing policies was hard to swallow for management.What could be done to speed up

4 Dmn1 400 C/(Y*E) 0.6 Year 2 DmnI 250 C/(Y*E) 0.3 Year 0 DmnI 100 C/(Y*E) 0 Year

I veragethroughut time (years)

400 60,000 c/Y

10

200 E/Y E
Fl 100E/Y 250 E

Newcaserate (C/Y)
T tal numberof e nployees (E)

Preivewok

prssure (dmn)

40,000 C/Y 0 E/Y 100 E 20,000 C/Y

of Productivity experien ed employee?(C/(Y*E)).

0 Timing of

2 3 Time (Year)

.v

23

Timingof

(Year) Time

BSCmodelling

BSCmodelling

Figure 4 Work pressure,throughputtime and employee over time. productivity

Figure 5 Delays in increasinghiring rates in response to demandrate. increased

HA 939 AkkermansKE Oorschot-System forBSC and van development dynamics

There are many limitationsto the study we could mention and discuss in this section, for instancethat since it reports on a single case, generalizability findingsis problematic. of One limitationthat is specifically relevantin this case is the distinction one has to make between modelling 'mental models' and modelling'the real world'.The processthat the Interpolis management team went through focuses on making explicit the mental models of the individual managers, on sharing them, on challenging their internal Unexpected leverage from a counter-intuitive policy consistencyand on aligning them. What this approach did The importanceof employee productivityhad been repeat- not aim or claim to do is to model the 'real world', edly stressed and confirmed. The crucial importance of independentof what the managers'perceptionof this real world was. The philosophicaldimensionof this distinction experienced staff for overall performancehad also been noted several times. As more and more junior staff would we will not solve in this article,as this goes back all the way come on board, the higher productivityof the experienced to Plato's cave and there still remain two camps of academics: those who insist that all models are social employees would be all the more a strategic asset. So, an initiative that would limit the time that experiencedstaffT constructionsof realityand those who believethat thereare could spend on handling cases could well be labelled as at the very least significantelements of objectivity in all 'counter-intuitive'. And yet, this was what the initiativeof social system models.34,44 the high-qualitycase intake process was all about. Fortunately,the practicalside to this limitationis easier When new cases come in, they have to be evaluatedand resolved than the philosophicalone. We stress that deveassigned to specific staff members. Experiencedemployees loping a rigorous model of real-worldbusiness processes
0.2 1/Y 4 Dmnl 4 Dmnl 0.6 Year 400 C/(Y*E) Base Case: time (Y) Throughput Base Case: Workpressure(Dmnl) 2 DmnI 0.3 Year Intake: High-Quality time (Y) Throughput

One obvious remedywas to outsourcemore cases. progress? The systemdynamicsmodel showed that indeed,in the short term, outsourcinghas a positive effect on work pressure,as shown in Figure 6. In Year 2, almost 15% of all cases are outsourced and, indeed, between Years 1 and 2 work pressure is somewhat lower than in the scenario of no outsourcing.However,after Year 3, work pressurebecomes slightly higherin the outsourcingscenario.This is because, when cases are outsourced,employees cannot gain experience from those cases, and in the long run these missed opportunitieshave a negative effect on their productivity and, consequently,on work pressure. This experimentshowed that outsourcingmight well be a two-edgedsword.Whenappliedselectivelyto specialsettings (one example includeda departmentwith very experienced staff, so with limitedlearningopportunities,but a very high case load and a tight job market),outsourcingcould work well to alleviate short-term pressures. However, when in applied too lavishly it might be counter-productive the run. So, it was considered better not to make '% long outsourced'a KPI in itself. Therewas a similarstory to tell for the policy of increased staff training.More staff trainingwould improveproductivity and motivation, but it would also reduce the time available to learn from handling real cases. Again, this proposedpolicy was found to have unclearbenefitsand was thereforeremovedfrom the BSC.

not only can 'read'a case veryearlyon, they are also familiar with the strengthsand weaknesses,preferences and dislikes of their fellow staff members. When more experienced employeesare involvedin the intake process,it is likely that the fit betweencases and employeeswill improve.The better this fit, the higher employee satisfaction becomes. This operational measure was tested; the results are shown in Figure 7. In the base case, 50% of the intake of cases is handled by experiencedemployees (and consequently50% by new employees). In the high-quality intake-scenario, experienced employeesdo 75% of the intake.Figure7 shows that this policy has indeed a significantpositive effect on work pressure,and consequentlyalso on employeeproductivity and throughputtimes. Discussion 'mentalmodels', not of the Limitations:.modelling of
reall World

Base Case: Workpressure_ (Dml)n

'tigh-Quality

Intake: Workpressure (Dmnl)

0.1 I/Y
2 Dmnl Outsourcin?Sce rio: 0 I/Y 0 Dmnl 0

250 ('/(Y*l.:)
( I %Yof" cssout~our~cass/Y)
I 2 Work pressure(Dmnl) t, 3 4
OusourcingS r cn Searo

0 Dmnl 0 Year 100 ('.;(Y*i)

Time(Year)

Base Case Productiv new employees ity (C/(Y*E)) 1 T 2 Time (Year)

High-Qualiy Intake: nes employees Productivity (C/(Y E)) 4 5 3

on Figure6 Effectof outsourcing work pressure.

7 a intakeprocess. Figure The effectof installing high-quality

940 JournaltheOperational Society 56, No. of Research Vol. 8

through direct observation is a laudable, but fairly timeconsuming process. The approach that we have presented here is intended to supplementa strategicdecision-making process. So, the fair comparisonto be made is not between the model that one develops through the process we have outlined and some theoretical'optimal'model. Rather, one should set a BSC developmentapproachinformedby system dynamics against a conventional approach of developing BSCs and then see if the benefitsof using SD outweigh the drawbacks. The added value of system dynamics to BSC development Management of SRB and Interpolis has remained quite positiveabout the processdescribedin this article.Generally speaking,the future has unfolded pretty much as had been predicted during the modelling effort. So, after an initial in period of seeminglylittle progressin performance, fact of further deterioration of throughput times, considerable improvementsin operation could be noted, especially in the second half of the year. To what extent was this due to the use of the systemdynamicsapproach?This is a question that is difficultto answerobjectively,but SRB management did indicate that both the causal diagrammingworkshops and the simulation effort had had clear added value for them. So, some kind of causal diagramming exercise seems worthwhile.Obviously,this is prettymuch what Kaplan and Norton themselveshave been stressing recently with their notion of 'strategymaps'. And, as observedearlieron, there are other mappingtechniquessuch as SODA and SSM that one could employ equally well. We do not see a distinctive 'competitiveadvantage' to the use of SD here, rather an approach that is in line with current best practice in BSC development. The use of simulation modelling, and SD simulation in particular,is far less frequentlyattemptedin BSC development and yet yielded considerableadditionalinsightsin this case. In particular,the quantificationeffort helped to be focused on the importanceof a good appreciationof time delays and accumulations.The MT found it essential to recognize that adjusting to the sharp increasein customer demandtook a significantamount of time, and that the case load that had accumulatedover the past few yearswould not
disappear overnight, even when staffing levels would be in line with current demand rates. It is doubtful if understanding the importance of delays and accumulations, for which SD seems eminently suited, is essential in all BSC development settings. Certainly, this single case cannot prove that. But, in general, one would expect that specific problem settings ask for specific tools. The case of Santos et a139 describes a setting where multicriteria analysis seemed most appropriate at the end of a qualitative modelling stage where SD was applied. We hope that, in the future, BSC development practitioners will

continue to 'mix and match' methods where appropriate, and use screwdrivers screws, hammersfor nails, rather for than regardingeverythingas a nail since all they have is a hammer.

Conclusion Some 2 decades ago, balanced scorecardpioneers Johnson and Kaplan' statedthat conventionalmanagementaccountlost'. As an alternative, ing deservedthe label 'Relevance they introducedthe concept of the BSC, in which the organization tries to focus on a small number of truly relevant indicatorsto monitorand improveperformance. This article has looked at a settingof BSC developmentfor management of one businessunit of the Dutch insurerInterpolis.These managersfound the BSC concept helpfulto arriveat a list of financial and non-financial performance measures, which they saw as the most important ones. However, they were uncertainif these were really the right measuresto monitor. Rather than assumingthat their scorecardwas correct,this managementteam went through a system dynamics-based approachthat was both thorough and practicallyfeasibleto question the relevanceof the measuresit contained. The use of system dynamics has proved to be very was beneficialin this process.The use of causal diagramming very instrumental during the first stage of modelling in The identifyingkey variablesand their causal interrelations. use of SD simulationmodellingwas essentialin arrivingat a proper appreciationof the importance of time delays and in accumulations the key businessprocessesof handlinglegal cases and of attractingand retainingemployees. System dynamics remains, from the palette of systems interventions available, the technique that, in terms of modelling,has from the onset been developedto quantitative 'boldly go where no one has gone before' in areas where reliable data and theoreticalmodels are lacking but nevertheless the need for simulation, for scenario analysis, is clearly apparent. Surely, balanced scorecard development fits this description well. Although other techniques and approaches will be more appropriate in some cases, SD remainsa good choice to test for relevancein a wide variety of BSC developmentsettings.
References 1 Johnson HT and Kaplan RS (1986). RelevanceLost: The Rise and Fall of Management Accounting.HarvardBusiness School Press:Boston. 2 Kaplan RS and Norton DP (1992). The balanced scorecard: measuresthat drive performance.HarvardBusinessReview70: 71-79. 3 Kaplan RS and Norton DP (1996). The BalancedScorecard: TranslatingStrategy into Action. Harvard Business School Press:Boston. 4 Kaplan RS and Norton DP (2001). Leading change with the balancedscorecard.FinancialExecutive17: 64-66.

941 HA AkkermansKE Oorschot-System for development and van dynamics BSC

5 Dinesh D and PalmerE (1998). Managementby objectivesand the Balanced Scorecard:will Rome fall again? Management Decision36: 363-369. 6 Neely A, Gregory M and Platts K (1995). Performance measurementsystem design: A literaturereview and research
agenda. International Journal of Operations and Production

14: Management 80-116. 7 Bourne M et al (2000). Designing,implementingand updating performancemeasurement systems. Int J Opns Prod Mngt 20: 754-771. 8 Martinsons M, Davison R and Tse D (1999). The balanced scorecard: a foundation for the strategic management of
information systems. Decision Support Systems 25: 71-88. 9 Hill T (1989). Manufacturing Strategy. The Strategic Management of the Manufacturing Function. Macmillan: London. 10 Ackoff RL (1981). Creating the Corporate Future. Plan or be Planned for. Wiley: Chichester.

27 Hudson M, SmartA and BourneM (2001).Theoryand practice in SME performancemeasurementsystems. Int J Opns Prod Mngt 21: 1096-1115. 28 Tan KH, Platts K and Noble J (2004). Building performance through in-processmeasurement:toward an 'indicative'scorecard for business excellence.Int J Prod PerformMngt 53(3): 233-244. 29 Lane DC (1992). Modelling as learning: a consultancy methodology for enhancing learning in management teams.
Eur J Opl Res 59: 64-84.

30 Winch GW (1993). Consensusbuildingin the planningprocess: benefits from a 'hard' modelling approach. System Dynamics Rev 9: 287-300. 31 AkkermansHA (2001). Renga:a systemsapproachto facilitanetwork development.System Dynating inter-organisational
mics Rev 17: 179-194. 32 Rosenhead J (ed) (1989). Rational Analysis for a Problematic Problem Structuring Methods for Complexity, UncerWorld:. tainty and Conflict. Wiley: Chichester. 33 Pidd M (1996). Tools for Thinking. Modelling in Management

11 ForresterJW (1992).Policies,decisionsand informationsources for modelling.Eur J Opl Res 59: 42-63. 12 Kaplan RS and Norton DP (2000). Having trouble with
your strategy? Then map it. Harvard Business Rev 78(5):

167-178.
13 Kaplan RS and Norton DP (2004). Strategy Maps. Converting Intangible Assets into Tangible Outcomes. Harvard Business

School Press:Boston.
14 Senge P (1990). The Fifth Discipline. The Art and Practice of the

Science.Wiley: Chichester. 34 Eden C (1989). Using cognitive mapping for strategicoptions development and analysis (SODA). In: Rosenhead (ed). RationalAnalysisfor a ProblematicWorld.Wiley, Chichester, UK, pp 21-42.
35 Checkland P (1981). Systems Thinking, Systems Practice. Wiley:

New York. LearningOrganisation. DoubledayCurrency:

15 Vennix JAM (1996). Group Model Building. Facilitating Team Learning Using System Dynamics. Wiley: Chichester. 16 Sterman JS (2000). Business Dynamics. Systems Thinking and Modelling for a Complex World. McGraw-Hill: New York. 17 Haas M de (2000). Strategic In Search of Goal Dialogue:.

Coherence. PhD thesis, EindhovenUniversityof Technology.

18 Neely A (1998). Measuring Business Performance. Profile Books

Ltd: London. 19 HepworthP (1998). Weighingit up-a literaturereviewfor the balancedscorecard.J Mngt Dev 17: 559-563. 20 Malmi T (2001). Balancedscorecardsin Finnish companies:a researchnote. Mngt AccountRes 12: 207-220. 21 WisniewskiM and Dickson A (2001). Measuringperformance in Dumfries and Galloway Constabularywith the Balanced Scorecard.J Opl Res Soc 52: 1057-1066. 22 Newing R (1994).Benefitsof a balancedscorecard. Accountancy 114: 52-53. 23 N0rreklitH (2000). The balanceon the balancedscorecard-a criticalanalysisof some of its assumptions.Mngt AccountRes 11: 65-88. 24 Flapper SD, Fortuin L and Stoop PPM (1996). Towards consistentperformancemanagementsystems. Int J Opns Prod Mngt 16(7):27-37. 25 Kleijnen JPC and Smits MT (2003). Performancemetrics in supply chain management.J Opl Res Soc 54: 507-514. 26 Mooraj S, Oyon D and Hostettler D (1999). The balanced
scorecard: a necessary good or an unnecessary evil. Eur Mngt J 17: 481-491.

Chichester. 36 Eden C (1994). Cognitivemappingand problemstructuring for system dynamics model building. System Dynamics Rev 10: 257-276. 37 Eden C, WiliamsT, Ackermann and Howick S (2000).On the F nature of disruptionand delay (D&D) in majorprojects.J Opl Res Soc 51: 291-300. 38 Howick S (2003). Using systemdynamicsto analyse disruption and delay in complex projectsfor litigation:can the modelling purposesbe met?J Opl Res Soc 54(3): 222-229. 39 Santos SP, Belton V and Howick S (2002). Adding value to performance measurement by using system dynamics and multicriteria analysis.Int J OpnsProd Mngt 22(11): 1246-1272.
40 Warren K (2002). Competitive Strategy Dynamics. Wiley:

Chichester.
41 Warren K (2003). The Critical Path. Building Strategic Performance Through Time. Vola Press: London.

42 Wernerfelt B (1984). A resource-basedview of the firm.

Strategic Mngt J 5(2): 171-180.

43 Dierickx I and Cool K (1989). Asset stock accumulationand sustainability of competitive advantage. Mngt Sci 35(12): 1504-1511.
44 Flood RL and Jackson MC (1991). Creative Problem Solving:.

Total SystemsIntervention. Wiley:Chichester.

accepted

Received June 2002; October 2004 after one revision