Journal of Cleaner Production 14 (2006) 113e123

www.elsevier.com/locate/jclepro

Development of a methodology for benchmarking public

transportation organisations: a practical tool based
on an industry sound methodology*
Harry Geerlings a,*, Roman Klementschitz b, Corinne Mulley c,1
a
Erasmus University Rotterdam, Erasmus Centre for Sustainable Development and Management,
PO Box 1738, 3000 DR Rotterdam, The Netherlands
b
University Bodenkultur, Austria
c
Transport Economics at the University of Newcastle upon Tyne, United Kingdom
Received 10 November 2004; accepted 11 March 2005
Available online 10 May 2005

Abstract

Transportation plays an important role in modern societies. Despite the advantages inherent to well functioning transportation
systems, they always generate significant negative side effects that put a burden on the quality-of-life, especially in urban areas. This
is especially true of the way in which increasing automobile usage generates important negative impacts such as congestion, increases
in accidents and additional vehicle emissions. The European Commission proposes to address these impacts by shifting modal share
away from private transportation: in the local transportation sector this means improving the performance of public transportation
operators so as to compete with the advantages of private car use.
Recent legislation has increasingly required public transportation companies to operate in an environment of increasing
competition; consequently there is constant pressure on the operators to improve the efficiency of their performance in order to
survive. Benchmarking is considered as one of the most promising ways to help maintain and improve the quality and efficiency of
this sector by the sharing data on good practices among operators. This paper concentrates on the practice of ‘benchmarking’ e.g.
the measurement of indicators for an individual operator and ‘a comparison of results’ with other operators. The empirical basis for
the paper comes from the EU-DG project called EQUIP (Extending the Quality of Public Transportation).
The paper has the following structure: in the introduction of the paper, the theoretical framework is discussed: what are the
experiences in the industry with benchmarking in public transportation? What is measured with benchmarking? How are the
potential benefits of benchmarking exploited? The second section deals with the methodology of the benchmark process. Attention
is then turned, in the third section, to practical experience. It deals with the complications of data collection and how to select
indicators to provide relevant information for public transportation.
After some interim conclusions on achieving and sustaining improvements in Section 4. In the final Section (5), the paper
concludes with the experiences of piloting this methodology and makes recommendations for changes in the institutional
environment for the successful introduction of benchmarking on a national and European basis for this sector.
Ó 2005 Elsevier Ltd. All rights reserved.

Keywords: Public transport; Bench marking; Quality improvement

*
This paper is based on EU-DG TREN project called EQUIP (Extending the Quality of Public Transport). This project concentrates on
developing a benchmarking handbook for the self-assessment of internal quality performance by public transport operators and the building of
a centralised database for comparison.
* Corresponding author. Tel.: C31 10 408 2050; fax: C31 10 408 9104.
E-mail address: geerlings@fsw.eur.nl (H. Geerlings).
1
She was also the co-ordinator of the EQUIP project.

0959-6526/$ - see front matter Ó 2005 Elsevier Ltd. All rights reserved.
doi:10.1016/j.jclepro.2005.03.021
114 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
1. Introduction
Transportation has been a fascinating phenomenon
in society of all times. Since ancient times, fast and
efficient transportation was connected with connota-
tions of economic prosperity, social desire and welfare,
with many positive (side)-effects.
Transport is a fascinating phenomenon in society of
all times. Since ancient times fast and efficient transport
is connected with connotations of economic prosperity,
social desire and welfare. In fact, they have been so
influential that economic historians have termed whole
periods of economic development after various trans-
portation infrastructures, e.g. the ‘age of canals’ in the
first half of the 19th century, or the ‘railway and coal
era’, the expansion of which ended with the Great
Depression in the 1930s. The oil and automobile alliance
was the symbol of ‘the age of automobile’. The car is one
of the main contributing factors to a period of expan-
sion unprecedented in the economic history of mankind.
The success of technology is reflected in changes within
society and the associated need for transportation. For
instance, the success of the private car goes hand-in-
hand with the process of individualisation in society.
At the same time transportation generates negative
external (side-) effects. These side effects are a direct result
of the success of transportation technologies. For in-
stance, technology development contributes to an un-
equal growth in the different transportation modes; the
fastest growing mode of transportation is air trans-
portation. Also the usage of the private car is still
increasing. Another example is the impressive growth of
the cheap, no frills, aircraft carriers leading to congestion
in airports. Increases in air traffic has had the conse-
quence that 16 of the European Union’s main airports
recorded delays of more than a quarter of an hour on
more than 30% of their flights [3]. Altogether, these
delays result in consumption of an extra 1.9 billion litres
of fuel, which is 6% of the annual consumption [4]. In
fact, congestion is increasing in all networks. In Europe
7500 km, i.e. 10% of the road network, is affected daily
by traffic jams and 16,000 km of railways, 20% of the
network, are classed as bottlenecks. The cost attributed
to congestion is estimated to be EUR 80 billion
a year, which is approximately 1% of the community’s
GDP [3].
The problems associated with congestion have been
noted for some time and its threat to economic
competitiveness was already identified by the European
Commission in 1992 [2]. The effects of increasing
emissions, which increase more than linearly with
congestion and the effect of increasing traffic on road
safety, have led to significant effort in trying to redress
the problem. In terms of transportation policy docu-
ments, the European transportation White Paper of
2001 [3] explicitly recognises that the success of trans-
portation has created significant negative side effects.
Sustainable mobility has become a central notion in
community policy-making. This White paper highlights
the need to integrate environmental considerations into
transportation policy (and other community policies).
The White paper proposes some 60 specific measures
to be taken at community level under the transportation
policy. It includes an action programme extending until
2010 with clear milestones. Detailed proposals are based
on a number of guidelines. For example, in response
to the general deterioration in the quality-of-life of
European citizens arising from growing congestion in
towns and cities, the Commission proposes to place the
emphasis on the role of public transportation in inner
cities and urban areas. An important policy objective is
the ‘‘exchanges of good practices aiming at making better
use of public transportation and existing infrastructure’’.
The Commission states that a better approach is needed
from local public authorities to reconcile modernisation
of the public service and rational use of the car. This
paper deals with a tool that can be considered as
applying a best practice, namely the introduction of
benchmarking as a way to help in the process of making
improvements. This methodology is already used in
numerous industrial sectors and can be useful in the
public transportation sector but in order to do so, the
question of how to measure the performance of a service
oriented industry, needs to be addressed.
As a result of the most recent policy initiatives, public
transportation operators face a wide range of business
pressures including working in a competitive and less
regulated market [12]. Local transportation operators
rarely have the right to assume that they will be the sole
operator throughout their territory. Operators respond
to these pressures either through making improvements
(thus, remaining viable or becoming stronger) or by
inertia (requiring stronger intervention at a later point,
or failing).
2. The role of benchmarking in continuous improvement
Improvements in a production of a good or service
can either occur on a one-off basis, or they can be
part of a permanent, value-adding process. Continuous
improvement can be described as an ongoing process
within an organisation, and is really a corporate frame
of mind. The organisation recognises that it must evolve
to meet the demands of a changing environment and
market, and its customers (external and internal) have
ever-increasing expectations. In response to the dynamic
environment, the organisation seeks out its weak points,
and implements actions to redress them so as to achieve
better performance. This process is more likely to assure
customer satisfaction, and minimises opportunities for
competitors.
 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
Continuously improving processes within a firm are
inclusive and should lead to team building within the
organisation by its all embracing improvement and will
consequently involve all layers of management, admin-
istration and staff. It is also a cyclical process, which
challenges complacency. At no point does the organi-
sation consider that it is perfect, so that even after
a full iteration, it presumes that more improvements
can be made.
The process is illustrated in Fig. 1. This shows a nine-
stage model which begins by identifying factors which
are critical for the success of a business and measure-
ment of relevant indicators (stages 1e3), proceeds to
analysis of the operator’s performance and business
(stages 4e6), and finally develops, implements, and
monitors affirmative action plans (stages 7e9). The loop
returns to stage 1, and there may be a number of loops
operating in parallel.
If one is comparing the performance monitored
during the continuous improvement process over time
or with other units, this is called benchmarking. The
concept of benchmarking originates from land survey-
ing, being a point of reference of known altitude against
which other subjects that are assessed. Frederick Taylor
adopted this term in his scientific management practices
and a benchmark became the efficiency standard by
which a job could be performed [17]. Tarricone [18]
describes how the total quality management subsequent-
ly picked up benchmarking as a comparative process to
judge quality. Definitions of benchmarking vary. Key
themes include measurement, comparison, identification
of best practices, implementation and improvement.
Collective elements in defining benchmarking should
refer to analyses of processes, products, services or
performances, compared within or between organisa-
tions with the objective of assessing an organisation’s
standards, obtaining information for self improvement
and implementing changes to affect improvement (for an
extended description see also [5]). We define bench-
marking as a structured approach to collecting and
sharing of data, information, ideas and methods with the
1. Define and agree on
critical success factors
of business
9. Monitor performance
8. Plan and implement Scope of
improvements paper
7. Learn best practice
from benchmark partners
6. Review relevant 5. Identify areas to be
business processes improved
Fig. 1. The continuous improvement process e a nine-stage model [10].
115
objective to come to comparisons that will be of mutual
benefit amongst all groups involved. As such, bench-
marking is used as a tool to identify key improvement
opportunities by comparing relevant performance in-
dicators. In the process industry, this methodology is
generally accepted and applied in combination with the
so-called ‘total quality management system (TQM)’.
This paper is concerned with the development of the first
three stages of benchmarking for local public trans-
portation systems.
For operators who continue with the later stages of
benchmarking, this methodology offers the opportunity
to compare performance with companies operating in
the same and in different areas of competition. These
areas of competition may relate to the degree of
regulatory control, the type of service area covered by
the company and the different modalities used by the
company. Should a company feel unable to benchmark
with competitors (potential or actual), benchmarking
will be restricted after stage 3; the company may proceed
with internal benchmarking by planning and implement-
ing improvements (stage 8) followed by repeat measure-
ments over time (stage 9). In this case, stage 8 will not
benefit from external inputs (stages 4e7).
In addition there are also opportunities to benchmark
with companies in a different industrial sector in some
areas e.g. road haulage and buses in the area of vehicle
maintenance. This provides an alternative focus for
comparison at stage 4.
In general, three progressive levels of benchmarking
can be defined which involve increasing commitment
and co-operation (see Table 1). Self-assessment covers
stages 1e3 of the continuous improvement process,
whilst comparison with data from other organisations
takes a user to stage 6. Beyond this, co-operation with
other companies (inside or outside the industry) can lead
to the full cycle being completed. This means that this
level includes all nine stages.
Thus, benchmarking is about the measurement and
comparison. It helps the company move from percep-
tion to facts, and provides a holistic view of the
organisation. It is important that relevant, measurable
and comparable indicators are selected which can
describe the processes of the organisation on behalf of
and in consultation with, the users of the process.
2. Develop indicators to Using indicators in this way, benchmarking allows
measure performance the organisation to compare its performance with that
of its peers so that:
3. Measure indicators for
an individual operator
 areas of good performance i.e. strengths, can be
identified;
4. Compare performance
with that of others  areas of potential improvement i.e. weaknesses, can
be identified;
 the scale of the potential improvements is quantified,
so that the potential added value from change can be
estimated.
116 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
Table 1
Benchmarking levels [8]
Level Category Elements
I Self-assessment  Measure your own company’s
performance (over time).
II Comparison  Compare your performance with a
database of anonymous indicator values.
 Identify improvement areas and best
‘‘standards’’.
III Partnering  Work with relevant partners, perhaps with
some outside the direct business sector.
 Exchange confidential information.
 Learn best practice and the means of
implementing the necessary changes.
 Ideally, this should be a two-way process.
The measurement process provides a baseline data set
for improvements and for target setting, which is
understood throughout the organisation. The areas for
potential improvement will be identified, target values
(perhaps with intermediate milestones) will be set, and
there will be an estimate of the potential added value the
operator is able to gain. The team building achieved in
the benchmarking activity will provide the platform for
the action teams, which must achieve the improvements.
If successfully done, there will be both consensus and
a corporate motivation for change.
3. Towards a methodology of benchmarking in public
transportation in Europe
A literature review on benchmarking in public
transportation as part of the EU project EQUIP
(Extending the Quality in Public Transportation),
examined 153 project-reports, publications and other
related documents, identified that much work within the
EU and international bodies had been carried out in
terms of identifying specific indicators for performance.
This review produced a list of over 400 indicators that
had been recommended for use in public transportation
to measure its performance [7]. However, whilst there
were many indicators recommended for carrying out
benchmarking there were, at that time, only a few
documented experiences of benchmarking activities in
public transportation.
In Europe, the Commission had made an early step
to promote the performance of the public transportation
system by launching a project to consider the existing
organisational structures for public transportation
operations in European countries (ISOTOPE (Improved
Structure and Organisation for Transportation Oper-
ations of Passengers in Europe)). The ISOTOPE project
[11] team developed a set of key performance indicators
for different public transportation operators and differ-
ent cities to compare their performance but were unable
to complete benchmarking activities due to difficulties in
data collection.
The next step of the Commission’s plan to support
the actors in public transportation was the funding of
the QUATTRO (Quality approach in tendering urban
public transportation operations) [14]. This project
described a number of quality improvement tools for
public transportation including benchmarking, which
was concluded to be a key tool for operators to improve
their performance.
In addition to benchmarking transportation services,
the European Commission funded the development of
techniques for assessing local transportation systems as
a whole. During 1998, a pilot benchmarking exercise for
local passenger transportation systems was set up to
work within 15 urban areas. The so called PILOT project
[13] was a benchmarking exercise for local transportation
systems covering all transportation modes at the city
level. This project ran alongside the EQUIP project that
was started in January 1999. EQUIP concentrated on
the development of a benchmarking handbook for self-
assessment of internal quality performance by public
transportation operators and the establishment of
a centralised database for comparison. The study was
completed in July 2000 with the publication of a hand-
book for five different types of public transportation and
a computerised supporting system. The experiences of
EQUIP form the basis for this article. Subsequent to
EQUIP, a DG TREN project named BEST (2001) is
drawing together information from the different bench-
mark systems in Europe.
Parallel to the projects funded by the European
Commission, CEN (the European Commission for
Standardisation) started to develop a list of quality
indicators for public passenger transportation. In March
1999 ‘‘CEN TC 320 Working Group Five’’ issued draft
recommendations for the definition [1], targeting and
measurement of service quality which was finalised and
summarised in the document N73: ‘‘Service quality e
definition, targeting and measurement’’. The standard
itself is not intended to be compulsory e the purpose of
the CEN recommendations was to standardise indicators
and terminology. This document has been incorporated
into individual member countries’ National Standards
where these exist.
Besides these, several international and local bench-
marking initiatives were launched in the field of public
transportation (see also www.transportationbenchmarks.
org):
 CoMET and NOVA are two different, but related,
projects which were designed to compare the urban
transportation systems of various cities from
around the world. This involves using insight taken
from other metro systems in order to attempt to
improve internal working cultures and more specif-
ically to improve on service levels. The initial step
of these studies was begun in 1982 as a trilateral
 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
study between Paris, Hamburg and London. The
CoMET benchmarking group began in 1994 and
was deemed a success upon completion in 1998.
The NOVA group is effectively a spin-off from
the CoMET benchmarking group and is currently
comparing medium sized metro systems. The
CoMET and NOVA schemes have attracted cities
from all over the world, not just in Europe and
include the three cities of London, New York and
Tokyo.
 A project aiming at the developing and realising
competitive transportation serviced in Finland was
the KiPa project in Finland which relied on external
experts entering the company to collect the relevant
information to form an assessment of overall quality
and efficiency.
 The BOB project was established as part of the
Benchmarking of Environmentally Sustainable
Transportation (BEST) initiative in order to test
the recommendations of the BEST network. The
railway working group collected data upon punctu-
ality and contractual relationships between author-
ities and operators and focused particularly upon the
conditions for passenger growth and institutional
organisation. National railway operators predomi-
nantly from Europe were involved in the bench-
marking working group, with some taking an
observer status. There was also input from the
Japanese National Railway (JNR) operator Kyushu
as a non-European ‘‘best practice’’ railway.
 The Metis-Conseil benchmarking: This project was
a benchmarking exercise undertaken in the Emilia-
Romagna region of Italy. The study was conducted
by the Metis-Conseil consulting group and focused
upon benchmarking the performance of public bus
and train transportation modes.
 The Scandinavian BEST Benchmarking Survey has
been running since 1999 and was set up by the
Stockholm public transportation authority ‘‘SL’’. It
initially involved the four Nordic capitals of Oslo,
Stockholm, Copenhagen and Helsinki and took the
form of a survey of 36 questions based on 10
categories carried out in spring 2000.
An interesting development is the integration of
benchmarking elements in contracts between authorities
and public transportation operators, if public trans-
portation services are subsidised. Examples in Austria
and Germany can be named as [16]:
 The Verkehrsverbund Ostregion (Vienna region,
Austria): passenger satisfaction surveys are carried
out and benchmarked within 164 regional bus lines
to award the best in the area.
 Die Grazer Stadtwerke AG (Styria, Austria) signed
contracts with sub-contractors who are running
117
public transportation services within the city of
Graz, if operators do not meet the threshold values,
a penalty is foreseen.
 The Niederösterreichische Verkehrsorganisation
(Lower Austria/Austria) benchmarks the regional
express bus system called ‘‘Wieselbus’’ with the help
of mystery shoppers and publish the results in local
newspapers.
 The Bayeriche Eisenbahn GmbH (Bavaria/Germany)
benchmarks the Bavarian regional railway traffic
with the help of mystery shoppers and online recor-
ding (punctuality), a reduced payment of a maximum
of minus 5% of the agreed subsidy can be considered
as stated in the contract with the operators, if pre-
defined thresholds are not met.
 The Verkehrsservicegesellschaft mbH Schelswig
Holstein (Germany) requests monthly based bench-
mark reports from all public transportation oper-
ators under contract, a reduced payment of a
maximum of minus 5% of the agreed subsidy can
be considered, if predefined thresholds are not met.
 The Verkehrsgesellschaft Mecklenburg Vorpom-
mern (Germany) is carrying out a combined cus-
tomer satisfaction survey and mystery shopping
survey. If thresholds were not achieved, the maxi-
mum penalty is 5% of the total subsidies given.
Benchmarking becomes more and more an estab-
lished tool in economic theory to improve and assess
processes in comparing indicators between production
or service entities such as accounting, information
systems, banking, pharmaceutical, automotive, human
resources, procurement or telecommunications and
(inter-) national benchmarking networks are established
(see for example www.benchnet.com, http://www.
benchmarkingnetwork.com/ or www.benchmarking.de).
There is no indication, why the field of public trans-
portation cannot benefit from a standardized and
institutionalized tool as well, but there is an indication
of too less competition and fear of submitting confiden-
tial data to third party in this sector, what decelerates
the widespread implementation of such tools. For
EQUIP, therefore, the starting point was to analyse
the indicators identified by the literature sources to
develop, with the industry represented by the EQUIP
Network (as described in Section 3.1), appropriate and
useful measures of internal efficiency.
3.1. Internal efficiency public transportation
performance indicators
During the period in which the indicator list was
rationalised, each of the EQUIP Consortium members
established national workshops of operators and au-
thorities interested in benchmarking. The feedback from
the entire national EQUIP workshops was analysed and
118 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123

Table 2 easy as possible for users. It was agreed that company

The clusters of quality indicators [8] profile (examining size, organisation) and external
Cluster Cluster title Number of influences on operator, would be key factors in any
indicators in cluster choice of benchmarking partner e so this related to the
1 Company profile 21 first two clusters of indicators. The relevant criteria
2 External influences on the 13 varied according to the circumstances of the operator
operators
3 Revenue and fare structure 9
and the areas that they wished to benchmark.
4 Asset/capacity utilisation 8 The definitions of the indicators enabled e and
5 Reliability 5 emphasised e the collection of comparable data. The
6 Production costs 3 handbook offered guidance in selecting and measuring
7 Company performance 4 appropriate data over appropriate periods of time. By
8 Technical performance 6
9 Employee satisfaction 12
specifying the actual date for the measurement period,
10 Customer satisfaction 7 the operator has a powerful tool for making ongoing
11 Safety and security 3 internal comparisons, as well as explaining unusual
Total: 91 differences between companies. Whilst it is essential for
a benchmarking process to use comparable data, it was
recognised that institutional and other differences (e.g.
confirmed that clusters were useful. Furthermore, there size of service area, size of company, type of service
were suggestions for improvement. For instance the area) can give operators different values in different
indicators in the environmental impact clusters were environments. Therefore, it is important to be able to
deleted or moved to other clusters. A company profile explain differences in the absolute indicator values. For
cluster was introduced with the object of grouping instance, it makes a huge difference if an operator only
together a range of operators. Some indicators were runs one bus line in the countryside based on 1 trip per
removed from the proposed handbook, either because 2 hours due to the low population density, or if
they were considered too difficult to measure or they a company operates a line in a metropolitan area during
would not contribute significantly to the aims of evening peak hours.
benchmarking within the EQUIP project. An underly-
ing objective was to create a handbook of high-level 3.3. Collecting good data
indicators which would reduce the total number of
indicators by retaining the most important indicators for The three main characteristics of good quality data
operator’s performance rather than include too many are the collection of the same measurements (i.e. using
indicators or to include those which were subsets of the same definition); ensuring that there is no bias in the
a single indicator. At the same time, new indicators were measurements; and taking measurements with sufficient
added in order to satisfy the feedback from the EQUIP precision. Using self-assessment meant that each of
network workshops. Table 2 shows the cluster titles and these three aspects required careful specification. This
the number of indicators per cluster that were included section describes how the EQUIP Handbook ap-
in the EQUIP Handbook. proached this. Definitions were made very clear in the
EQUIP Handbook. Provision was made for highlighting
3.2. How to present the indicators: the EQUIP common and system definitions that were used regularly
Handbook in the handbook. Electronic versions of the handbook
transferred numbers entered into the common elements
A handbook on benchmarking was developed as section to relevant indicators throughout the handbook.
a suitable tool for local public transportation operators In the Methodology and User Guide which accompa-
in diverse operating environments, whether rural, small nied the handbook, stress was placed on the need to read
and regulated or urban, large and subject to competi- the definitions carefully. This was important in the inter-
tion. It was to be used for self-assessment (level I national context since conventions for management
benchmarking) and for comparing with other operators information measurements varied significantly not
using an anonymous centralised database (level II only between companies but also particularly between
benchmarking) and was suitable for internal, national countries.
and international benchmarking. Although the level of The handbook authors began by asserting that there
application varied according to the individual indica- were two possible routes for collecting data for
tors, e.g. cost related indicators are more suitable for benchmarking: 100% records or sampling. These made
internal and national benchmarking than ones which use of two possible routes for collecting data: existing
relate to institutional factors. records or new recording systems.
The EQUIP Handbook, being based on self-assess- Generally, two categories of survey were identified,
ment, sought to make the process of benchmarking as which themselves can be sub-divided into (i) technical
 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
and performance data (off-road and on-road data)
which is generally objective, and (ii) opinion survey data
(passenger and employee) which is necessarily subjective.
It was suggested that off the road performance data were
to be found in licensing records, company accounts,
internal management accounting information and mar-
keting information whilst on the road data requires
surveys of service provision.
Opinion survey data, of course, are subject to cultural
variations and personal variations. The attitudes and
tolerance of individuals may even vary on different oc-
casions. Passenger data can be collected and completed
on the vehicle with or without the assistance of the
surveyor. It was suggested that employee data are best
gathered by distributing the form with wage slips and
returning them to a box (or boxes) located conveniently
at the workplace.
A collection using a 100% record as described above
is straightforward. If 100% record collection is not
possible, whether for cost or practical considerations,
then sampling can be used. But it is important that the
sample should be unbiased and representative of the
relevant population. For samples involving opinion
surveys of employees and off the road technical and
performance data, samples can achieve a balanced view
of the company by one of two methods. First, a sample
can be taken in proportion to the size of the employee/
vehicle population by ranking vehicles/employees by
date of entry into the company and then sampling every
nth vehicle/person to obtain the required sample size.
Alternatively, for vehicles a sample may take the last 100
vehicles monitored by the company.
Opinion survey data from passengers should reflect
variations in age, gender, ethnic, socio-economic status
and level of mobility. Passenger numbers and compo-
sition are affected by time-related factors such as time
of day, week or season. On the road technical and
performance data are subject to these variations in
passenger numbers as well as other traffic conditions,
e.g. congestion.
A critical element of sampling is to ensure that the
sample size is large enough to be representative. It is
preferable to collect a random sample because it gives
rise to good data qualities: this means that the response
or characteristic of one member of the sample is not
affected by those of the other members. EQUIP
recommended that if it is not practicable to obtain
a genuine random sample, a representative range of
conditions must be sampled. So far as sample size is
concerned, EQUIP made the suggestion, for indicators
with a yes/no response (e.g. whether a departure of
a vehicles is more than 5 min late) was that the minimum
sample size should be 400. For indicators where the
answer is coded as a number (e.g. the coded responses
from an opinion survey) either the minimum sample size
should be 400 or the sample size should be 100 and
119
estimation made of its variability in order to determine
whether the sample needs to be larger. These numbers
were derived from statistical theory and based upon the
construction of confidence intervals.
As the quality indicators cover all aspects of quality
in the field of public transportation (see Table 2), much
effort has to be taken to collect data on all 91 indicators.
There is an option to collect on only a selection of
indicators and benchmark them with benchmarking
partners or a data base. Therefore, the indicators are
clustered into 11 topics, 9 of them are flexible in their use
for benchmark. Additionally, there are 2 clusters called
‘‘company profile’’ and ‘‘external influences on the
operator.’’ These clusters are including indicators to be
able to categorize the company and select comparable
benchmarking data sets/partners, if the operator wishes
to compare between operators. If one operator prefers
self-assessment over time, these two clusters can be
skipped.
It must be highlighted that, despite this methodolog-
ical justification, the data collection is always a weak
element in the benchmarking of public transportation.
This was already a concern in the ISOTOPE and
QUATTRO projects in the past (see Section 3.1).
However, we wanted to avoid this pitfall in the EQUIP
project, in Section 4 we indicate that our carefulness in
methodology did not prevent the lack of confidence in
the collected data set.
3.4. Use of the EQUIP Handbook
Against a background of strict confidentiality whereby
the EQUIP Consortium undertook to hold all data
confidentially, 49 operators and 21 authorities and other
interest groups all from EU countries have used the
EQUIP Handbook. There were also contacts with
accession countries like Romania. For all of these
partners, there were strict terms and conditions for
the use of these data. A strict confidentiality was
guaranteed.
Furthermore, there are good contacts with different
representatives of the public transportation industry.
However, in contrast with the most recent knowledge
and experiences in other sectors, there is almost no co-
operation with UITP (the international federation of
transportation operators) because UITP was remark-
ably not convinced that their members had an interest in
performance measurement.
Whilst benchmarking is a key step in the continuous
improvement process it will not add value in itself; it can
only be a catalyst for change. The true value is only
added through achieving real improvements. Whilst this
was beyond the scope of the study, some recommenda-
tions are offered which have been drawn from the
120 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
experiences of users of the handbook and members of
the Consortium:
 improvements must be linked to clear and achiev-
able benefits;
 potential improvements must be prioritised accord-
ing to value-added or ‘‘must-do’’. This is particularly
important in the first iteration, when many possible
improvements may be identified;
 performance improvement teams should be estab-
lished with a well-defined brief and working method;
 there needs to be adequate committed and dedicated
resources, backed by commitment from manage-
ment and problem-solving support;
 where possible, partnerships with peers to impart
knowledge, experience and techniques should be
involved as this gives mutual support to move forward;
 during the process of change, make ongoing mea-
surements and retain the possibility for fine-tuning
and adjustment;
 monitor the implementation and results for a suffi-
cient period so that quantifying the benefits and
costs can be made. These both provides feedback to
the improvement processes, and will help to sustain
commitment from the stakeholders.
4. Application of the handbook for future
benchmarking activities
Operators expressed a willingness to participate in
benchmarking activities. The national workshops were
regarded as an opportunity to overcome a sense of
isolation when embarking on a new exercise.
Lack of confidence in the completed data set
manifested itself in a number of ways. Companies may
make excuses for poor performance by suggesting that
it is not possible to compare with other operators.
Experience showed that once this problem was over-
come, excuses are reduced and plans for improvement
are made [15]. More basically, companies recognise that
there is a lack of knowledge in benchmarking and that
education and support are needed.
The greatest problem facing operators in the EQUIP
Network was the lack of resources such as time and
manpower, to establish the necessary systems to collect
and record data for the handbook. This issue was
present in all types of companies, whether large or small,
privately or publicly owned. The impetus for assisting
the companies could come from several institutional
sources, e.g. support from national governments,
support from national and international organisations
that represent the interests of public transportation
operators, and support internally from decision makers
within the organisation. The lack of manpower resour-
ces could be resolved by the external completion of the
handbook. Whilst the handbook was designed to enable
self-assessment, i.e. it contains all the information that
an operator should require in order to complete the
handbook. This does not preclude an external agency
carrying out the task. The assistance could cover some
or all aspects of the handbook, i.e. access to internal
systems, opinion surveys, on-road and off-road technical
surveys.
Lack of resources also related to operator involve-
ment in pre-existing benchmarking activities. Operators
faced problems in collecting two sets of data: even
indicators with the same title may not have the same
definition and sampling criteria. The temptation to
‘‘misuse’’ data is understandably, high. In order for
benchmarking activities to progress, it is clear that
agreed standards must be established and utilised. The
practicalities of maintaining these standards mean that
external support would again, be advisable.
Operators faced the problem of incomplete data sets
and therefore unable to make much progress. Whilst
users of the handbook were encouraged to complete as
many indicators as possible in order to maximise the
benefit from the handbook, as it was planned to restrict
bilateral sharing to indicators completed by both
parties. Clearly, the solutions of increased resource
and support would help to overcome this problem.
Although operators were keen to comment and
discuss the content of the EQUIP Handbook, there
was a reluctance to invest resources in a pilot project,
since it was known that the final handbook would
maybe substantially different from earlier versions. This
should be regarded as a short-term problem e but the
wealth of information that has been assembled in the
study should be capitalised upon by moving forward
with the handbook which has achieved some consensus
in trans-European common definitions.
Throughout Europe there is a trend towards greater
privatisation and more competition for public trans-
portation services. This is reflected by rapid changes in
the character of the companies. During the course of
the study there were numerous instances of company
reorganisation in the private sector. Such activities may
be seen as initial obstacles to benchmarking, e.g.
company accounts and other data may be reorganised,
making it difficult to access relevant data, and to make
internal comparisons over a period of time. However,
reorganisation could be regarded as an opportunity to
introduce new systems such as those required for
benchmarking.
The confidentiality of financial and other delicate
information is very important and was guaranteed in the
EQUIP project. All confidential information that could
be placed on a public database must of course be in
an anonymous format. Strong measures must be taken
to overcome perceptions and problems relating to
confidentiality.
 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
The EQUIP Handbook has begun the cyclical pro-
cess by allowing operators to identify areas that need
improvement; the next stage is to develop the method-
ology that will enable the improvements to be made,
thereby achieving the real objective of improved
competitiveness. This would cover organisational issues,
e.g. the separation of manpower between the identifiers
of the areas for improvement and those who plan to
make it happen [15].
In summary, the basic requirements for a successful
self-assessment benchmarking exercise in public trans-
portation include:
 motivation of the operators to put effort into
participating in the process. There should be evident
benefits that are easily available and noticeable from
the use of the tool;
 clear definition of how the benchmarking process
should be accomplished as a self-assessment process;
 clear definition and ways of calculating values of the
indicators that measure the performance of the
operator;
 pushing the benchmarking tool by compulsory use, if
receive funding for public transportation services
based on a tendering process.
5. Conclusions and recommendations
5.1. The results
It was originally envisaged that benchmarking would
become a powerful public tool utilised by a well-
established network that would generate good quality
data. It was also hoped that a potential sponsor such as
the UITP would welcome the initiative and support the
development of the process. In light of demand from the
industry, it was expected that the EQUIP Consortium
would generate further initiatives. However, the reality
is a little different. Based on the experiences with field
tests involved, operators state that the EQUIP Hand-
book is a good product, but the UITP has not been in
a position to take ownership of the products and no
active proposal is in place. The motivation for their
absence is the fact that the transportation market is very
dynamic due to EU policies that push forward the
processes of privatisation, deregulation and tendering. It
is perceived that these changes need highest priority in
the available UITP management capacity.
Additionally, few operators have been able to commit
the resources necessary to generate data of a suitably
high quality. Against this background, there is interest
by operators in benchmarking and the value of
networking has been established. Although it was
initially difficult to convince people to use the EQUIP
Handbook, the awareness raising and rapidly changing
121
environment of the public transportation operators, is
creating a common understanding for the need for
performance improvement.
5.2. The future
The main challenge which remains is to mobilise the
industry to embrace benchmarking. For operators
proposing to take benchmarking seriously, there are
still many unanswered questions: (a) what resources
will they need to commit? (b) To what extent are they
interested in continuous improvement? (c) How can they
make benchmarking part of their business? (d) Who will
champion the initiatives required?
The UITP is identified as a possible common access
point to international benchmarking for public trans-
portation operators and as the co-ordinator of national
associations which in turn could be responsible for
national networks.
One of the clear outcomes of EQUIP was the
necessity to move towards the institutionalisation of
the benchmarking process. It was clear that effective
benchmarking must be a permanent process in order to
increase quality and competition in the public trans-
portation sector. Similarly, a permanent process needs
an institutionalised framework with clear responsibilities
for managing the benchmarking exercise. Finally, a re-
liable and independent co-ordinator is needed to over-
come the competitive characteristics of the sector.
However, the picture of the actual results is not
completed yet, a number of products from the study
strengthen the argument for institutionalising the pro-
cess: the project has created a list of standardised
indicators to measure technical performance and
quality of output together with a documented method-
ology. Secondly, the collection of data starts the
population of the reference database and it is essential
that the expansion of the database should take place
based on standardised and comparable indicators.
Thirdly, the results can be used to benchmark dynamic
changes and to support the tendering process for
supported services.
A number of products from the EQUIP project
strengthen the argument for institutionalising the
process:
 the EQUIP project has created a list of standardised
indicators to measure technical performance and
quality of output together with a documented
methodology;
 the collection of data commences the reference
database and it is essential that the expansion of
the database should take place based on stand-
ardised and comparable indicators;
 the set of indicators can be used to benchmark
dynamic changes over time within one company,
122 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123

this procedure avoids the problem of submitting
confidential data to a third party. Operators who
took part in the Equip pilot benchmarking exercise,
expressed interest to repeat this exercise after
a certain period;
 the results can be used to support tendering
processes in the field of public transportation. First
approaches can be observed on national and local
level, where benchmarking is beginning to be an
obligatory element in contracts, if tendered public
transportation services. E.g. the authorities of the
Austrian provinces of Salzburg and Upper-Austria
used the Equip project results as a starting point to
integrate an obligatory assessment scheme based on
the EQUIP Handbook [16]. As the interest of the
authorities lies primarily with the customer’s satis-
faction, a selection of appropriate indicators was
used. To receive comparable data sets in advance,
a pilot test was carried out in the area on six
representative bus-lines.
5.3. Developing a benchmarking network
The development of a successful network of oper-
ators and authorities is an important part of the
institutionalisation of benchmarking. This is a lesson
that can be learned from the process industry where
there are extensive experiences for many years with the
application of benchmarking. Various lessons specifi-
cally for the public transportation domain were learnt
from the experience of the EQUIP Network [6]. Mutual
support in benchmarking is extremely valuable, as it is
important for operators (especially small sized) that
they do not work in isolation. Language can be a most
serious impediment to benchmarking and so native
languages, using the correct technical language, must be
used for effective benchmarking. It has been shown that
formal networks generate informal networks and that
this sometimes creates new business opportunities as
well as new business contacts. The importance of an
independent trusted party as the holder of the data has
been firmly confirmed.
The EQUIP international network has brought
together operators, authorities, user groups and repre-
sentatives of users. However, institutionalisation is not
straightforward and five scenarios have been identified
by EQUIP as possible models for providing the very
important networking processes required for the suc-
cessful institutionalising of benchmarking. These are
shown in Table 3.
Whilst there has been no further international
research specifically based on the EQUIP outputs, there
have been a number of national studies on benchmark-
ing carried out using EQUIP indicators as a basis. This
would fit into category 4 of Table 3.

Table 3
Possible scenarios for the institutionalisation of benchmarking [9]
Scenario Advantages Disadvantages
1 International interest group  Standardised data collection on an  Data collection must be done by operators
international level themselves
 Easy access to the database
for public transportation operators
 Low cost for benchmarking exercise
2 One private company at  Indicators and handbook will be  Operators have to pay for a commercial
international level developed further service
 Very convenient for public  The company could be a monopoly
transportation operators
 Professional management
3 National associations  Easy access to the database for public  If indicators are developed further they are
(e.g. chamber of commerce, transportation operators no longer comparable at an international level
quality groups, public  Low cost for benchmarking exercise  Data collection must be done by operators
transportation consortia) themselves
 Low resources
4 Several private companies  Developing tailor-made indicators  The indicators are no longer comparable at an
on national level for clients international level
 Very convenient for public  Operators have to pay for a commercial service
transportation operators
 Professional management
5 A mixture of private  Combine the advantages of both  Distribution of responsibilities
companies and interest alternatives
groups  Data collection and analyses are
carried out by the private sector
 Good access for operators, the
interest group can change their
partners in the private sector if it is
worthwhile to do so
 H. Geerlings et al. / Journal of Cleaner Production 14 (2006) 113e123
6. Final remark
The final handbook produced by EQUIP is available
in a number of formats and languages. There are five
separate but compatible handbooks for each of the land-
based public transportation modes (bus, trolley bus,
tram/light rail, metro and local heavy rail) as well as
a short version of 27 ‘‘super indicators’’ to provide an
entry to benchmarking. The final handbook is composed
of two parts. Part I contains the Method, which covers
the background to benchmarking and the motivation for
carrying it out. Part II is divided into two sections: the list
of indicators is in a format that is ready to be completed
by the users. It is accompanied by a separate Guide to
Completion. These are available from the EQUIP-
co-ordinator, Dr. Corinne Mulley from Transportation
Operations Research Group from the University of
Newcastle upon Tyne (corinne.mulley@ncl.ac.uk).
Acknowledgement
The EQUIP consortium has the following partners:
ASM Brescia SPA; Erasmus University Rotterdam,
European Transportation and Telematics Systems Ltd,
Viatek Ltd Tampere; University Bodenkultur Vienna
and University of Newcastle upon Tyne (co-ordinator).
The authors would like to acknowledge the input of all
members of the Consortium and all members of the
established EQUIP network of public transportation
operators to the materials and outcomes discussed in
this paper.
References
[1] CEN document N73. Service quality e definition, targeting and
measurement. Brussels: European Commission; 1999.
[2] Commission of the European Communities. White paper on the
future development of the common transportation policy.
[COM(1992)494]. Luxembourg: Office for Official Publications
of the European Communities; 1992.
[3] Commission of the European Communities. European trans-
portation policy for 2020: time to decide. [COM(2001)370].
Luxembourg: Office for Official Publications of the European
Communities; 2001.
123
[4] Commission of the European Communities. Green paper towards
a European strategy for the security of energy supply.
[COM(2003)321]. Luxembourg: Office for Official Publications
of the European Communities; 2003.
[5] Dattakumar R, Jagadeesh R: A review of literature on
benchmarking, in: Emerald Group Publishing Limited (Ed.):
Benchmarking: an international journal, vol. 10, no. 3, pp. 176e
209(34), ISSN: 1463e5771, June 2003.
[6] EQUIP-Consortium. Awareness raising, publicity and dissemina-
tion plans. Deliverable D2. Brussels and Newcastle upon Tyne:
European Commission DG-TREN and TORG; 2000.
[7] EQUIP-Consortium. State-of-the-art in benchmarking of quality
of public transportation. Deliverable D3. Brussels and Newcastle
upon Tyne: European Commission DG-TREN and TORG; 2000.
[8] EQUIP-Consortium. The benchmarking handbook. Deliverable
D4. Brussels and Newcastle upon Tyne: European Commission
DG-TREN and the EQUIP Consortium; 2000.
[9] EQUIP-Consortium. Recommendations and conclusions. Brussels
and Newcastle upon Tyne: European Commission DG-TREN
and the EQUIP Consortium; 2000.
[10] Hanman R. Benchmarking your firm’s performance with best
practice. International Journal of Logistics Management 1997;
8(2):1e8.
[11] ISOTOPE. Improved structure and organisation for transporta-
tion operations of passengers in Europe. Final report by
ISOTOPE consortium on behalf of European Commission
DGVII, Luxembourg, Office for Official Publications of the
European Communities; 1998.
[12] OECD. Urban public transportation: evaluation of performance.
Paris; 1980.
[13] PILOT. The PILOT benchmark exercise. Brussels; 1999.
[14] QUATTRO. Quality approach in tendering urban public trans-
portation operations. Final report by QUATTRO. Consortium
on behalf of European Commission DGVII, Luxembourg,
Office for Official Publications of the European Communities;
1998.
[15] Rönnqvist T, Keskitalo J. Report on the EQUIP Draft
Handbook (confidential). Trans Control, Finland; 2000.
[16] Sammer G, Klementschitz R, Roider O. ÖeQuis e Öffentlicher
Busverkehr e Qualitätssicherung im Rahmen von Leistungsbes-
tellungen im Land Salzburg und im Land Oberösterreich,
Entwicklung eines Bewertungssystems, (Ensuring quality aspects
if tendering public-transportation bus services for the provinces of
Salzburg and Upper-Austria, creation of an assessment scheme),
Research report 4.1/2000, Institute for Transportation Studies,
University Bodenkultur, Universität für Bodenkultur, Vienna/
Austria; 2000.
[17] Sydney University of Technology Sydney. Benchmark; a
literature review. Sydney: Planning and Review unit; 1998,
reviewed 2002.
[18] Tarricone P. Best practices make perfect. Facilities design and
management 1998;17(3):50e2.