What is Benchmarking?
The process of benchmarking can be defined as “a structured approach to identify actions that lead to superior performance”. Benchmarking is not merely a comparison of performance data or a creation of rankings or league tables. Performance measurements, for example, deliver little benefit on their own, but they stimulate productive questions and lines of inquiry for more in-depth analysis and research.
The performance of a metro organisation is impacted by factors both within and outside management control. It is therefore very important to understand the effects of city context and external factors on metros’ performance. To account for this, the KPIs are often normalised in multiple ways, such as for the local cost of labour or the density of the city. Statistical analyses are used to provide greater understanding of the results, while time series analyses allow for trends in performance to be identified. This helps to highlight which members are changing their practices and what improvements are relatively achievable.
In the public transport benchmarking groups, there has been a strong focus on results that can be implemented, on performance improvement, and on informing strategy. Benchmarking stimulates lines of enquiry to identify best practices in operations and management, offer key insights that can drive strategy and policy and provide information to support better dialogue with city governments, regulators, and other stakeholders.
Key Benchmarking Principles
Fundamentally, the Community of Metros provide a forum for metro organisations to share their experiences and share information. The following key principles have led to successful benchmarking over 30 years:
- Collaboration – by giving and taking the good and the bad, members help each other improve
- Confidentiality – completely open data and information exchange within the groups and complete confidentiality to the outside, which overcomes political and commercial sensitivities (with anonymisation protocols enabling information to be carefully and properly used publicly where appropriate)
- Continuity – a long-term approach with an annual cycle is required; as one-time benchmarking studies are rarely successful because it can take several years and iterative cycles to achieve comparability and consistently
- Speed – the moderate size of the groups, the carefully managed scope of the work, and the online interactions enable fast results
- Independence – the groups are owned and run by the participants (with the TSC at Imperial College London acting as the administrator, facilitating the process and providing the research resources), enabling flexibility and ensuring that the focus is directed toward their highest priority needs and the areas that will produce the greatest benefits
Primary Benchmarking Elements
The main elements of the benchmarking work programme include:
- The Key Performance Indicator (KPI) system – the basis of the benchmarking process, designed to compare performance and identify lines of inquiry. COMET uses a KPI system with approximately 30 top-level indicators, which are designed to measure the overall performance of the organisation in six distinct areas: Growth, Learning & Innovation; Financial; Customer; Internal Processes; Safety & Security; and Environment. The design is comprehensive enough so as to represent all of the different parts of the metro business, yet concise enough to be able to be used effectively by management. The data items behind this set of indicators is collected to standard definitions by all members on an annual basis. In recent years, the KPI work has been supplemented by detailed customer-focused benchmarking based on an annual Customer Satisfaction Survey to metro users.
- Case studies – larger pieces of work where specific topics of interest to metros are studied. Case studies have a range of scopes but typically include an assessment of metro performance in the area across a number of items, and a deep dive into the practices metros adopt. The process includes detailed data analysis, questionnaires/surveys, and interviews with functional experts. Examples of case studies in recent years include Metro Reliability, Performance of New Fleet, Non-Fare Revenue.
- Peer-to-peer exchange between metros – metros have access to a forum to pose questions to each other on discrete areas and receive quick responses in return. This provides COMET members with convenient access to how other metros operate and can share this information widely within their organisation.
- Meetings – a fundamental opportunity for metros to hear the latest benchmarking results, insight and analysis and participate in structured discussion with other metros on their performance and opportunities. In-person meetings are usually supplemented by technical visits to metro systems for COMET members to see other metros’ best practices in action. Expert workshops are also organised to allow subject matter experts at metros to come together and discuss practices on detailed topics – recent examples have included CBTC Signalling and Cybersecurity.