Benchmarking the Applications Development, and Maintenance and Support (AD/M) Environment can potentially provide significant insights into the performance of IT processes and identify how and where they can be most effectively improved. However, many organisations initiate a Benchmarking Activity, without first fully determining the objective of the benchmark, the benefits they hope to achieve, or establishing the criteria by which projects and applications will be selected and compared. This Chapter explores the reasons why organisations choose to benchmark, identifies the potential benefits of benchmarking, and highlights the pitfalls of failing to appropriately plan the benchmarking activity or introduce rigor into the benchmarking process to ensure its success.
Benchmarking involves the measurement and comparison of the performance and outcomes (products) from selected IT processes for the purpose of improvement, establishing a competitive position, and/or to provide input into management decision-making. Whilst many IT organisations routinely collect cost, effort, defect, and in some cases functional size data, they rarely go the extra step of turning this raw ‘data’ into ‘information’ that would facilitate change. Benchmarking is the activity that turns data into information by measuring current practices, comparing current performance to past performance, or peer performance, and interpreting the results. Usually organisations start by focusing on internal benchmarking to target areas for improvement before going the next step of comparing themselves to external business units or wider industry performance. However this is not always the case; it often takes the results of an industry comparison to identify the high cost of poor IT practices which then motivates management to rethink their AD/M strategy and start some internal measurement. As a result of a benchmark report, management may choose to reduce costs by outsourcing their IT Development and Support or alternatively by targeting internal processes for improvement. Benchmarking also enables organisations to assess their performance against their competitors, and evaluate the benefits and cost savings of investing in new tools, techniques or technologies.
In more recent years benchmarking has been progressively used as a means to assess and compare the cost-effectiveness of IT suppliers. Most large fixed term outsourcing contracts include clauses that financially reward or penalize the supplier, based on the supplier’s performance against an industry benchmark or an established client performance baseline. These bonus/penalty incentives are often priced as a percentage of the total worth of the contract and can result in payments of millions of dollars flowing either way. In some cases, as the contract expiry approaches, the contract requires an ‘independent’ organization to benchmark the supplier’s performance, prior to renewal. If the outcome of the performance comparison is positive then both client and supplier are encouraged to continue, but if the outcome is negative then it may result in contract cancellation. Given the ongoing cost of a benchmarking activity, the potentially high risk of incurring large payouts and/or contract cancellation, it is surprising how few organisations define a rigorous process around the benchmarking activity and develop an agreed ‘Terms of Reference’ for the benchmarking activity, prior to starting.