Delivery
Using rankings to incentivise performance improvements
By Fenton Whelan & Will Anderson
Rankings can drive big improvements in performance. They can also generate disagreements and gaming. Understanding the nuances of ranking systems is key to using them effectively.

Used effectively, rankings harness competitive forces to incentivise higher performance

The World Bank’s ‘Doing Business’ ranks nearly all countries annually on ten ease of doing business topics. In part, the ranking seeks to harness competitive forces by “encouraging economies to compete towards efficient regulation”. For example China’s recent 32 place increase on the ranking was supposedly heavily influenced by India’s 30 place increase the year prior. Both nations achieved these increases by reforming parts of their business environment.

The 10 Ease of Doing Business topics used in 2019

1. Starting a business
2. Dealing with construction permits
3. Getting electricity
4. Registering property
5. Getting credit

6. Protecting minority investors
7. Paying taxes
8. Trading across borders
9. Enforcing contracts
10. Resolving insolvency


undefined

However rankings can also be prone to unintended consequences

The Doing Business ranking has some limitations. For example, it only uses data collected from the country's largest city (sub-national indicators are beginning to be incorporated for some countries). Additionally the ranking captures data mostly related to a specific size of limited liability company, excluding all other types of common business structures. These types of methodological limitations can lead to disagreements about whether the ranking accurately depicts reality. For example disagreements over the ranking’s methodological robustness arose as a result of Chile’s ranking wildly fluctuating scores between 2006-2017.


undefined

Limitations in ranking design can also lead to gaming. For example, the India edition of the Huffington Post recently suggested that India’s rapid increase on the Doing Business ranking was the result of “cracking the code” rather than substantive economic reforms.

Careful design may reduce disagreements and gaming

There are many ways to develop rankings, each with their own nuanced advantages and limitations (outlined below). Understanding these nuances can help ensure that rankings are used effectively.

Ranking system

Advantages

Limitations

First to last

Participants are ranked first to last (e.g. 40 districts listed 1 to 40)

  • Simple to understand

  • Intuitive to most people

  • Engaging and motivating

  • Small improvements still show up in ranking

  • Someone is always last, regardless of overall performance

  • Hard to correct for starting points

  • Misleading when the range is narrow

Quartiles or divisions

Participants are divided into a number of groups based on performance (e.g. top third, middle third, bottom third)

  • Simple to understand

  • Avoids ‘margin of error’ differences in data affecting ranking

  • Avoids signalling out individuals

  • There is always a bottom group

  • Performance within groups can be wide

  • Top quartile may still not be good

  • Improvements may not register (e.g. within top quartile)

Performance thresholds

Participants are scored based on whether they meet certain thresholds (e.g. above 90%)

  • Reasonably simple to use

  • Allows everyone to succeed

  • Difficult to use in systems where there is a wide variation in starting points

  • May not reward improvement for individuals who’s starting point is low

Based on directions

Participants are scored based on whether they are improving (e.g. improved by 10% this month)

  • Reasonably simple to use

  • Allows everyone to succeed

  • Rewards most recent improvements

  • May be confusing and misleading (very bad areas score highly if they are improving)

  • Becomes difficult to use once performance approaches its maximum

Based on targets

Participants are scored based on whether they meet targets, which may be different for each participant

  • Complicated and difficult to explain

  • Target calculation may be controversial

  • May entrench existing patterns of behaviour

  • Fairer and more realistic than performance thresholds where staring performance varies widely

Based on trajectories

Participants are scored based on whether they meet a trajectory, which may be different for each participant

  • Complicated and difficult to explain

  • Calculation may be controversial

  • May entrench existing patterns of behaviour

  • Allows high level of fine tuning to individual circumstances and variables



About the authors

Fenton Whelan founded Acasus. He has fifteen years of experience in public health and education development. Will Anderson supports business development and research at Acasus.

MORE INSIGHTS
Delivery
Using mobile technology to rapidly improve schools & health facilities
By Joseph Marsico & Will Anderson
|
21 February 2018
Mobile technology can play a vital role in rapid and effective improvements of schools and health facilities.
Learn more
Delivery
Using Delivery to launch reform
By Richelle George
|
14 December 2018
Launching reform isn’t easy; Delivery can help.
Learn more
Delivery
Shining the spotlights on solutions
By Fenton Whelan
|
20 May 2019
Focusing on solutions is far more powerful than focusing on problems.
Learn more
|