Using rankings to incentivise performance improvements

By Fenton Whelan & Will Anderson

On 26 April 2019

Rankings can drive big improvements in performance. They can also generate disagreements and gaming. Understanding the nuances of ranking systems is key to using them effectively.

Used effectively, rankings harness competitive forces to incentivise higher performance

The World Bank’s ‘Doing Business’ ranks nearly all countries annually on ten ease of doing business topics. In part, the ranking seeks to harness competitive forces by “encouraging economies to compete towards efficient regulation”. For example China’s recent 32 place increase on the ranking was supposedly heavily influenced by India’s 30 place increase the year prior. Both nations achieved these increases by reforming parts of their business environment.

The 10 Ease of Doing Business topics used in 2019

1. Starting a business
2. Dealing with construction permits
3. Getting electricity
4. Registering property
5. Getting credit

6. Protecting minority investors
7. Paying taxes
8. Trading across borders
9. Enforcing contracts
10. Resolving insolvency

undefined

However rankings can also be prone to unintended consequences

The Doing Business ranking has some limitations. For example, it only uses data collected from the country's largest city (sub-national indicators are beginning to be incorporated for some countries). Additionally the ranking captures data mostly related to a specific size of limited liability company, excluding all other types of common business structures. These types of methodological limitations can lead to disagreements about whether the ranking accurately depicts reality. For example disagreements over the ranking’s methodological robustness arose as a result of Chile’s ranking wildly fluctuating scores between 2006-2017.

undefined

Limitations in ranking design can also lead to gaming. For example, the India edition of the Huffington Post recently suggested that India’s rapid increase on the Doing Business ranking was the result of “cracking the code” rather than substantive economic reforms.

Careful design may reduce disagreements and gaming

There are many ways to develop rankings, each with their own nuanced advantages and limitations (outlined below). Understanding these nuances can help ensure that rankings are used effectively.

Ranking system	Advantages	Limitations
First to last Participants are ranked first to last (e.g. 40 districts listed 1 to 40)	Simple to understand Intuitive to most people Engaging and motivating Small improvements still show up in ranking	Someone is always last, regardless of overall performance Hard to correct for starting points Misleading when the range is narrow
Quartiles or divisions Participants are divided into a number of groups based on performance (e.g. top third, middle third, bottom third)	Simple to understand Avoids ‘margin of error’ differences in data affecting ranking Avoids signalling out individuals	There is always a bottom group Performance within groups can be wide Top quartile may still not be good Improvements may not register (e.g. within top quartile)
Performance thresholds Participants are scored based on whether they meet certain thresholds (e.g. above 90%)	Reasonably simple to use Allows everyone to succeed	Difficult to use in systems where there is a wide variation in starting points May not reward improvement for individuals who’s starting point is low
Based on directions Participants are scored based on whether they are improving (e.g. improved by 10% this month)	Reasonably simple to use Allows everyone to succeed Rewards most recent improvements	May be confusing and misleading (very bad areas score highly if they are improving) Becomes difficult to use once performance approaches its maximum
Based on targets Participants are scored based on whether they meet targets, which may be different for each participant	Complicated and difficult to explain Target calculation may be controversial May entrench existing patterns of behaviour	Fairer and more realistic than performance thresholds where staring performance varies widely
Based on trajectories Participants are scored based on whether they meet a trajectory, which may be different for each participant	Complicated and difficult to explain Calculation may be controversial May entrench existing patterns of behaviour	Allows high level of fine tuning to individual circumstances and variables

About the authors

Fenton Whelan founded Acasus. He has fifteen years of experience in public health and education development. Will Anderson supports business development and research at Acasus.

Strengthening school leadership to improve education in Peru

Improving school leadership is critical to raising educational outcomes.

Why not to build a Delivery Unit

“Delivery” is a set of approaches, tools, and guidelines used to help governments improve implementation.

The power of granular data systems