Don’t trip over the metric: a caution for monitoring risk appetite

January 2, 2019

In recent decades, the financial services industry has made great strides with respect to risk governance; risk management frameworks have become more mature, articulation of risk appetite has been enhanced and accountability for risk has improved through the use of objective metrics, namely Key Risk Indicators (KRIs).

 

While we endorse the use of KRIs themselves, a big caveat applies to the use of any metric. In the context of remuneration and financial advice, this caveat was skilfully articulated by Ken Hayne in his Interim Report for the Royal Commission into Misconduct in the Banking, Superannuation and Financial Services Industry:

 

Banks have sought to manage their staff by measurement. (...) Management by measurement assumes, wrongly, that measurement can capture all that matters in dealings between bank and customer. It cannot and does not. So much was illustrated most clearly in the financial advice cases considered by the Commission.[1]

 

Much like financial advice, risk is complicated and multifaceted. It would be erroneous to assume that a KRI could capture all that matters in the management of risk and they can present a number of potential concerns that need to be addressed. In the spirit of wanting to avoid any future Royal Commissions into what has gone wrong in the employment of KRIs in managing financial sector risk appetite, we discuss some of the key difficulties with these types of measurements and provide a simple list of dos and don’ts to keep you on track. 

 

Beware of the predicable consequences

 

In his fascinating book, “The Tyranny of Metrics”, Jerry Muller traces management by measurement to early efforts in Victorian Britain where it was used to improve school performance. Following this it was used for a range of things including for refining mass manufacturing, then by accounting professionals who began advocating managerialism in all walks of life, and it was even used in the Vietnam war when Robert McNamara forced the US military to begin body counting. Muller lists 7 common pitfalls of the use of metrics which he defines in two categories[2]:

 

Distortion:

 

1. Measuring the most easily measurable (or measuring the simple when the desired outcome is complex)

 

2. Measuring inputs rather than outputs

 

3. Degrading information through standardisation

 

Gaming:

 

1. Gaming through creaming

 

2. Improving numbers by lowering standards

 

3. Improving numbers through omission or distortion of data

 

4. Cheating

 

Distortion

 

The first three pitfalls need to be addressed in the initial design of KRIs, but cannot be entirely avoided given the complexities of risk as a concept. An example of the first pitfall is the measurement of minor IT incidents, where the main concern may in fact be to prevent a full-blown system outage. While there may be a correlation between the causes of both minor and large IT incidents, care should be taken not to assume that an 80% reduction in small incidents equates to a corresponding reduction in the risk of large incidents. Combatting this issue requires clear communication of what is being measured and why when a KRI is introduced or reported. 

 

Additionally, KRIs may directly or indirectly measure inputs rather than focussing on outputs. Examples include: staff turnover in risk departments, budgets for risk remediation and progress/delays in projects. Generally, the potential disconnect between risk and process inputs is well understood, so there may be less need for explanations provided that remuneration is not tied to these KRIs.

 

The third pitfall, degrading information through standardisation, invariably occurs when benchmarking organisations or staff against peers. The information is typically stripped of context, history and meaning, and is then presented as authoritative. No matter how obvious this is, the public, journalists and government officials keep comparing school or hospital results without adjusting for location or demographics. Likewise, regulators might be tempted to benchmark entities on risk metrics as a substitute for a deeper understanding of risk. Unfortunately standardisation cannot be completely avoided when using KRIs, hence providing a warning for these dangers is important: put those Ken Hayne quotes on the wall! 

 

Gaming

 

The gaming of metrics has the potential to take a big toll on our society when you consider the impact that creaming can have. Consider how a surgeon might be tempted not to operate on a very sick patient, so as to avoid impacting on their relevant statistics of success. Schools could prevent weaker students from sitting an exam in order to drive better results for the school, while inadvertently complicating the student's future education. This idea of creaming occurs naturally in certain contexts, but in any case, it should not be allowed to compromise risk appetite monitoring.

 

Lowering standards in order to make results look better than they otherwise are is always a temptation that can impact the success of KRIs. This should be less of problem for risk appetite monitoring where Management and Boards provide oversight to control the standards.

 

Improving metrics at the data collection stage, on the other hand, is more relevant because of the temptation to omit or misclassify information to improve the KRI outcomes. Consider the potential for a Risk Manager to skew information on incidents by relying on poor or subjective definitions of a ‘material incident’ (or whatever classification affects the KRI) and thus present more favourable KRI results. 

 

The final pitfall, cheating or fraud, is unfortunately an aspect that the financial sector is all too familiar with. In order to reach targets, we hear stories of sales staff fraudulently opening accounts and CFOs and traders fabricating accounting profits. Perhaps it is only a matter of time before we see fraud affecting KRIs too.

 

A straightforward way to reduce gaming is by never tying remuneration to metrics. If this is too hard to resist, the link should not be mechanical but subject to managerial review, and even this should be limited to the metrics that are the hardest to game. It is critical that Management never tie promotion, much less continued employment, to the meeting of KRIs. Boards should not express a zero tolerance for failing KRIs either, no matter how important the risk is. There may be a valid reason for a KRI moving into the ‘red zone’ and the KRI should be scrutinised to ensure that it is measuring the true risk in the first instance.

 

Getting risk appetite accountability right

 

Getting risk management right is critical for all financial sector organisations, which means expressing a risk appetite and reporting against it. Qualitative reporting remains too sterile and so some quantification is desirable[3]. Based on our experience, organisations can get the use of metrics right by sticking to the following dos and don’ts.

 

It is good practice to:

 

- complement your RAS with metrics (KRIs)

- seek to understand longer term trends in KRIs 

- holistically focus on the underlying risks, using KRIs as a guide 

- explain why some KRIs have entered the red zone

- replace KRIs that don’t work, even if this is discovered after entering the red zone

 

While it is seen as bad practice to:

 

- mechanically tie compensation to KRIs

- focus on meeting next quarter’s KRIs

- assume KRIs are objective and can replace subjectively rated risk register entries

- berate management for KRIs entering the red zone

- insist dysfunctional KRIs revert to the green zone before discussing replacement

 

Concluding thoughts

 

Monitoring risk appetite with quantitative measures is good practice, however, metrics should not be overused. The 7 pitfalls identified by Jerry Muller provide a theoretical basis to identify metric dysfunction and are equally applicable to discussions on remuneration in the financial sector (which we expect will heat up after Ken Hayne publishes his final Royal Commission findings in February 2019). With regard to KRIs, consider the 10 simple dos and don’ts that we have listed above to help ensure that KRI’s won’t be the hot topic at the next financial services Royal Commission.

 

 

 

[1] See Interim Report, Royal Commission into Misconduct in the Banking, Superannuation and Financial Services Industry, 28 September 2018, Volume 1, Section 9.2 “Management by measurement”, pp. 302-303, available at 

https://financialservices.royalcommission.gov.au/Pages/interim-report.aspx.

 

[2] See The Tyranny of Metrics, Jerry Z. Muller, 23 January 2018, Princeton University Press, pp. 22-23, available at https://www.amazon.com/dp/B076ZWW2MN. The book is a light read, running the reader through a vast array of the misuse of metrics through education, medicine, policing, military, business and philanthropy, providing clear analysis and useful suggestions for reducing the misuse of metrics.

 

[3] APRA’s emphasis on risk appetite may have been the driver of risk appetite quantification in the financial sector, as metrification is strongly encouraged by APRA supervisors consistent with CPG 220 “Risk Management” (available at 

https://www.apra.gov.au/sites/default/files/cpg_220_april_2018_version.pdf).

Paragraph 57 reads: “Where possible, risk tolerance would be expressed as a measurable limit (...). An institution may also define key indicators with thresholds around the risk tolerance”. Nevertheless, APRA’s prudential framework carefully avoids prescribing metrification, even putting the burden on management to avoid the predictable consequences, see CPG 220 paragraph 52: “(Risk quantification techniques) may not be appropriate for all types of risk. APRA expects senior management to assess the appropriateness of such techniques (...)” and paragraph 58 “APRA recognises that, for some risks, a qualitative risk tolerance may be appropriate”.

 

Share on Twitter
Please reload

Featured Posts

APRA releases final version of CPS 234 Information Security

November 21, 2018

1/2
Please reload

Recent Posts
Please reload

Hall Advisory Services Pty. Ltd.

ABN: 63 615 549 909

Hall Advisory Services Pte. Ltd.

UEN: 201830706N

Sydney

A: L36, Gateway, 1 Macquarie Place, Sydney NSW 2000

T: +61 2 8035 4400

Melbourne

A: L27, 101 Collins Street, Melbourne VIC 3000

T: +61 3 9653 6493

Singapore

A: L24, CapitaGreen,  138 Market Street, Singapore 048946

T: +65 6679 6146

Follow Us
  • LinkedIn Social Icon
  • Twitter Basic Square