UMR backtesting: here to stay
01 February 2022
A Cassini Systems panel discusses all things backtesting as the Regulatory Technical Standards on Standard Initial Margin Model Validation come into effect in September 2022. Carmella Haswell reports
Image: stock.adobe.com/greenbutterfly
The Uncleared Margin Rules (UMR) have acted as a dark cloud across the financial industry for the past several years. Phase 5 and 6 firms have long hoped that the regulation’s backtesting-benchmarking requirements were to be dropped completely, but are now being forced to face the reality that backtesting is here to stay. Since the release of the European Banking Authority’s (EBA’s) Draft Regulatory Technical Standards (RTS) in November 2021, firms are scrambling to catch up before the September 2022 implementation date.
Although backtesting has been prevalent across the finance industry, the topic cropped up less often within client conversations prior to the EBA’s Draft RTS. According to head of product at Cassini Systems, Thomas Griffiths, clients that were previously interested in backtesting were typically larger firms that were accustomed to model governance committees and had large internal approval of models set up, which they required for internal controls.
However, over the past three months since the release of Draft RTS, this has changed massively. “Backtesting is now mentioned in all of our client conversations — from the time the Draft RTS was released onwards, it has become a talking point,” says Griffiths. Some firms are unaware of what backtesting is and why it is needed, having instead whittled it down to yet another thing on the UMR compliance checklist.
Founder of Link Risk, Andy Shaw, explains: “Backtesting is the science of establishing whether or not your risk model is doing what it is supposed to be doing. It is a quantitative process that underpins any particular risk model and demonstrates to the users that it is doing what it should. It is used as a way of measuring Standard Initial Margin Model (SIMMs) validity.”
There are two different types of backtesting, static and dynamic. The dynamic backtest is designed to be the simpler of the two, described as “a routine P&L comparison to the portfolio SIMM level on any particular day”. However, there are limitations in terms of what it can tell firms. Shaw explains that the static backtest is aimed at the bigger dealers, which involves statistical performance measures, where SIMM validity is established via the static backtest. “It is required to answer the question: is my SIMM level enough to cover 99 per cent of the P&L events that my portfolio might experience?” he adds.
The panel explained that the EU rule set on the initial margin (IM) model validation originally existed to ensure firms have a strong and robust SIMM governance framework that sits around their calculations and it is backed up with suitable controls, procedures and oversight. The rules contain both quantitative (static backtesting) and qualitative (model governance) components. The end goal for regulators is to ensure that firms produce accurate and reliable SIMM numbers. Reverting to the benchmark rule, the panel says SIMM is required to cover exposure for at least a 10-day Margin Period of Risk (MPOR) at least 99 per cent of the time.
Co-founder and director at Margin Tonic, Chris Watts, says the original EU rules which apply to all firms equally are a vast difference to the US rules which apply directly and only to the swap dealers and major swap participants (MSPs). “This has effectively meant that until now, all firms covered by the EU rules have had to meet the same heavy set of model governance and backtesting rules, regardless of whether they are a huge dealer or a smaller buy-side player. Most buy-side firms do not have the sophisticated risk infrastructure seen at the dealers, making these rules difficult to comply with for the Phase 5 and 6 firms in particular.
“On top of that, the International Swaps and Derivatives Association (ISDA) have previously stressed that the SIMM calculation is an approved industry model already. Therefore, these strict requirements are overly punitive for the buy-side, especially when SIMM is already being regularly backtested by the dealers.”
The EBA Draft RTS has introduced two sets of model validation rules to add some relief to smaller firms. There is the standard model — which offers a full set of rules for the dealer banks and larger firms (€750 billion) — and a simplified model — which presents a reduced set of rules for smaller firms (€8 billion to €750 billion). The phased transition of the model validation regulator submissions, which require an upfront submission and then at least an annual ongoing submission, will allow firms one to three years for the upfront initial submission, once rules are live.
The submission dates are dictated by a firm's Average Aggregate Notional Amount (AANA) threshold, meaning Phase 6 firms will be given three years from the live date, while dealers have one year to turn around their submissions. It is strongly advised for firms not to wait three years to address these submissions. Additionally, the regulators will have two years to come back on any submissions.
Watts adds: “ISDA has been pushing the EU to strip back these requirements and lobbied for the same model that we are seeing in the US, where model governance and backtesting is taken away for the buy-side. The buy-side has not got the same risk infrastructure as the larger firms and, at worst, firms could default to grid calculations if they see the SIMM implementation effort as being too heavy. According to ISDA, grid calculations could be two to three times more expensive for Phase 5 firms than SIMM.”
Attempting to pave a new and improved way for firms, the EBA released its Draft RTS three months ago. The Initial Margin Model Validation (IMMV) continues with both backtesting and governance components. Although it is more prescriptive than the original RTS, there are some areas open for interpretation, according to Margin Tonic’s Watts. Reduced requirements under the ‘Simplified’ model include the need to perform dynamic backtesting only (rather than the original, heavier static backtest requirement), as well as a lesser set of model governance documentation.
However, a change for all in-scope firms is that explicit model validation approval will be required upfront, in line with the European Market Infrastructure Regulation (EMIR) Refit, which follows best market practice for Market Risk Model Validation. Previously, EU firms were only required to self-notify the regulator of their model governance and backtesting compliance, rather than securing explicit approval. There is also an annual requirement for ongoing regulator submissions.
Watts adds: “The reality is, this relief looks little like what ISDA were lobbying for, with backtesting and model governance still in play. There is an ongoing burden on all in-scope firms going forward including the ‘smaller guys’. Backtesting looks different with its two levels, a static side which is aimed at the bigger dealers, and the dynamic side is a day-to-day SIMM versus P&L comparison. It is certainly now lighter touch for the buy-side, which is the good news.
“The model governance documentation is stripped back for those simplified firms as well," notes Watts. "The compliance burden clearly rests with the end client, so vendors can certainly help here, but regulators have stated that clients need to understand their SIMM governance framework, rather than outsourcing it entirely.”
The implementation process
The process of implementing backtesting will not come easily for clients with a lack of resources, expertise or interest in keeping it in-house. However, there are a number of firms on hand to aid clients through this period. Managing director of funding and collateral transformation at State Street, Ingvar Sigurjonsson, says State Street can do the backtesting for clients by providing them with reports and documentation — all of which has gone through the firm’s internal model validation audit.
Cassini Systems’ Griffiths has also been heavily involved in backtesting, helping clients from Phase 4 through to Phase 6 onboard for full regulatory UMR compliance. Cassini Systems offers a backtesting product, rather than a service, which differs from a full compliance solution. Its role is to provide clients with the numbers and a large amount of documentation about why the numbers are what they are, the assumptions that go into the numbers, the models that are employed, and why they comply with regulatory requirements.
Griffiths explains: “For us, building a backtesting solution came quite naturally. When we inputted our SIMM calculations we produced large amounts of model documentation about how we had implemented them and the details of the models. This leads nicely onto model risk documentation. It fitted into a suite of documentation we already had, so then producing the backtesting calculations themselves was a straightforward development build for us. The fact that we had something pre-packaged for SIMM which we could build off helped us a lot.”
Margin Tonic has been involved in every phase of UMR, working through the full set of end-to-end changes over recent years and believes it is in a good position to aid other firms with their UMR challenges going forwards. Margin Tonic’s Watts advises firms to engage with vendors to receive help with the process of implementing backtesting. He predicts the vast majority of firms will outsource their SIMM calculations to vendors of service providers, because SIMM is a ‘heavy lift’, with input data requirements in particular being prescriptive. This will otherwise prove challenging for firms without a mature risk infrastructure.
Watts adds: “If you think about implementation and the assumption that the majority of firms will use vendors for the SIMM calculations itself, the natural extension for that service is for firms to be asking their vendors whether they can also help them with any ancillary SIMM requirements — that includes backtesting and model governance.” Some vendors are more experienced through the challenges of UMR. Having ‘burnt the scars and battle wounds’, they are now in a position to share lessons learnt with clients to reduce the compliance burden.
End clients are reminded that there is still an internal requirement on firms within the Draft RTS — for example with senior management governance, where firms cannot outsource SIMM entirely.
In addressing the question of who is responsible for SIMM and the relevant model governance, Watts indicates that we need to differentiate between the larger firms and smaller Phase 5 and 6 firms. “There is no one-size-fits-all for the SIMM owner within the smaller buy-side firms. At the dealers, the market risk teams will almost all own SIMM, based on their risk and quantitative expertise. However, on the buy-side that team does not always exist. For those firms who do not have a market risk team already, SIMM ownership will be up-for-grabs and dependent on the set-up at each firm. The reality is that there normally are not teams putting their hands up to be the SIMM owner,” explains Watts.
A time constraint
With the implementation of Draft RTS looming, firms are advised to hit the ground running. The process of choosing a SIMM vendor and setting up the relevant model governance and backtesting framework takes time and Watts suggests that firms should think about vendor onboarding. “To share all of your data and scope and set-up with that vendor, in order to get the best tailored service, could take up to 12 months to go live on any particular solution,” he says.
In terms of data, clients will not need to bring anything extra to the table when implementing this process. However, vendors will need to provide additional data requirements. Cassini Systems’ Griffiths comments: “One good piece of news about backtesting is that there are no additional trade data requirements for clients. If you have a vendor that is producing your SIMM calculations, they should also be able to produce your backtesting calculations with the same trade data that you provided to them for SIMM — assuming they have a backtesting service.
“As a vendor, there is quite a large amount of additional market data requirements. We need to source all of our historic market data and we use industry standard market data vendors to source three years of backtesting data plus the stress years of 2008 and 2009.”
Inevitably, regulators play an important part in these discussions, so what do they expect from clients? Link Risk’s Shaw explains that the key purpose of the European regulation is for firms to monitor their margin models to evaluate whether or not they’re meeting the requirements to cover 99 per cent of losses. As the governance framework to do this is built, it will be down to companies to look at their P&L or their backtest to see if it passes the threshold — regulators could ask to see the retained records as proof of this. If there appears to be a breach, firms will need to follow their governance framework to put the margin back in the green.
Shaw continues: “The Draft RTS may change, it is more of a question of following the right procedures and regulators may well subsequently come in and audit you on it, and then there are different thresholds for when you might have to tell them about the breach. So if things get really bad at that point, you do have to tell the regulator and seek re-validation of the model.”
Shaw adds that if firms decided to continue to ignore a model that is flashing red (failing), they would be in breach of the regulation as well as their own governance framework. The impact of this may vary. Firms could face fines for not correcting their margin — although Shaw believes regulators would give firms a chance to rectify the breach first. For firms with no governance framework, more punitive measures may be applied.
In closing, participants held a vote on whether there was a possibility that backtesting could disappear from the Draft RTS. It seems a unanimous decision was reached, with Margin Tonic’s Watts stating: “There’s more chance of Tottenham winning the Premier League than backtesting being removed.”
Although backtesting has been prevalent across the finance industry, the topic cropped up less often within client conversations prior to the EBA’s Draft RTS. According to head of product at Cassini Systems, Thomas Griffiths, clients that were previously interested in backtesting were typically larger firms that were accustomed to model governance committees and had large internal approval of models set up, which they required for internal controls.
However, over the past three months since the release of Draft RTS, this has changed massively. “Backtesting is now mentioned in all of our client conversations — from the time the Draft RTS was released onwards, it has become a talking point,” says Griffiths. Some firms are unaware of what backtesting is and why it is needed, having instead whittled it down to yet another thing on the UMR compliance checklist.
Founder of Link Risk, Andy Shaw, explains: “Backtesting is the science of establishing whether or not your risk model is doing what it is supposed to be doing. It is a quantitative process that underpins any particular risk model and demonstrates to the users that it is doing what it should. It is used as a way of measuring Standard Initial Margin Model (SIMMs) validity.”
There are two different types of backtesting, static and dynamic. The dynamic backtest is designed to be the simpler of the two, described as “a routine P&L comparison to the portfolio SIMM level on any particular day”. However, there are limitations in terms of what it can tell firms. Shaw explains that the static backtest is aimed at the bigger dealers, which involves statistical performance measures, where SIMM validity is established via the static backtest. “It is required to answer the question: is my SIMM level enough to cover 99 per cent of the P&L events that my portfolio might experience?” he adds.
The panel explained that the EU rule set on the initial margin (IM) model validation originally existed to ensure firms have a strong and robust SIMM governance framework that sits around their calculations and it is backed up with suitable controls, procedures and oversight. The rules contain both quantitative (static backtesting) and qualitative (model governance) components. The end goal for regulators is to ensure that firms produce accurate and reliable SIMM numbers. Reverting to the benchmark rule, the panel says SIMM is required to cover exposure for at least a 10-day Margin Period of Risk (MPOR) at least 99 per cent of the time.
Co-founder and director at Margin Tonic, Chris Watts, says the original EU rules which apply to all firms equally are a vast difference to the US rules which apply directly and only to the swap dealers and major swap participants (MSPs). “This has effectively meant that until now, all firms covered by the EU rules have had to meet the same heavy set of model governance and backtesting rules, regardless of whether they are a huge dealer or a smaller buy-side player. Most buy-side firms do not have the sophisticated risk infrastructure seen at the dealers, making these rules difficult to comply with for the Phase 5 and 6 firms in particular.
“On top of that, the International Swaps and Derivatives Association (ISDA) have previously stressed that the SIMM calculation is an approved industry model already. Therefore, these strict requirements are overly punitive for the buy-side, especially when SIMM is already being regularly backtested by the dealers.”
The EBA Draft RTS has introduced two sets of model validation rules to add some relief to smaller firms. There is the standard model — which offers a full set of rules for the dealer banks and larger firms (€750 billion) — and a simplified model — which presents a reduced set of rules for smaller firms (€8 billion to €750 billion). The phased transition of the model validation regulator submissions, which require an upfront submission and then at least an annual ongoing submission, will allow firms one to three years for the upfront initial submission, once rules are live.
The submission dates are dictated by a firm's Average Aggregate Notional Amount (AANA) threshold, meaning Phase 6 firms will be given three years from the live date, while dealers have one year to turn around their submissions. It is strongly advised for firms not to wait three years to address these submissions. Additionally, the regulators will have two years to come back on any submissions.
Watts adds: “ISDA has been pushing the EU to strip back these requirements and lobbied for the same model that we are seeing in the US, where model governance and backtesting is taken away for the buy-side. The buy-side has not got the same risk infrastructure as the larger firms and, at worst, firms could default to grid calculations if they see the SIMM implementation effort as being too heavy. According to ISDA, grid calculations could be two to three times more expensive for Phase 5 firms than SIMM.”
Attempting to pave a new and improved way for firms, the EBA released its Draft RTS three months ago. The Initial Margin Model Validation (IMMV) continues with both backtesting and governance components. Although it is more prescriptive than the original RTS, there are some areas open for interpretation, according to Margin Tonic’s Watts. Reduced requirements under the ‘Simplified’ model include the need to perform dynamic backtesting only (rather than the original, heavier static backtest requirement), as well as a lesser set of model governance documentation.
However, a change for all in-scope firms is that explicit model validation approval will be required upfront, in line with the European Market Infrastructure Regulation (EMIR) Refit, which follows best market practice for Market Risk Model Validation. Previously, EU firms were only required to self-notify the regulator of their model governance and backtesting compliance, rather than securing explicit approval. There is also an annual requirement for ongoing regulator submissions.
Watts adds: “The reality is, this relief looks little like what ISDA were lobbying for, with backtesting and model governance still in play. There is an ongoing burden on all in-scope firms going forward including the ‘smaller guys’. Backtesting looks different with its two levels, a static side which is aimed at the bigger dealers, and the dynamic side is a day-to-day SIMM versus P&L comparison. It is certainly now lighter touch for the buy-side, which is the good news.
“The model governance documentation is stripped back for those simplified firms as well," notes Watts. "The compliance burden clearly rests with the end client, so vendors can certainly help here, but regulators have stated that clients need to understand their SIMM governance framework, rather than outsourcing it entirely.”
The implementation process
The process of implementing backtesting will not come easily for clients with a lack of resources, expertise or interest in keeping it in-house. However, there are a number of firms on hand to aid clients through this period. Managing director of funding and collateral transformation at State Street, Ingvar Sigurjonsson, says State Street can do the backtesting for clients by providing them with reports and documentation — all of which has gone through the firm’s internal model validation audit.
Cassini Systems’ Griffiths has also been heavily involved in backtesting, helping clients from Phase 4 through to Phase 6 onboard for full regulatory UMR compliance. Cassini Systems offers a backtesting product, rather than a service, which differs from a full compliance solution. Its role is to provide clients with the numbers and a large amount of documentation about why the numbers are what they are, the assumptions that go into the numbers, the models that are employed, and why they comply with regulatory requirements.
Griffiths explains: “For us, building a backtesting solution came quite naturally. When we inputted our SIMM calculations we produced large amounts of model documentation about how we had implemented them and the details of the models. This leads nicely onto model risk documentation. It fitted into a suite of documentation we already had, so then producing the backtesting calculations themselves was a straightforward development build for us. The fact that we had something pre-packaged for SIMM which we could build off helped us a lot.”
Margin Tonic has been involved in every phase of UMR, working through the full set of end-to-end changes over recent years and believes it is in a good position to aid other firms with their UMR challenges going forwards. Margin Tonic’s Watts advises firms to engage with vendors to receive help with the process of implementing backtesting. He predicts the vast majority of firms will outsource their SIMM calculations to vendors of service providers, because SIMM is a ‘heavy lift’, with input data requirements in particular being prescriptive. This will otherwise prove challenging for firms without a mature risk infrastructure.
Watts adds: “If you think about implementation and the assumption that the majority of firms will use vendors for the SIMM calculations itself, the natural extension for that service is for firms to be asking their vendors whether they can also help them with any ancillary SIMM requirements — that includes backtesting and model governance.” Some vendors are more experienced through the challenges of UMR. Having ‘burnt the scars and battle wounds’, they are now in a position to share lessons learnt with clients to reduce the compliance burden.
End clients are reminded that there is still an internal requirement on firms within the Draft RTS — for example with senior management governance, where firms cannot outsource SIMM entirely.
In addressing the question of who is responsible for SIMM and the relevant model governance, Watts indicates that we need to differentiate between the larger firms and smaller Phase 5 and 6 firms. “There is no one-size-fits-all for the SIMM owner within the smaller buy-side firms. At the dealers, the market risk teams will almost all own SIMM, based on their risk and quantitative expertise. However, on the buy-side that team does not always exist. For those firms who do not have a market risk team already, SIMM ownership will be up-for-grabs and dependent on the set-up at each firm. The reality is that there normally are not teams putting their hands up to be the SIMM owner,” explains Watts.
A time constraint
With the implementation of Draft RTS looming, firms are advised to hit the ground running. The process of choosing a SIMM vendor and setting up the relevant model governance and backtesting framework takes time and Watts suggests that firms should think about vendor onboarding. “To share all of your data and scope and set-up with that vendor, in order to get the best tailored service, could take up to 12 months to go live on any particular solution,” he says.
In terms of data, clients will not need to bring anything extra to the table when implementing this process. However, vendors will need to provide additional data requirements. Cassini Systems’ Griffiths comments: “One good piece of news about backtesting is that there are no additional trade data requirements for clients. If you have a vendor that is producing your SIMM calculations, they should also be able to produce your backtesting calculations with the same trade data that you provided to them for SIMM — assuming they have a backtesting service.
“As a vendor, there is quite a large amount of additional market data requirements. We need to source all of our historic market data and we use industry standard market data vendors to source three years of backtesting data plus the stress years of 2008 and 2009.”
Inevitably, regulators play an important part in these discussions, so what do they expect from clients? Link Risk’s Shaw explains that the key purpose of the European regulation is for firms to monitor their margin models to evaluate whether or not they’re meeting the requirements to cover 99 per cent of losses. As the governance framework to do this is built, it will be down to companies to look at their P&L or their backtest to see if it passes the threshold — regulators could ask to see the retained records as proof of this. If there appears to be a breach, firms will need to follow their governance framework to put the margin back in the green.
Shaw continues: “The Draft RTS may change, it is more of a question of following the right procedures and regulators may well subsequently come in and audit you on it, and then there are different thresholds for when you might have to tell them about the breach. So if things get really bad at that point, you do have to tell the regulator and seek re-validation of the model.”
Shaw adds that if firms decided to continue to ignore a model that is flashing red (failing), they would be in breach of the regulation as well as their own governance framework. The impact of this may vary. Firms could face fines for not correcting their margin — although Shaw believes regulators would give firms a chance to rectify the breach first. For firms with no governance framework, more punitive measures may be applied.
In closing, participants held a vote on whether there was a possibility that backtesting could disappear from the Draft RTS. It seems a unanimous decision was reached, with Margin Tonic’s Watts stating: “There’s more chance of Tottenham winning the Premier League than backtesting being removed.”
NO FEE, NO RISK
100% ON RETURNS If you invest in only one securities finance news source this year, make sure it is your free subscription to Securities Finance Times
100% ON RETURNS If you invest in only one securities finance news source this year, make sure it is your free subscription to Securities Finance Times
