Start: 2021-07-16 15:13:53 UTC
End: 2021-07-17 19:16 UTC
Authors: Kurt Barry (kmbarry1)
Summary: The executive spell posted on July 16th, 2021 was supposed to raise the debt ceiling on the
RWA002-A collateral type (a Centrifuge asset) from 5 million to 20 million DAI. For such assets, simply raising the per-collateral and global debt ceilings is insufficient to enable more DAI to be drawn; additionally, the price of the fixed-quantity dummy asset held by the system must also be increased (otherwise, the collateralization requirement enforces an effective maximum debt). The posted spell did not perform this latter step, so had it executed, the counterparty (in this case, New Silver) would not have been able to draw additional DAI as intended.
Impact: Extra work for all teams involved with the governance process, wasted gas fees for those that voted on the original spell, and more cognitive load on MKR governance participants.
Root Causes: The most proximate root cause is that the spell author and reviewers were not sufficiently familiar with the differences between RWA collateral types and normal, crypto-native collateral types. This was also the first time a follow-up debt ceiling raise had been performed on an RWA after its initial addition to the protocol, so there was no prior example to refer to. At a deeper level, there was no reusable utility function for doing such operations correctly, and the standard spell regression test suite had no coverage for this quirk of RWA collateral types.
Trigger: The need to update the debt ceiling of a Centrifuge asset.
Resolution: A corrected spell was written, deployed, tested, and placed into the voting portal since the faulty one had so far received a low amount of MKR votes and the issue was discovered soon after the original spell had been posted.
Detection: A member of the Centrifuge team realized the error that had been made and alerted the MakerDAO core units to the issue.
|Action Item||Type||Owner (GH handle)||Ticket|
|add RWA debt ceiling draw tests to spell test suite||prevent||kmbarry1||627|
|add function in DssExecLib for raising RWA debt ceilings||prevent||brianmcmichael||628|
|educational session to ensure all PE team members understand how RWAs work||prevent||godsflaw||629|
|add spell checklist item to notify external teams to review spells that interact with their systems||prevent||kmbarry1||630|
What Went Well:
- The error was caught quickly, before very many MKR holders had voted, and in plenty of time to create a corrected spell.
- The worst-case outcome would only have been an ineffective debt ceiling modification and a delay in making it fully effective–spell bugs have the potential to be much worse.
- Were able to mobilize and coordinate people from several different teams on short notice and on the weekend to get a corrected spell out promptly.
What Went Wrong:
- A faulty spell made it all the way to the governance voting portal.
- Review and testing failed to catch the issue.
All times UTC.
2021-07-15 3:29: First draft of spell available for review.
2021-07-16 15:13:53: Faulty spell deployed. (incident start)
2021-07-17 9:18: A post is made in the Maker Rocket Chat #governance-and-risk channel by a Centrifuge team member about the issue with the spell.
2021-07-17 around 13:00: Protocol Engineering team members discuss the issue and decide to begin work on a corrected spell.
2021-07-17 13:20: Protocol Engineering team member Brian McMichael posts in #governance-and-risk that a fix is being worked on.
2021-07-17 14:43: Faulty spell removed from the governance portal, announcement made in #governance-and-risk.
2021-07-17 18:39:11: Corrected spell deployed.
2021-07-17 19:12: Sufficient test confirmations of deployed spell obtained to clear it for release to the community.
2021-07-17 19:16: Corrected spell available in the governance portal, announcement made in #governance-and-risk.
@prose11’s forum post informing the community of what occurred: