Faulty 2021-07-16 Executive Spell Postmortem

Start: 2021-07-16 15:13:53 UTC

End: 2021-07-17 19:16 UTC

Authors: Kurt Barry (kmbarry1)

Status: Resolved

Summary: The executive spell posted on July 16th, 2021 was supposed to raise the debt ceiling on the RWA002-A collateral type (a Centrifuge asset) from 5 million to 20 million DAI. For such assets, simply raising the per-collateral and global debt ceilings is insufficient to enable more DAI to be drawn; additionally, the price of the fixed-quantity dummy asset held by the system must also be increased (otherwise, the collateralization requirement enforces an effective maximum debt). The posted spell did not perform this latter step, so had it executed, the counterparty (in this case, New Silver) would not have been able to draw additional DAI as intended.

Impact: Extra work for all teams involved with the governance process, wasted gas fees for those that voted on the original spell, and more cognitive load on MKR governance participants.

Root Causes: The most proximate root cause is that the spell author and reviewers were not sufficiently familiar with the differences between RWA collateral types and normal, crypto-native collateral types. This was also the first time a follow-up debt ceiling raise had been performed on an RWA after its initial addition to the protocol, so there was no prior example to refer to. At a deeper level, there was no reusable utility function for doing such operations correctly, and the standard spell regression test suite had no coverage for this quirk of RWA collateral types.

Trigger: The need to update the debt ceiling of a Centrifuge asset.

Resolution: A corrected spell was written, deployed, tested, and placed into the voting portal since the faulty one had so far received a low amount of MKR votes and the issue was discovered soon after the original spell had been posted.

Detection: A member of the Centrifuge team realized the error that had been made and alerted the MakerDAO core units to the issue.

Action Items:

Action Item Type Owner (GH handle) Ticket
add RWA debt ceiling draw tests to spell test suite prevent kmbarry1 627
add function in DssExecLib for raising RWA debt ceilings prevent brianmcmichael 628
educational session to ensure all PE team members understand how RWAs work prevent godsflaw 629
add spell checklist item to notify external teams to review spells that interact with their systems prevent kmbarry1 630

Lessons Learned

What Went Well:

  • The error was caught quickly, before very many MKR holders had voted, and in plenty of time to create a corrected spell.
  • The worst-case outcome would only have been an ineffective debt ceiling modification and a delay in making it fully effective–spell bugs have the potential to be much worse.
  • Were able to mobilize and coordinate people from several different teams on short notice and on the weekend to get a corrected spell out promptly.

What Went Wrong:

  • A faulty spell made it all the way to the governance voting portal.
  • Review and testing failed to catch the issue.

Timeline

All times UTC.

2021-07-15 3:29: First draft of spell available for review.

2021-07-16 15:13:53: Faulty spell deployed. (incident start)

2021-07-17 9:18: A post is made in the Maker Rocket Chat #governance-and-risk channel by a Centrifuge team member about the issue with the spell.

2021-07-17 around 13:00: Protocol Engineering team members discuss the issue and decide to begin work on a corrected spell.

2021-07-17 13:20: Protocol Engineering team member Brian McMichael posts in #governance-and-risk that a fix is being worked on.

2021-07-17 14:43: Faulty spell removed from the governance portal, announcement made in #governance-and-risk.

2021-07-17 18:39:11: Corrected spell deployed.

2021-07-17 19:12: Sufficient test confirmations of deployed spell obtained to clear it for release to the community.

2021-07-17 19:16: Corrected spell available in the governance portal, announcement made in #governance-and-risk.

Supporting Information

@prose11’s forum post informing the community of what occurred:

14 Likes