Some learnings from Wild Wednesday

The dust seems to have now settled on yesterday’s events and it seems like we have some peace and quiet again in the crypto world. I think what is the most important thing to say is congratulations to the developers for making Maker a far more robust protocol than it was this time a year ago. This was a much bigger stress test than Black Thursday and this time, the protocol performed amazingly!!

That said, I’m gonna add a short list of things that didn’t go quite perfectly last night. Hopefully, these are things we can discuss and improve before the next seismic crash - whenever that may be. We talked about some of this on the chat but I think it would be nice to put it one place for different teams to look through.

  1. The OSM feed did not update at 13:00 UTC on 19/05 as it usually does due to insufficient gas. Gas prices were insane at this point, around 2500 gwei. As a result, the liquidation prices remained several hundred dollars above the true price, even though the crash persisted for well over an hour. I believe we had ETH at $2440 when it was actually under $2000 for an extended period of time. We were lucky that ETH bounced back from there but had it crashed further, this price feed delay might have caused bad debt issues.

  2. Two vaults were liquidated and we could not recover their debt fully. This was simply liquidations being stretched to the limit presumably. Several vaults were liquidated and no collateral was returned - again a sign that this stress test pushed our systems pretty hard. Would we have survived had ETH crashed down to, say, $1400? This may also be related to the oracle issues in 1 - perhaps we were just too slow to update the prices and hence fell behind on the liquidation process.

  3. ZRX had really low liquidity which meant that the auctions took a long time to clear. Moreover, some vaults which were underwater could not be liquidated because there simply wasn’t enough ZRX liquidity anywhere. I’m not sure if the same situation affected any other tokens but it was an interesting situation that played out.

  4. The PSM generally helped keep DAI pegged. However, I’m not sure if USDC itself was on peg - sources like Coingecko indicate that it was not. USDT certainly did not stay pegged. Perhaps it’s worth thinking about how the PSM should function when our oracles suggest that USDC is off peg?

These were my four observations from yesterday. And again, congrats to all, especially the Liq 2.0 guys for massively improving the protocol and making it ready for Wild Wednesday.

12 Likes

Let me also take a moment to point out that the liquidations in ZRX (and it looks like that big vault is eligible to be kicked again even now) are what has made that vault class profitable. I estimate it has only produced ~30k DAI in fees over the 11 months this class has been active. My understanding is that this does not begin to cover the oracle costs for this token.

Given that most of the debt has been cleared out of this vault class, this is a good opportunity to retire the ZRX vault class. However we choose to encourage vault closure in this vault class can also serve as a template for offloading other vault classes that lose money by not even covering oracle costs.

7 Likes

Emotional attachments to Matcha or 2017 ICO ZRX bags aside, I agree with PaperImperium’s initiative. The DAO should not take on assets that fail to produce revenue above and beyond the cost of making them available as collateral. At the same time, this is a good exercise for governance, removing non or underperforming assets (all eyes on you, KNC). Tldr, agree with @PaperImperium 100%.

2 Likes

Are there any other assets that are uneconomical? This is possibly a good time to reflect on the assets that we are using - maybe just because we can onboard a collateral to MCD doesn’t mean we should.

On the other hand, is there a situation where a collateral doesn’t make us money but has a role in stabilising the peg, which is an obvious benefit to the protocol that may outweigh the cost. The clear consideration here being the USDC-A vault which currently has no stability fee but is our 4th biggest vault type. I’d also be interested in the costs the PSM incurs. We have allowed nearly a billion DAI to be generated and recouped less than a million in fees - are we subsiding stability here?

Yes, I can put together a list, though I think @NikKunkel mentioned most of them on the governance call today. It’s mainly ones that doesn’t even carry the Oracle costs, much less compensate Maker for the risk of bad debt.

Note that all the eligible ZRX debt has not been liquidated, so it’s possible this vault class will still swing to a total loss, even after making profit from the liquidation penalties yesterday.

2 Likes

The biggest risk of Liquidations 2.0 has been the timeline. To work on something for more than a year and finish 2 weeks before a likely catastrophe would have unfolded, that ought to make us pause.

We were balancing on the edge and we just got incredibly lucky because obviously the timing of the crash is something that was entirely out of our control.

The few imperfections in its performance do not even begin to compare to the risk of having been exposed for too long imo.

This is a time to be relieved and happy and proud of what we’ve built. But also to realize that we need to find ways to be more proactive about risk like this because next time, we probably won’t get this lucky.

23 Likes

I think this statement can be generalized about more than liquidations. Maker has the wind at its back, but there’s a lot of work to be done, and it’s not clear who will do it or how it will be done.

4 Likes

I want to echo this.

While some of the delay was dealing with auditor schedules post DeFi summer, much of this delay was related to how we set priorities. We were working on collateral onboarding, to keep up with DeFi summer DAI demand, PSMs, RWAs, flash loan fixes for governance, staffing problems, etc. Some of this was just us responding to the market conditions or fire fighting, but everything else should not have been as high a priority. Even the weekly spell cadence is aggressive and impacts our schedules.

The fix is about communication of risks, rewards, and timelines and then a decisive follow-up on priorities. We got very lucky. A slip of 2 weeks more on the schedule would have been disastrous for the protocol this past week.

This is my takeaway. It was clear internally where the risk was, but we could have done better about pushing back on collateral onboarding to make sure the risk was better mitigated. 20/20 hindsight would have indicated we should have done the PSM immediately. This would have taken the dire need for DAI supply off the table, and allowed us to focus earlier on completion of LIQ-2.0. After that, like now, we can go back to being more focused on MIPs and collateral onboarding. Risks and firefighting are always the high priority interrupt. Maker should better have classified LIQ-2.0 as a high priority emergency.

This should be a blameless exercise, but I think it’s a good idea for each domain to ask how they could have enabled LIQ-2.0 better. It may be that it was not communicated clearly enough how important this upgrade was for teams to make that choice, but look beyond that at what you can control. What changes would you make today?

In the end, we did know this was a priority, and we push aggressively in a few areas to ensure the schedule didn’t slip more than it needed to. That is a success, and teams should take a well deserved victory lap.

18 Likes

I want to bang this drum again that we may want to consider reducing weekly executive cadence to bi-weekly except for emergencies so that voters and developers can be less fatigued.

2 Likes

I want to push back against some of the luck talk here. The Liquidation 2.0 timeline has definitely been long as hell, and more than anything that has appeared to be due to the long wait for audits. That said, both @Risk-Core-Unit and @Protocol-Engineering have been pushing really hard since then to get it out.

They’ve been pushing so hard because they knew that we needed to beat the next major crash. Everyone knew it was coming sooner or later. Everyone involved was aware of the urgency as the market continued to rise. Everyone pushed themselves hard to make sure it didn’t slip more than absolutely necessary. We made our own luck here.

And the outcome was that we did beat the crash on the major vault types. Not only that, but we also managed to onboard the PSM to prevent the peg from deviating significantly during the supply-demand imbalance. We also managed to onboard the first RWA’s and AMM LPTs, both of which should help to provide DAI supply when the bear-market hits in earnest.

Looking at what we needed to do, and what we’ve done, I think everyone involved has done a damn good job at prioritizing. We’ve staved off multiple existential threats to the protocol, and have probably laid the groundwork in time for us to stave off the next one (prolonged bear market.) We also (imo correctly) deprioritized several technical improvements that do not represent existential threats.

There are always ways in which we can improve, and I think everyone will be more comfortable if we have more breathing room in the future, but we should not dismiss that which we did well as luck. Nor attribute that which went badly only to failures of process or prioritization.

19 Likes

Is it perhaps worth starting a signal request for this item because otherwise, it will be forgotten with this thread?

Since you and @cmooney both have made this point, maybe you could point out the pros and cons?

I believe you meant 13:00 UTC, as prices at 01:00 were very different. Let’s look at what happened around 13:00 UTC

At 12:13 UTC, the current OSM became $2829 and the next OSM $2705. Both of these prices are representive of market prices at the time. This OSM action was delayed by about 8 minutes, as it usually happens around the fifth minute of the hour.

At 13:05 UTC, current OSM became $2705 (expected) and the next OSM was queued up at $2627. Between 12:13 and 13:05, ETH price dropped from $2723 to a low of $2003 before recovering to $2257 at the top of the hour. Due to network congestion and high gas cost during that period, the medianizer price couldn’t be updated in time for the OSM update of 13:05, so the price that was queued up was the current price of approximately 12:30 instead of 13:05.

5 Likes

Yes, I meant to say 13:00, not 01:00 - corrected now, thank you!