Maker Vault Liquidation Protection Analysis

Introduction

MakerDAO’s understanding of vault liquidation behavior is crucial to long-term economic resilience of its system of smart contracts. While a tail event, there is significant solvency risk involved in case of too much bad debt being liquidated at the same time during large price drops of ilk’s underlying collateral assets. While a lot of effort has been already invested across different teams to maximize robustness of the liquidation system, there is an external element of vault activity which can also aid in preventing insolvency from ever happening.
@Risk-Core-Unit team decided to look into this dynamic to derive useful insights as a part of a series of analyses on vault behavior.

We begin by exploring the high-level vault behavior, both looking at how many vaults are being protected over time and also how vault owners protect their vaults.
As expected, vaults tend to have the most liquidation protection activity during large drops in collateral asset prices. Besides that, we also look at how successful vaults are at protecting their borrowed funds, split over various dimensions.

We extend the research by comparing vault liquidation activity during two days of major drops in collateral asset prices: March 12th 2020 (Black Thursday) and May 19th 2021. Both days include the most data points to analyze liquidation activity in isolation and also in comparison.

Methodology

Key component behind the analysis is how we identify liquidation protection activity. As we can see in the below taxonomy of liquidation protection action patterns, a liquidation protection action can be performed in either a single operation within a single transaction, multiple operations within a single transaction and multiple operations within multiple transactions. For this reason we decided not to describe activity in terms of transactions or operations and rather defined all three liquidation protection patterns as a single liquidation protection action.
The analysis includes all vault types except those collateralized by stablecoins.

The analysis looks at three main ways to increase collateralization ratio via a liquidation protection action type:

  • deposit collateral (DEPOSIT)

  • payback DAI (PAYBACK)

  • withdraw collateral, swap for DAI (possibly at a DEX, outside of Maker) followed by paying back DAI (WITHDRAW|PAYBACK)

Only the third action type (WITHDRAW|PAYBACK) can include two separate operations which can be executed in either 1 or 2 separate transactions/blocks. To interact with the vault, the vault owner could execute the action via their own bot, use a Maker vault management UI or through one of the specialized liquidation protection services (DeFi Saver, InstaDapp, etc.)

For actions performed in two separate blocks, we filter out only those that were executed during the same OSM price to minimize identified false positives.

Another requirement for a liquidation protection action is that it was performed on an unsafe vault position. Unsafe position is defined when the collateralization ratio is less than or equal to 20% above the liquidation ratio in relative terms (CR <= LR *1.2). In the example of ETH-A it would mean actions executed when the collateralization ratio was 180% or below. We are looking at the collateralization ratio before the liquidation protection action was executed.

Important to mention is that our detection methodology is not a ground truth for every liquidation protection action. It’s a heuristic to classify patterns that we’re interested in while trying to minimize both false positives and false negatives. Setting the 1.2 multiplier is a typical precision/recall trade-off problem where we can’t increase one without decreasing the other.

When looking at the liquidation protection success rate, we define an action as successful if the vault wasn’t liquidated within 24 hours after action execution.

High-level Metric Overview

Overall, we identified around 38000 liquidation protection actions performed by around 5000 unique vaults.

Chart 1: Protected Vaults vs. Liquidated Vaults

In the chart below we can see that the most monthly protected vaults align with the months with the two major price drops (March 2020 and May 2021). Meanwhile, it’s clear that the number of liquidations relative to the number of protected vaults is much lower in May 2021 compared to March 2020.
This reflects well on the responsiveness and success rate of protection activity of the vaults affected by the collateral price drops. Other than that, the usual number of monthly protected vaults range from 200 to 1200.

Chart 2: Protected Debt Size vs. Liquidated Debt Size

We can also aggregate monthly overview on debt size instead of the number of vaults. As expected there is a strong relationship between protected and liquidated amounts. There is a stronger influence of DAI supply growth which causes the recent months to be larger both in protected and liquidated amounts.
A possible extended metric is to look at the ratio between liquidated debt size and protected debt size. As it can be derived from the chart, this has been relatively constant since a major increase in the month of Black Thursday.

Chart 3: Liquidation Protection Actions per Month by Action Type

Distribution of liquidation protection action types over time shows that DEPOSIT, PAYBACK and WITHDRAW|PAYBACK_ST (ST = single transaction) make most of the total actions. Each of them currently contributes to a bit less than ⅓ of total actions while the WITHDRAW|PAYBACK_MT (MT = multiple transactions) action type contributes to less than 5%.

Meanwhile, DEPOSIT has lost some of its share since MCD’s launch to WITHDRAW|PAYBACK_ST. Similar applies to WITHDRAW|PAYBACK_MT.

Chart 4: Average Liquidation Protection Debt Size by Action Type

When looking at protected debt size across different action types, we can see that PAYBACK tends to protect the highest average debt size. Meanwhile, the other three action types have a similar average value. Considering the impact of outliers, this indicates that larger vaults (whales) tend to use the PAYBACK action type more when entering the specified danger zone. That’s likely because it’s common for whale activity to include farming with DAI on other venues, which means that they have enough DAI at hand for repayment in case of an unsafe drop in vault collateralization ratio.

Chart 5: Liquidation Protection Debt Size over Time by Action Type

Next, we compare the contribution of protected debt size across different action types. It’s immediately clear that relative shares are less consistent over time than in Chart 3. Also, WITHDRAW|PAYBACK_ST action type loses in relative contribution to total protected debt size which is partly caused by a lower average protected debt size seen in Chart 4.

Chart 6: Most Active Liquidation Protection Days

The most active liquidation protection days tend to occur during large market price drops which doesn’t come as a surprise. For this reason March 2020 and May 2021 again pop up as days with a high liquidation protection activity.

Chart 7: Largest Liquidation Protection Debt Size Days

When we look at days with the most protected debt size we can see that all days happened in the last few months. This makes sense considering a rapidly growing DAI supply where, at a given amount of liquidation protection activity, later dates are more likely to protect larger debt sizes.

Chart 8: Liquidation Protection Actions per Vault Type

Liquidation protection actions distributed across vault types show that as debt exposure became distributed across different collateral assets, so did vault protection activity.

While ETH-A contributed to more than 90% of total action in the first months after MCD launch, the number currently stands at around 50% which mirrors its current DAI collateralization share.

Chart 9: Liquidation Protection Success Rate by Vault Type

Overall success rate of all liquidation protection actions is 95%. That means that 1 in 20 actions doesn’t save the vault from being liquidated within 24 hours after action execution.
There is no difference when looking at success rate across different action types so we decided not to include the chart in the analysis. Meanwhile, when looking at distribution by vault type, we can see that while there is a minor difference in their success rate, actions on Uniswap LP vault types tend to be the most successful (99%+) and actions on ETH-A tend to be the least successful (90%). Both make sense, the former because of lower asset volatility and the latter because only ETH-A and BAT-A vault types were available during Black Thursday market crash which had the highest number of liquidations (unsuccessful protection actions).

Tail Market Risk Behavior Analysis: Black Thursday vs. May 19th

Table 1: BT vs. M19 Comparison: Liquidation Protection Metrics

In the comparison of two downfall events, Black Thursday to May 19th, we can see that the latter had about twice as many active vaults at the time. An active vault is defined as a vault that has had a vault operation within 3 months before the chosen date and more than 100 DAI being borrowed at the time of the executed action.

Even with that, there were both more protected vaults on Black Thursday and also the vaults were being more actively protected (average protection actions per protected vault).

That also impacted a lower percentage of daily active protecting vaults during the May 19th event. On the other hand, the liquidation protection success rate was considerably higher in the price crash this year which sheds a positive light on the improved robustness of the system since Black Thursday.

One important factor to consider is that during Black Thursday, there was no flash loan feature available yet which increased the number of actions.

Chart 10: BT vs. M19 Comparison: Actions by Action Type

Splitting number of liquidation protection actions by action type shows us that while there was an overall higher vault protection activity during Black Thursday, there is the largest difference in the amount of executed WITHDRAW|PAYBACK_ST and WITHDRAW|PAYBACK_MT actions (3.5 BT actions for every M19 action).
Meanwhile, the difference is lower for DEPOSIT and PAYBACK action types (1.7 and 1.15 actions during BT for every 1 action during M19, respectively).

Chart 11: BT to M19 comparison: CR Increase Distribution

The last chart shows distribution of relative collateralization increase before and after the liquidation protection actions (e.g. 5% increase means increasing collateralization ratio from 160% to 168%).

Most increases were smaller, with 60% of them being lower than 10%. For the sake of clear visualization, we filtered out all increases that more than doubled their collateralization ratio (increase > 100% which constitute about 2% of total actions).

We can see that during Black Thursday, vault owners boosted their vault collateralization ratios on average lower than they did on May 19th.

This indicates that they performed more actions with lower amounts relative to the May 19th tail event.

Conclusion

There are a couple of key takeaways from our analysis.

When looking at the number of liquidation protection actions, there is a relatively equal distribution across between DEPOSIT, PAYBACK and single-transaction WITHDRAW|PAYBACK action types. As expected, there is not much multi-transaction WITHDRAW|PAYBACK activity on unsafe vault positions.

When looking at action type distribution normalized by debt size, we can see that whales tend to use PAYBACK actions more relative to other action types. The best explanation seems to be that they are just more likely to have DAI on hand as they’re extending their activity across other protocols in the ecosystem.

The number of liquidation protection actions per vault type maps well with DAI supply distribution. Vault types with longer MCD tenure tend to have a lower liquidation protection success rate.

When comparing Black Thursday with May 19th price drop, it was somewhat surprising to us that there was much more liquidation protection activity during the former event. This can be partially explained by lower average collateralization increases during Black Thursday (more actions with lower increases). Meanwhile, a higher success rate during the latter event is to be expected.

Next steps for us are to automate metrics that would make sense to monitor on a continuous basis and potentially add a vault-level risk score. This means moving towards quantifying potential liquidated debt also by taking into account vault behavior. This could include historical liquidation protection activity, DAI capital availability score and other factors.

Finally, we would like to thank Token Flow (@tmierzwa, @piotr.klis and the team) for giving us access to well modeled data that enabled this analysis.

12 Likes

I still have to make heads or tails of data in graphs. Presenting this data numerically helps me dramatically because then I can twist/turn it however I want in my own spreadsheet.

Main points I want to make here.

  1. Gas cost to access network is incredibly important. Please include that in the analysis here and in general.
  2. There will be no comparison of tx’s that basically were overwritten because blockchain only saves the successful transaction. Given what I saw on the auction bidding and from some discussions of people where were trying to save their vaults, network access on Black Thursday was significantly impaired.
  3. Something that is dramatically different today than during BT was dust size in vaults was much lower. Put simply network access fees relative to size of vaults is such that probably a lot more smaller vaults simply took the hit because the cost to ‘do anything’ basically was 10-20% of their vault.
  4. Please in this liquidation analysis look at collateral returning to vaults which were liquidated. This is an important metric generally as it shows how efficient the liquidation process is in terms of price.

There are other things I am interested in here that I have talked about in the past.

Analysis of if given the auction prices being fetched a particular vault could have had more collateral saved (or the CR brought above the LR) if the liquidation were to happen in stages. Or a better question. Based on the price of auction and collateral returned, what is the minimum amount of collateral that needed to be auctioned to bring the vault back to neutral CR>=LR.

Other points. I have mentioned for a long time it is one of my goals to change the system to reduce liquidations generally. Maker should steward better over people’s hard earned collateral. THis is better for users, better for the Maker system, and better for DeFI as it will mean less collateral trying to find capital.

One idea here is whether during high stress times it makes sense to adjust the liquidation fee to save vaults from coming under the gun. Hence an analysis of how much less liquidations would come to market if a 1,3,5,10% adjustment in the Liquidation Fee as being applied to the LR to vault were to occur is also useful.

Thanks for the analysis and I look forward to analysis leading to changes to the system so more collateral is preserved.

One thing from my own analysis on this. Paying down debt is the most efficient way to protect a vault. It gives the biggest bang for the buck vs. withdraw collateral, or depositing collateral. $1000 DAI in is better then $1000 collateral for obvious CR/LR reasons.

1 Like

Thank you for your feedback, @MakerMan. Much appreciated.

In general we decided to use the data currently available and continue by integrating more external sources to make improvements on the critical points of the analysis. At the moment we only have data on all (successful) vault operations which limits our scope of answerable research questions.

Regarding your main points:

  1. Agreed, gas cost is an important factor to consider in the liquidation process. We will include that moving forward.
  2. Important thing to take into account for comparison. Although as you mentioned, the unsuccessful transactions are not available in the vault operations for us to investigate (eg tendency of vault operations to fail during downfall events over time)
  3. Agreed, dust size is also an important factor to consider and it impacted how vaults acted in BT/M19 events.
  4. This is a good point for further improvement, will also be adding this one.

I will give more thought into simulation analysis ideas that you shared (the impact of multi-stage liquidation process on returned collateral + its extended version, the impact of liquidation fee adjustment during high stress times on liquidations).

2 Likes

Thank you for taking a look at this - and the new analytical data coming out!

My own example analysis was posted over a year ago here when Maker was beginning to think about changing the liquidation process.