Dark Fix Mechanism: A Proposal for Handling Critical Vulnerabilities in the Maker Protocol

Introduction

The Foundation development team proposes a Dark Fix mechanism, which provides a way to pre-authorize a specific bug fix without exposing its bytecode on-chain. This is important because the exposed bytecode would point potential attackers to the critical vulnerability, allowing them to exploit it.

With this principle laid out, the Foundation development team believes that there are no further reasons to prolong the Governance Security Module delay activation, and recommends to proceed with a second executive vote.

Problem Summary

A Governance Security Module (GSM) with a delay of zero allows for instantaneous patching of critical bugs. However, it opens the system up to a governance attack. Conversely, a non-zero GSM delay enables a set time period to counter governance attacks, where bug patches would sit on-chain for the delay period prior to implementation.

Therefore, we need a GSM delay greater than zero. This solution addresses both the need for a critical bug fix mechanism as well as a defense system to prevent governance attacks.

Recommended Solution

Description

The Maker Foundation development team has proposed a solution called the Dark Fix Mechanism, which involves the following steps:

  1. A bug fix is developed and tested.
  2. Fingerprint the bug fix, referring to it onward as the “dark fix.”
  3. Use the fingerprint to predetermine a dark fix deploy address.
  4. The Maker community votes in the dark fix based on trusted third-party attestation (see next section).
  5. The dark fix is scheduled and a wait period goes into effect until GSM delay has passed.
  6. Lastly, the dark fix is deployed, via an atomic transaction, where the dark fix bytecode is mapped to the predetermined address and thus, instantly patches the bug.
    • Note that if there are any discrepancies in the bytecode, the fix will fail.

For the full technical description of the Dark Fix Mechanism, see the document here.

Attestation

The Dark Fix mechanism introduces a minimal trust requirement in order to avoid an Emergency Shutdown (ES) of the Maker Protocol. In order to implement a solution, the Maker community needs to decide if the third-party attestation is convincing enough to prefer the pre-approval to an Emergency Shutdown of the Maker Protocol.

It is, therefore, a critical consideration as to which form the attestation should take. The Foundation development team has proposed that a committee of MKR holders be selected, and has also discussed suggested alternatives on last week’s governance call:

  • Major MKR Holders Model: The Dark Fix developer team reaches out to a pre-selected committee of MKR holders and shares the details about the critical vulnerability and patch. The selected committee of MKR holders would then sign a transaction confirming their support for the proposed dark fix solution.
  • Independent Auditor Attestation: An agreement is signed with an independent auditor in advance to perform an audit when the need arises. When a critical vulnerability is detected, the dark fix development team reaches out to the pre-approved auditor who will vet and vouch for the dark fix solution.
  • A Community-appointed Trusted Party: When the dark fix development team is ready with their implementation, they will ask the community to appoint an independent trusted party to attest to the validity of the vulnerability and the dark fix solution.

Benefits of the proposed solution

  • A GSM delay can be introduced while maintaining the possibility to safely fix critical vulnerabilities in the system.
  • The fingerprint (hash code) is cryptographically secure and cannot be reverse engineered.
    • For example, if this crypto primitive is broken, the whole internet is broken.
  • The bug fix isn’t deployed until after the delay has passed and then it is immediately applied.
  • Emergency Shutdown can still be triggered at any point in time during the GSM delay.
  • The community can drop the dark fix spell with an MKR majority vote, preventing it from casting.

Note: The dark fix mechanism still openly signals that there is a known critical vulnerability in the system. This proposed mechanism won’t stop an active attack and the GSM delay would still apply to defensive spells.

Protocol Integration Steps

The proposed solution does not require a change to the Maker protocol. This is because the current governance system already enables the ability to propose a spell that would pre-authorize a dark fix based on its hash code.

From a community standpoint, it is necessary to process the ramifications of this proposal and come to a general understanding and agreement on the accepted approach should the need arise in the future.

Conclusions

  • The Maker Foundation development team presented a mechanism for fixing critical protocol vulnerabilities in a system with a GSM delay greater than 0.
  • The Dark Fix mechanism provides a way to pre-authorize the patch without exposing the bytecode, which would point attackers to the critical vulnerability.
  • This mechanism requires one or more trusted parties vouching for the patch that would be voted in by the MKR community.
  • With this principle laid out, the Maker Foundation development team believes that there are no further arguments to prolong the GSM delay activation.

Next Steps

  • Review of the template code for dark fix spells by the security community.
  • Further discussion is required by the community regarding the proposed dark fix mechanism to fix critical vulnerabilities with a GSM delay greater than 0 in place.
  • In parallel, continue with the second GSM activation executive vote. Currently, the community has been asked to signal their support for a second executive vote in this thread.
  • On Thursday, February 20th, the Maker Foundation development team will attend the Scientific Governance and Risk call for questions and comments about this proposed solution.

References


7 Likes

Would this committee need to vote on the executive enacting the darkfix separately or does signing off entail an automatic vote with their mkr?

There’s no requirement to vote, though ideally they would vote in support of the fix. MKR holders signing a message only indicates that they were consulted and believe the bug and the fix to be valid.

4 Likes

Thanks! Thats what I thought the situation was, just wanted to make sure.

With regards to attestations, the most desirable model would be to use all three solutions. An MKR committee signs off on it, an auditor signs it with their PGP key, and a trusted third party as well (it should be a criteria that both have widely distributed PGP keys).

If the attesting MKR holders attest in bad faith, is there a mechanism to easily burn their MKR if the attack doesn’t warrant an ES?

This part seems crazy to me. How can you have assurance that the auditor will not exploit the bug? When there are 100s of millions at stake, a dark fix should be vetted by a very select set of people. Recall that Arthur Andersen aided and abetted the Enron fraud.

Has anyone actually reached out to the large MKR holders to see if they would even participate? I imagine such a process would create massive headaches for them. It may even impact their fiduciary responsibility in some way?

Additionally, wouldn’t the large MKR need to go straight to an independent auditor anyways in order to claim they know that the bug fix does what it’s supposed to. So why not skip the “large MKR holder” step and go straight to the auditor?

4 Likes

After some minor discussion with @wouter in chat I think we should be clear here.

The GSM activation is in response to a threat by ‘flash loans’ making MKR available for a governance attack.

The darkfix addresses an issue related to not exposing critical software flaws and proposing a mechanism/protocol for how the community deals with fixing these problems without revealing them publicly.

While they appear ‘related’ ANY code changes will still have to obey the GSM delay before they can be implemented. What this means if during the GSM delay if there is a critical vulnerability being exploited rather than applying an immediate fix using the GSM=0 or pausing the system (which I understand not to be possible) the only solution is to ES the system.

My point here is that these two issues are almost completely orthogonal to each other and the whole darkfix issue is NOT something that has to be dealt with immediately where as the governance attack appears to be popping up with this ‘emergency’ feeling about it again. IMHO - If the community had a good understanding of who the large MKR holders are, their participation or lack there of in governance, could really come up with a real assessment of the conditions under which a successful governance attack could occur given the state of MKR holders and holdings)

FYI: With respect to addressing the governance attack while the GSM > 0 seems like the easy fix I would like to suggest that the Maker community consider adding in a time delay before MKR deposited into the voting contract becomes active to vote. I suggest using at least a 1 month time period before MKR deposited into the governance contract can actually vote. MKR at any time can be withdrawn or deposited like normal. The reasoning behind this suggestion is that anyone with MKR who wants to participate in governance should be serious enough to leave their MKR for at least a month before they then gain the privledge to participate in governance. This completely eliminates the vast majority of instant kinds of governance attacks far better than the GSM can. Now whether one actually wants to lock MKR into the system that votes before it can be withdrawn is another question (i.e. whether voters are required to have a long term financial stake by having their voting MKR locked or not).

1 Like

The auditor would be vetted beforehand and the relationship would be contractual with an explicit vulnerability response process. Personally, I like that this method keeps the number of people in possession of the vulnerability limited.

Being under contract isn’t foolproof, but it does introduce a significant amount of accountability to both parties. I do recall Arthur Andersen, I also understand that the actions of that firm don’t reflect those of the greater assurance industry.

Could you please elaborate on who the select set of people should be? How can we have assurance that select set of people will not exploit the bug?

3 Likes

I don’t feel a priori comfortable with darkfixes. Now that the idea is out, some darkfix has a reasonable chance to be offered to the community when an emergency finally occurs. So it seems important to find some consensus view of how darkfixes should be evaluated when they arise. In particular, we should ask at least the following questions:

  • How does the darkfix risk compare to ES risk? Using an emergency as an opportunity to run an ES seems healthy. Having an expectation that ES is very much on the table would motivate us to push towards creating ES-related tooling in the meantime. However, discussing whether the emergency would or would not immediately be solved by an ES could give unwanted hints to potential attackers, so that discussion itself would present a risk.
  • How much can the darkfix’s effects be constrained? For instance, could a darkfix be executed within a public wrapper that checks that the effects of the darkfix occured within a bounded area? Again, checks that are too precise would be a dangerous hint.
1 Like

What stops any groups of devs from proposing darkfixes? How do these large mkr holders really know who is reaching out to them?

It’s difficult to weigh the risks of something of which we are unaware. The Emergency Shutdown (ES) module is a contract just like any other MCD contract. We stand behind the ES as a tool, but it would be remiss of us to claim it an infallible mechanism. There was literally a bug discovered that only affected the system during Emergency Shutdown.

Let me be unequivocal: ES is very much on the table.

Furthermore, community development of additional tooling around it is highly encouraged. What kind of ES tooling do you have in mind?

This is certainly a point of discussion. You made note of it yourself that parameters could potentially narrow the field for exploiters. This public wrapper also introduces additional code and complexity.

On cursory thought, I’m uncomfortable introducing parameters given my opinions on the sophistication of today’s exploits.

I’m open to persuasion. What would be an acceptable trade-off from your perspective? What constraints would ease your concerns, yet still maintain acceptable concealment for the bug fix?

I guess I’m wondering if the dark fix mechanism can be split into a public and private part. For example, suppose the vulnerability only affects auctions. Could a dark fix be proposed that is limited to the auctions part of the Maker code while the specific fix is kept confidential? Is that technically possible?

Nothing is stopping them. The code is open source: https://github.com/makerdao/dss-darkfix

Performing good due diligence is a responsibility of the large MKR holders.

1 Like

Im not sure what communication methods you guys would use, but couldn’t there be a concern that someone impersonates the Maker devs trying to push through a malicious dark fix? I think the communication aspect of this proposal could use fleshing out.

2 Likes

Thank you for your reply, and sorry for responding so late. I think I wasn’t clear that I meant we should ask those questions when a fix is on the table. And I appreciate the clarity of your position on ES. I haven’t given enough thoughts to ES tooling but: a redeeming interface, an ES guide with actions to take / what’s at risk / what’s safe for each type of ecosystem participant.

I don’t think I can give an informed view of what would be acceptable tradeoffs on a darkfix’s constrained vs. information leaked by those constraints right now.