[Nov29 '21] Post-Mortem: Impaired Poll Voting On Governance Portal

Summary

Voting on governance polls was temporarily hindered due to a bug in the Governance Portal that was triggered by the latest weekly governance polls posted on November 29th '21 at 4pm UTC. The DUX team was able to deploy a hotfix 53 minutes after the bug was reported, after which a root cause analysis was conducted and more changes were applied to mitigate the issue and prevent similar ones from occurring in the future. The issue has been resolved and no governance poll results were affected by the issue.

Background

Today, on November 29th 2021 at 4:03pm UTC the DUX team was contacted on our own Discord server regarding an issue with the polling feature of the Governance Portal. The application threw an error on both the landing page and the polling page, rendering the polling feature unusable.

Upon reading the message the DUX team jointly investigated the issue and concluded that the issue was caused by an empty poll tally data structure, that was affecting a UI component—a recently added feature of the Governance Portal that visualizes the current tally on each poll card. (screenshot below)

Detailed Analysis

The poll tally component was included in the v0.2.4 release. The data required for this feature to function is calculated in the backend using raw voting data that is queried from gov-polling-db; a custom (but open-sourced) blockchain scraper service which we host. This function of calculating poll tally from raw data did not account for poll vote options that had zero MKR voted on them.

When the latest batch of Maker governance polls were uploaded on November 29th at 4pm UTC, the polls had no voters yet and thus the vote options had zero MKR voting on them. This caused the aforementioned function to fail and generate an error, which then lead to the Governance Portal application crashing and returning the following error to the user:
unknown
This error occurred on the Governance Portal landing page and polling page. Even though the Governance Portal was still accessible and most pages were still functioning correctly (eg. delegates overview page, executive page, account pages, ESM module), the polling feature was rendered unusable until a hotfix was deployed.

Bug Discoverability

The bug has been live on the production environment of the Governance Portal since November 23rd, when v0.2.4 was deployed. Even though we conduct testing with mock polls before deploying a new release of the Governance Portal, these mock polls do not include options with zero MKR voted on them. Such a test would have uncovered the bug that caused the issue today.

Mitigation Steps

53 minutes after the bug was disclosed, DUX team deployed a hotfix for the issue that would hide the poll tally component for polls that have 0 MKR voted on one or more options. This hotfix mitigated the issue and made it once again possible to interact with Maker governance polls.

In addition to the hotfix mentioned above, we have performed the following changes to the Governance Portal codebase to mitigate the issue:

  • Added a default value for empty polls on the poll tally instead of an empty object;
  • Hidden the UI components in conditions that were causing the crash;
  • Wrote tests to ensure data consistency on the API on different poll tally statuses;
  • Changed dai.js response to return consistent results even when there are no votes.

In addition, we have defined the following actions that would prevent similar events from happening in the future:

  • Create an error handling mechanism for the Governance Portal so that crashes are isolated;
  • Improve the manual testing setup on Göerli testnet to account for the aforementioned scenario;
  • Refactor the tally calculation results to remove a redundant object/array.
13 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.