SourceCred Trial - Final Report

Introduction

In an effort to decentralize and explore new options for sustainably incentivizing work on the Maker Protocol, the Foundation tried using SourceCred to identify and reward meaningful contributions to Maker’s forum.

The following summarizes the results of the trial, presenting objective and subjective success metrics laid out in SourceCred’s original proposal, as well as observations from the SourceCred team and plans for the next phase of the trial.

Executive Summary

The trial has been a success.

Both the objective engagement metrics on Maker’s Discourse forum, and the subjective feedback in the form of user surveys from the Maker community seem to indicate that the SourceCred algorithm will be a valuable tool for rewarding contributors in a decentralized and incentive-compatible way.

As a result, the Maker-SourceCred working group has decided to extend the trial for an additional three months. If the trial continues to be successful, a MIP will be put to MKR holders to decide whether SourceCred should be used to pay contributors directly from the protocol.

Objective Metrics

The original proposal placed importance on the engagement metrics available in the Discourse Admin Panel. It was agreed that an overall increase in these metrics would serve as a good indicator of success. Conversely, decreases in engagement and/or overt gaming would be grounds for cancelling the trial.

Our dataset consisted of a CSV file output by a Maker Discourse admin (@LongForWisdom) which provided daily numbers for every relevant engagement metric in the Discourse admin panel. The table below shows the percentage change for the most relevant metrics we analyzed. Note that this is using a 1st order linear regression model, which provides only a rough estimate.

Metric % Change (6/1 - 9/1)
Visits + 28%
Topics + 88%
Posts +120%
Likes +130%
Daily Engaged Users +70%
New Contributors +55%
Signups +2%
Daily Active Users (DAU) / Monthly Active Users (MAU) + 2%

Below are charts showing a subset of these metrics. The raw CSV file with the full dataset, scripts for generating the tables and charts below, and additional charts can also be found in the Appendix.

Daily Engaged Users

Topics Created

Post Created

New Contributors

In addition to these metrics from Discourse, the amount of Cred created over time can also serve as a proxy for overall activity. If there is more Cred minted, there are more contributions that the SourceCred algorithm finds to be meaningful within the community.

Below is a chart of total Cred created over time for the history of the Maker forum, which again shows a marked increase.

Total Cred

In general, we see an increase across all engagement metrics. There was also a slight growth in new users, though this was less pronounced.

Subjective Feedback

To gauge subjective feedback, participants were sent a short survey at the end of the initial phase of the trial. Of the 24 contributors that opted into the trial, 17 responded. While the survey was opened up to the community at large, the only responses were from community members who had opted-in to the trial.

Below are the survey questions and summaries of the responses.

Contributor Survey

  1. How familiar are you with how SourceCred works (parameters, how to earn Cred, etc)? 1 = “not at all” and 10 = “very familiar”

RESPONSE: (17 responses) 5.5/10, General vibe is “somewhat familiar” but cloudy on details, might be good to follow up individually and offer links/information

  1. How satisfied are you with the Cred scores assigned to contributions and contributors on the forum? 1 = “scores were terrible” and 10 = “very satisfied”

RESPONSE: (15 responses) 7.1/10, General vibe is that it seemed ok enough, no red flags but some concerns about volume being rewarded over quality

  1. On a scale of 1-10, how positive an impact has SourceCred had on the Maker Community? 1 = “no positive impact” and 10 = “very positive impact”

RESPONSE: (15 responses) 6.6/10, General vibe is that it’s a bit early to tell, some references made to a lack of SourceCred related buzz on the forum

  1. How would you feel about expanding the use of SourceCred within Maker to other platforms such as Github and Maker Chat? 1 = “terrible idea” and 10 = “great idea”

RESPONSE: (16 responses) 6.2/10, General vibe is quite polarized for and against, Github raises the most questions and possibilities, some view chat as a valuable addition but some view it as more socializing that shouldn’t be rewarded, a reference was made to directly rewarding MIPS

  1. Do you, or would you prefer to receive MKR through SourceCred rather than DAI?

RESPONSE: (16 responses) 3 for MKR, 3 for DAI, 10 for No Preference, of those No Preference votes 3 advocated for 50/50 allocation, and several references made to leaving it up to users

  1. Do you have any additional feedback about the SourceCred trial or ideas to improve SourceCred generally?

RESPONSE: The general vibe is “keep on doing what you’re doing”, but several ideas may bear surfacing: UI/UX/Dashboard improvements, several references to a need for FAQ/s ELI5’s and general introductions, plug-ins for RocketChat and other social media platforms and IRL meetups, expand Cred to participation in Polls on the forum and Governance Voting in general

  1. On a scale of 1 to 5, How was your onboarding process for receiving DAI through SourceCred? 1 = very difficult and 5 = very easy

RESPONSE: (16 responses) 4.6 , several references to UI/UX improvements and potential privacy concerns

  1. Did the amount of money you received cause you to contribute more to the Maker forums?

RESPONSE: (15 responses) Most said it caused them to contribute some amount more, though a couple said it did not, due to simply not having time due to other commitments, or their primary motivation not being related to money. Contributors higher in the ranks (and thus paid more), were more likely to report contributing more. Highly paid professionals said it did not approach what they typically get paid hourly, and therefore contributing full-time was not viable at current payout levels. Some report slight changes in behavior; 1 person reported “consciously liking more posts”, and another reporting, “I would say it helps for more tedious posts.”). Some want to experiment with higher payouts, while some think current levels are fine.

  1. Do you feel the time you spend on Maker Governance is adequately rewarded through SourceCred?

RESPONSE: For highly paid professionals, the amount was not particularly material, though were generally appreciative of the “pocket change” or “side gig”. Some thought the reward was adequate, a few thought a pay bump would increase participation.

Conclusion

We saw a clear increase in engagement and activity on the forum. It is not clear how much of this was attributable to SourceCred however. The trial period saw a boom in DAI demand due to the yield farming craze, leading to issues with the peg and an increase in governance work overall. This may have driven much of the new activity. We did however have several survey respondents report increasing their contributions due to the payouts, and the trial was generally well-received by contributors. In any case, it is a reminder that MakerDAO’s governance will only grow over time as the DeFi ecosystem expands, as illustrated by the post by Planet_X which inspired SourceCred’s initial engagement with the community.

We believe SourceCred has scaled the amount of work it can compensate contributors for while also furthering MakerDAO’s goals around decentralization. Instead of increasing the workload (and potentially headcount) of Foundation employees engaged in a more centralized process, SourceCred has decentralized the valuation and compensation of contributions to the governance community at large participating on the forums.

Next Steps

In the next phase of the trial, SourceCred will continue working with the Maker-SourceCred working group to tweak the algorithm and parameters as necessary, improve the Maker SourceCred instance UI, create documentation and educational content to make SourceCred easier to understand, and design a technical integration to propose to MKR holders via the MIPS process.

Our next step will be gathering additional feedback from the Maker community and scoping out a technical integration. If you have any thoughts or opinions on the trial thus far, plans for the trial extension, technical integration, or anything else, please let us know by filling out the survey on the forum, messaging us privately at @sourcecred-trial-adm, or dropping by the SourceCred forum or chat!

Resources

Appendix

The full raw dataset used to create engagement metrics and create charts can be found here.

Below are additional figures not included in the Objective Metrics section.

Visits

Daily Active Users (DAU) / Monthly Active Users (MAU)

Likes

Signups

13 Likes

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.