MIP13c3-SP6: SourceCred Funding

Hello,

I have asked @s_ben and He told me this might be the most appropriate post to extend, to discuss SourceCred.

My experience with SC: I have joined Maker’s SC in mid September and have so far received ~300dai for the October payment and about ~350dai for the November payment. So I have a limited but not-null experience.

What do I think about SC supporting forum activity?

I was very skeptical initially. I completely changed my mind after 2 months.

The rewards are not astronomical but still not negligible (btw, I live in France, the perception will obviously be different in other countries). They are not enough for taking 'forum-farming :stuck_out_tongue: ’ as a job, but they are enough to give a sense of reward, and satisfaction.

The effect for me was the following: I moved my MakerDao’s interactions from the chat to (mostly) the Forum.

I feel the discussion is more organised, more precise, even more polite in the forum vs the chat. And it is also rewarded.

Would I increase the rewards for Forum activity?

As others, I have doubts about this.
But since the community is still very small and that we want to grow significantly, I would like to try to increase the rewards by a 2x or 3x factor in the next 3 months. Just to see if this generates some significant difference.

Would I add sourceCred to Maker’s chat?

NO. Maker’s chat is nice as it is. It’s very quiet. People are generally very polite and provide good information. I don’t want to see that place spammed.

Would I add sourceCred to GitHub?

I am not entirely sure. There might be 1commit of 50lines of code deserving a reward of 10k usd, and another commit of 50lines of code which is mostly copy/paste.

Unless there is a way of raking the value of git commits, I would not support SourceCred on github.

Would I add sourceCred to something Else?

I would like to have a system (perhaps integrated in the forum) where users could actively ‘push buttons’ to signal that they want to give MORE CRED to a certain user. This would provide a lot of flexibility to reward certain specific contributions (code, art, communication efforts, whatever) which are unlikely to have their one dedicated Cred system.

For example, recently @hexonaut has recently made an important contribution (PSM code). He was already paid for this, apparently, but still I would personally like to signal my desire to reward him with additional cred.

One simple implementation could be: if a user gets more than 15 ``signals’’, their cred collected that month will be doubled.

What else would you want from SourceCred?

  1. receive cred more frequently.

I don’t particularly like being ‘paid’ once per month.
Better would be a system where the cred accumulates continuously and, whenever I want, I can ‘extract’ the cred (i.e., receive some DAI) available. Of course I will have to pay the ethereum fees.

Or at least, weekly payments. Monthly payments are just not very much ‘blockchain technology’ honestly :stuck_out_tongue:

  1. Be paid in xDAI.

It’s annoying to pay fees. I personally would prefer to receive my DAI in the xDAI network where fees are negligible and, anyway, I can move my DAI to Ethereum whenever I want.

SourceCred is already working with other projects in the xDAI network (e.g., 1Hive), so this should be possible.

  1. Use a weighted average mechanism that adapts slowly.

To avoid the ‘pressure’ some might feel (“shit, I need to post something or I will not get cred this month!”), I think it would be a good idea to let Cred generation decrease slowly over time. So, e.g., if a very active user usually get 500cred/month , and in November (s)he doesn’t post anything, they should still receive something like ~450usd.
This of course would drop more if they are inactive also in December (eg. to 350, then to 200, then to 0).

This is a way of rewarding in a reliable way engaged users, and should be done especially now that the community is so little.

What didn’t you like from SourceCred?

I didn’t like the communication via the Forum.

It’s hard to find the posts with the right information. You always need to search and find the informations. I know engaged community members who didn’t know how to register!

The weekly posts of @s_ben are great, with a lot of infos.

But it would be nicer to have a webpage with all the basic informations (how to register, view the Cred history assigned to users, learn how much DAI is being given every month, etc), perhaps in the MakerDAO homepage.

6 Likes

Some great feedback @iammeeoh! Responding to your comments below, also ended up relaying some updates MKR holders may find useful.

People are invariably skeptical of SourceCred at first, and for good reason. However, in our experience, if applied responsibly in communities that can handle it, people tend to come around. The Maker community has been amazing in this respect, as contributors have offered lots of intelligent critiques and ideas for improvement. Already, feedback has resulted in two improvements (like-minted Cred and minting Cred on Discourse Trust Levels), that have improved the algorithm’s performance significantly.

We’ve see this in other communities as well. People respond to financial incentives. Which sounds totally obvious. But we’ve been surprised by the effect we’ve seen so far. For instance, when SourceCred increased weights on our Discourse, we saw such a surge in activity we had to dial it back.

As detailed in the final report on the first phase of the trial, the survey results showed that contributors were fairly mixed on increasing payouts. The SourceCred team thinks it would be valuable to experiment with increasing the payouts, in addition to potentially tweaking the payout formula to increase rewards for newer contributors. We’ve just suggested this to the Maker-SourceCred working group actually, and are putting together some analysis to support this hypothesis. At some point, I think we’ll need to increase payouts a bit just want to keep rewards the same (assuming Maker wants that), as the trial has proved popular, and we’ve seen a steady stream of new opt-ins (41 enrolled currently), which dilutes the DAI per Cred (a proxy for work) contributors are getting.

That said, some are reporting that the payouts are doing the job at current levels. The optimal amount could be less, depending on how you measure success. Ultimately, it will be up to MKR holders.

For now, we only have a Discord plugin. So a rocket chat plugin would need to be created to support this.

Your reservation about spam is also warranted. In our experience thus far, especially if you’re flowing a lot of money, SourceCred will increase activity considerably. It’s also sometimes challenging at first for communities to understand Cred flows in chat, and come up with norms around that (i.e. “Crediquette”). That said, since launching the trial with Maker, the Discord plugin has matured considerably. In addition to our internal dogfooding, the Discord plugin was recently deployed in a community (1Hive) whose token mooned, causing a significant amounts of money through SourceCred. This resulted in a bunch of new contributors attempting to game the system with spam. This forced us to build and deploy countermeasures that appear to be holding.

GitHub is our oldest plugin, but also immature in some respects. As you point out, some commits can be way more valuable, and the algorithm doesn’t do a great job of reflecting that. This is exaserbated by developers having different patterns of contributing. We have ideas for addressing this, including the inclusion of heuristics such as number of lines changed (before anyone comments that this is a bad metric, we agree; this would just be one input into the valuation), introducing more review such as emoji reactions, etc. That said, in my experience demo’ing the GitHub plugin for communities, even though different patterns of contributing can confuse the algorithm, over time, the results tend to converge on useful results. Also, gaming has not been a problem thus far with the GitHub plugin. Presumably because Cred is only minted on merged PRs, introducing a level of review by trusted community members. In the post that started it all, I ran an earlier version of SourceCred on the makerdao/mcd-cdp-portal repo (hosted instance).

Never got any feedback from devs, but would be curious if anyone has thoughts on this.

Another thing to note is that, if using another plugin, contributions in GitHub are connected to connections elsewhere in the graph (e.g. from Discourse). So, valuation of contributions from a developer on GitHub (or non-developer contributing Issues or comments) will be informed by contributions elsewhere. Or, the plugins could be kept separate if preferred.

We would like this as well :slight_smile: SourceCred is great at capturing work that leaves artifacts on supported platforms (code, posts, comments, etc.). However, it misses lots of work that happens outside of these platforms (art, logistics, emotional labor, admin, project management, etc.). To address this hole, we are currently working on a plugin called the Creditor, which allows for the insertion of arbitrary contributions into the graph. This may not be ready for a few months however.

Technically, you can increase the rewards for a contribution using your own contributions. When you react to or engage with another person’s contribution, an edge in the graph is formed between you and that contribution. Some of the Cred from your other contributions then flows to the other person’s contribution, and therefore its author (e.g. @hexonaut). But, presumably you’re imagining something more direct and higher impact. As is SourceCred. In fact, SourceCred’s cryptoeconomic model (how it hopes to become economically sustainable) is based on a feature called boosting, which is similar to this. Basically, you burn Grain (or DAI in Maker’s case) on a contribution. This creates new Cred in the contribution’s node. This gives the author of the contribution Cred, and also incentivizes others to build on it, as they get a piece of the boosted Cred (and eventually DAI). Alternately, we could also use something like the Creditor plugin when it’s ready, in which case you could lobby the community to give more Cred to whatever contribution you feel is undervalued, using whatever governance process you prefer.

SourceCred is architected so that you can add any type of heuristic like this to any of the plugins.

The monthly payouts are simply due to admin overhead. Payouts are currently done manually, and distributing costs time and effort. More automation is on the way though. We don’t like doing it manually either :sweat_smile: We’re currently overhauling the admin interface of SourceCred instances so that they can store ETH addresses and admins can schedule automated payments. Weekly payments makes sense. You could even do streaming payments with something like Sablier (which would be awesome), one concern is that if the system is gamed, there’s no stopping the payments. Currently, if someone starts overtly gaming, we can simply not send the payment. When SourceCred becomes more robust to gaming and battle tested though, adding this feature could make the system more trustless for contributors.

Personally a fan of xDAI. 1Hive is paying contributors over the xDAI network. However, the UX of moving tokens between mainnet and xDAI is still a bit painful according to reports I’m seeing from 1Hive community. As we’ve already received a bit of (fair) criticism about the onboarding process being high friction, I’m not sure adding more friction with another network if wise right now. We’ve also started using a new batch payment tool (disperse.app) that brings the fees down. Gas fees for Oct payouts were like $7 I think. xDAI is definitely something we’re following though, and open to exploring this more if there is demand from the community for it.

This is an example of us needing to explain the algorithm better, as this is basically how it works currently. Cred from contributions does exponentially decay as you suggest. Additionally, the payouts are currently calculated as a function of 75/25 (lifetime Cred score / immediate (las week) Cred score). In practice, this has meant that payouts decay pretty slowly. Especially for those with high Cred scores. In fact, we’re thinking of switching up this ratio, as OG contributors that haven’t contributed for a month or two are still getting about the same as when they were contributing. Which I think has been good up till now, as it’s given some “backpay” to long-time contributors that worked for free in the past. But this could also be discouraging to newer or more active contributors. We’re thinking of trying and experiment actually of changing the ratio to 50/50, so that everyone still get “royalties” into the future, but the algorithm will be more responsive to new contributors.

Helpful feedback. Where would you suggest posting more? I’ve been hesitant to post in chat, as the relevant channels seem generally quiet (e.g. #governance-and-risk), and I don’t want to spam.

Could also probably engage more with Maker gov comms (e.g. Maker Relay, @mkrgov on Twitter)…

We’ve also talked about coming on the Governance and Risk meeting to talk about funding SourceCred. @LongForWisdom just let us know!

This would definitely be useful. We are working on an overhaul to the instance system that includes some of this functionality. I can’t say right now how much, as that project is in flux, and the devs are very busy :sweat_smile: But if you (or anyone) would like to follow this work more closely, I would recommend popping over to our Discord, posting on our Discourse, or attending our weekly community call (every Tues at 11am PST) and sharing your thoughts. We’re very friendly:)

5 Likes

Thanks @s_ben for a very thorough answer to the feedback from @iammeeoh

This formula answers why some weeks I get paid for practically nothing. I will dare to make to make a suggestion: cap the lifetime cred down to 1 year. That is not in favor of OG contributors like me but crypto moves so fast that the value of contributions older than a year (or so) are not that valuable anyway. And it will be more fair to new members.

Interesting. Yeah we’ve put a lot of thought into creating a payout mechanism that will be robust generally. But of course there are many ways to map Cred scores to payouts, and wouldn’t be surprised if the community ended up coming up with something more customized to its needs. Could be something we experiment with in the coming months.

1 Like

One more thing - SourceCred is mainly centered on the forum, trying to encourage activity, involvement and contributions but lately there has been almost too much happening to keep track of it all.

Ask yourself this question: if the number of active participants increased from the present 30-40 to about 100-150 (the Dunbar limit) what would happen to the forum? Right now I think the answer is that it would become an unworkable jungle. Are there any thoughts on this over at SourceCred?

I will post this in the SourceCred chat to get more opinions, but my impression is that 100-150 on the Maker forum would be fine. If you’re below Dunbar’s number, gaming and other malicious behavior should be relatively easy to spot. Maker is helped by the fact that contributing meaningfully to Maker governance is a high bar. Malicious actors getting enough :heart:s to do real damage would be difficult. I imagine that any gaming attempts (and presumably there would be some), would be seen coming with plenty of time to mitigate. The two main defenses being: 1) solid moderation from @LongForWisdom and other core members, and 2) only minting Cred on higher Discourse Trust levels (a feature we developed in collaboration with Maker).

Actually, I wonder how close to Dunbar’s number Maker is already. It has seen growth since the trial started. I recently created the below bar chart race for the whole history of the forum, which I didn’t post here because it’s ugly and unscientific. But it does show a nice fattening long tail of contributors…

You could see more sophisticated attacks if Maker started distributing considerably higher rewards (say, in the hundreds of thousands per month). In which case you might need to build additional countermeasures, step up the moderation.

As a point of comparison, a few weeks ago, 1Hive started using SourceCred to distribute its token. At the time, the community was relatively small (only a dozen or so regular contributors). It was using all three plugins (Discourse, Discord, GitHub). The rewards were averaging around $2k/week worth of its token. Then the token mooned. Overnight, the tokens distributed were worth over $60k/week at their height :grimacing: :grimacing: :grimacing: They got several thousand new contributors in a few short weeks, and of course, there was gaming. Including on Discourse (mainly Sibyl accounts liking each others posts). I’m not going to say it was pretty. But the community rallied, worked with SourceCred to deploy new features to migitage the worst gaming (mainly on Discord), and the remaining cases were handled with effective moderation no the social level. They have since reduced the reward to more reasonable amounts, but are still using SourceCred. This unplanned test makes me think that Maker, with a more established and civil community, could probably handle a lot of growth before it saw real problems.

5 Likes

ah - you look at it from a gaming SourceCred perspective.
I do not - I look at it from the perspective of keeping an overview of the information flow.

I regard gaming SourceCred on Maker as rather improbable - as for just the reasons you outlined.

EDIT: but it is good to know SC can handle it.

2 Likes

I think, under the ‘keeping an overview of the information’, it is not SourceCred that should find the solution. SourceCred will just be one of the tools contributing to a solution.

If things at MakerDAO are successful, in the next 4-5 years we should have plenty more than 150 forum users.

This reminds me of scientific communities. They usually start small. The eventually grow and compartmentalise. And the whole larger scientific community is kept together by a system of conferences, peer-reviewed journals/summaries, grant-systems, prizes, etc.

In other words, we will have to develop a ‘scientific governance’.

What is encouraging, is that this was planned from the very beginning. We are still very very early its development, but it’s coming.

2 Likes

Right. I would view SourceCred as a source of data that can be used to better understand the community, which could help communication and coordination in some ways. But it doesn’t solve these issues on its own.

1 Like

Hi guys. I don’t know if it’s the right place to ask that.

I’m very new here. I’m already involved in several other DAO. And we thought about using SourceCred. I have to say I was very sceptical , so i made some research and it seems you are very advance here .

I also sign in to try and see how it works.

I really like this system. Already spending few hours here, the quality of each post is amazing. So it’s very time-consuming but you learn a lot. In that aspect i have a mixed opinion. To me, when you participate in a DAO like that, it’s because you like the project and you want to improve it. So first i didn’t really understand why you want to reward people for that ! I mean, i’m doing it anyway. And if people start to be involved just to be paid , i think you loss the main interest of a DAO.

But then i saw the quality of the majority of posts, it’s amazing. Some people really spend time to write/explain . So i that aspect it makes sense to be reward.

But then my question is: How does that work for other users. Let’s take my example :joy:
You spend A LOT of time reading posts. You vote to all polls. But you don’t really write comments/topic. Mainly because people already do it very well. You are still very important for the DAO, and you participate very carefully. But how this is taken into account ?

Hi French compatriot ^^ On that side. Do you have extra reward for people really involved ? I guess they are a kind of moderators. Do they receive something else ? or everything is included in SourceCred ?

4 Likes

Hey man! Welcome aboard. You’re best bet is to talk to @s_ben or @panchovillage from SourceCred. Totally–its such a good way to incentivize DAOs IMO.

2 Likes

I likewise was looking at SourceCred for another protocol (YFI) and have been pleasantly surprised by its implementation. So much so that I spend a lot more time contributing to Maker DAO rather than Yearn Finance. I think it is a nice pat on the back for making meaningful contributions to governance. Hopefully more protocols move in this direction.

3 Likes

I think what you’re experiencing is part of a trend @ejbarraza. SourceCred tends to increase engagement generally. I’m also generally finding myself spending more time where I get paid to post these days. E.g., when surfing Reddit, I spend more time on r/ethtrader and r/CryptoCurrency, as both are rewarding users in ERC-20 tokens ($DONUTS and $MOONS) that I can sell (and have) for DAI.

A couple days ago, a user on r/CryptoCurrency was wrestling with how much it had affected his behavior, and if it was good.

I say being paid for creating the same type of content you were previously creating for free is a good thing, though obviously I’m biased :slight_smile: :sweat_smile:

Yes i guess that’s the point. Rewarding people who already contributed and will do even without rewards. And people who will only participate for that .

ps: didn’t know for reddit ^^

I would say these are big, outstanding questions facing all incentive systems. Lots of interesting research out there around intrinsic (internal) vs extrinsic (external, e.g. money) motivation, and the problems introduced by financial incentives in open source projects. I don’t think anyone has really solved this. I’ll just say that we’ve thought a lot about the surrounding issues, and so far, in the communities that have flowed meaningful sums of money according to Cred scores, we’re seeing generally positive results.

For our final report on the first three months of the trial, we surveyed Maker contributors and asked them if they contributed more due to SourceCred, and most reported they contributed some amount more:

I suspect that as rewards increase, we’ll see some shifts in dynamics. If the rewards are large enough, we’ll start to see some people treating it more like a job, which may change their behavior. We’ll also see more gaming attempts. I think Maker is a great candidate to try larger payouts, due to its high barrier to entry (contributing to Maker governance meaningfully and getting likes is hard), general professionalism, and good moderation at the social level. SourceCred’s active support will also allow us to mitigate issues as they arise.

Most of the complaints we’ve seen so far haven’t been due to paying or not paying people, but people feeling some are being over or under paid. For instance, when 1Hive’s token mooned, increasing the rewards way beyond what anyone was imagining (from $2k -> $60k/week), there was a large influx of new contributors gaming the system. We were able to mitigate this with new features and active support, but while the gaming was ongoing, people that were putting in lots of honest work were naturally upset to see gamers making more than them. Likewise, contributions that the algorithm cannot currently capture well (emotional labor, project management, art, etc.) are generally undervalued. To an extent, this will always be an ongoing challenge, as we are always improving score quality, filling gaps in what we can value, and generally improving the system. I think the key here is keeping the scope limited to what SourceCred is good at, managing expectations (SourceCred isn’t a magic bullet that will solve all community problems), and having active support when pushing those limits with new features and use cases.

2 Likes

Maybe that is a good reason to make payments in Dai

Why ? how will it prevent influx of contributors gaming the system ?

I want to point out here that almost all the gaming for 1hive distributions was on the discord plugin, rather than discourse. Discourse is much less susceptible to gaming. Also, distributions were MUCH higher than Maker’s distributions are right now. Also Also, even the gaming that was done on discord has largely disappeared after implementing minting based on member cred scores.

3 Likes

Well for one thing it will be easier to keep everything in dollar terms that way I imagine. For example, what if on one particularly good day MKR goes up 5X over 24 hours? Will payments going out that day immediately be reduced to 1/5th the amount of MKR?

Ho ! Now I understand why you said that. I understood that the increase from 5k to 60k/week was due to an increase in the pool (Funding). Not Because of an increase of the price of the token. I don’t know what they use (I thought it was xDai), but if they use a volatile token, i guess it’s better to speak in terms of amount of token, and set the number of token shared in terms of number of token, and not in terms of value.