Utopian.io is the first platform to implement a decentralized review system that is not based on user stake alone, but also utilizes dynamic reputation.
To improve the scoring system, we created new and improved questionnaires for all categories.
Streamlined questions, answers and scoring weights
The score a contribution receives from the community depends on the answers selected in the category-specific questionnaire. The questions crafted needed to not only be relevant and cover all the various aspects used to review a contribution's value to the project it is made for, but also clear enough for the community to rate fairly.
Both questions and answers needed to work in cohesion to measure value. For this, we had to use a completely different approach in formulating questions and answers, and the weights they have on potenial contribution rewards.
Well-Balanced Scores and Weights
In all categories, we redesigned questions and answers to make sure that the scores and weights associated with them correctly evaluated contributions. Community Managers shared their expertise and experience in crafting the questionnaires that were then edited for uniformity and clarity across categories.
Clearer and More Relevant Questions
All the content in the questionnaires was edited to ensure that the wording was clear, relevant and consistent. Questions were compacted to cover the important aspects of review and to make the contribution scoring process easier to understand.
What This Means for Scoring
The actual scoring process itself has not been changed. The changes applied were to the content only. For more information on how scoring works, read the detailed post we recently published on the topic.
The new questionnaires were designed to improve the scoring process and we anticipate that this will result in more accurate evaluation of contributions on Utopian.
Studies We Used
Our team utilized various studies to compose the most relevant questions, answers and associate scores to each.
In formulating the spread of scores among the various questions and answers, lessons where taken from these two studies:
Tilmann Gneiting & Adrian E Raftery (2012) Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association, 102:477, 359-378, DOI: 10.1198/016214506000001437
Staël von Holstein, C.S., 1970: A Family of Strictly Proper Scoring Rules Which Are Sensitive to Distance. J. Appl. Meteor., 9, 360–364.
The ideas we found helped us to create a better distribution of scores, while making sure the questionnaires encourage people to be careful and honest, with accurately defined levels/measures of quality.
An Important Milestone on Our Roadmap
The mass of platform upgrades introduced in the past weeks are creating a foundation for a robust evaluation system that will allow for the curation and organization of all of the community's contributions. The reputation system we integrated is our first attempt at building a powerful and intelligently designed review system for Utopian.io.
It's a major iteration of the first few steps Utopian.io is taking to mature out of beta; and realize its vision of creating an Open Source Economy.
Got comments? Talk to us on Discord or write a comment below.
Let's make Open Source Economy a reality. Together.
Top Sponsors
 @freedom - Vote Witness |  @misterdelegation |
 @ned |  @wackou - Vote Witness |
 @xeldal - Vote Witness |  @fenghuang |
Top Moderators
 @tykee |  @jmromero |
 @syedumair |  @zoneboy |
 @samhamou |  @codingdefined |