As some of you may know we have been reworking the Utopian guidelines completely for quite a while now. We will be splitting the guidelines into moderation guidelines (will be used by moderators to judge the contributions) and contribution guidelines (guidelines to help contributors and much less overwhelming than before). Since we are nearing the deadline most of us have been receiving feedback about the reworked guidelines from others. In this post I specifically touch on the moderation guidelines.
Most of the feedback I've received is that part of the moderation guidelines are too informal and lack specific objective metrics. Since most of the questions on the questionnaire are subjective I've been having some trouble fixing this. This is where you guys come in! I have pasted the (reworked) moderation guidelines below (I've added some of my thoughts as quotes) and would love to hear what you guys can think can be improved.
Moderation guidelines
Evaluation of Development contributions is primarily based on the significance of the added feature(s), the volume of work and the quality of the code.
Contributions should include a description and images (if applicable) of the added feature(s). A justification of technical choices is welcome, but not mandatory. Contributions should also include links to the relevant commits or pull requests, so that it's clear which are part of the contribution described.
Evaluating the amount of work and significance of the added feature(s) will of course be subjective. For the significance it is recommended to distinguish between the addition of core features, cosmetic changes, bug fixes and so on and weigh their overall impact on the project.
Feedback received for the above says it should be more formal and use more objective metrics to define the significance. It also still includes the amount of work, but that has apparently been removed from the guidelines?!
Consider the following during the evaluation:
Submission content
Very straightforward. All of these things should make the post more pleasant to read, while not requiring much effort from the contributor and also making it easier for us to review.
- Can the project's description be found in the current post, README or elsewhere (linked in the post itself)?
- Does the contribution contain a description/pictures/code snippets of the added feature(s)?
- Is it clear which commits or pull requests are relevant to the contribution and were they committed or merged in the given timeframe?
Significance of the contribution
The problem. How do you accurately judge the significance of an added feature? It's even more subjective than the removed "How would you rate the total amount of work?" question (I still disagree with the removal of this), so I really need help with this one. I have no idea how to define this using objective metrics since this is basically all down to the reviewer's experience and knowledge...
- What is the total significance of the added features on the project as a whole?
Quality of the code
Not much to say about this since I tried to make it as objective as possible. If you think anything should be included and/or excluded, then please let me know what and why.
- Does the code follow the best practices of the specific language used?
- Does it include outdated, potentially insecure or low-quality code?
- Does it follow the DRY principle or similar principles? Is it maintainable?
- Does the code include unit tests?
- Is the code formatted correctly and consistently.
- How efficient is the code and does it avoid using unnecessary resources?
Commit messages
Pretty straightforward. It should be clear what was added in each commit and it's better if they follow established commit message guidelines/conventions.
- Is it clear what was added in each commit?
- Do they follow established commit message guidelines/conventions?
Readability
One of the things I've wanted to change for a very long while now is this. We currently judge the quality of comments, but to me it doesn't make much sense to penalise someone for not including comments, while they have made their code readable and understandable in other ways. So instead of judging the quality of the comments, we should simply judge the readability of the code, right?
- Do the comments make the code easier to understand when necessary?
- Are the components, modules, classes, functions, arguments, exceptions and variables named clearly?
- Is whitespace used in a way that improves the readability of the code?
I greatly appreciate any help and feedback, negative or otherwise, so please leave your thoughts below if you have time to spare!