This editorial synthesizes 18 independent visions for reforming scientific evaluation into a set of design features, finding broad agreement across diverse disciplines and perspectives. The paper's enduring value lies not in any single proposal but in demonstrating that the research community had, by 2012, already converged on what a better system would look like, and that the obstacles were cultural and political, not technical.

The methodology is sound for what it is: a qualitative synthesis of an invited collection, not a systematic review. The visual summary table mapping design features against the 18 visions is effective and instantly communicates the degree of consensus. One limitation worth noting is that the 18 visions were solicited rather than sampled, which may overrepresent perspectives sympathetic to open evaluation.

A more adversarial inclusion - voices defending the status quo - would have strengthened the analysis.
The paper identifies 17 design features that an open evaluation system should consider. Nearly all of them - open reviews, post-publication evaluation, reviewer incentives, transparent ratings, decoupling evaluation from journals, quantitative peer ratings, and community-driven scoring - have remained unimplemented in mainstream publishing thirteen years later. The gap between this consensus and the actual state of academic publishing in 2026 is itself a damning indictment of institutional inertia.
The most interesting tension in the paper is around anonymity. Most visions favor transparency, but the authors acknowledge the case for an optional anonymous phase. In practice, the retaliation risks documented since 2012, including legal threats against reviewers and post-publication critics, have validated the need for protected anonymity as a complement to transparency, not a contradiction of it.
What the paper could not have anticipated is the role of immutable, decentralized infrastructure in solving several of the implementation challenges it raises. Permanence of reviews, resistance to editorial suppression, and tamper-proof timestamping are all achievable now in ways they were not in 2012.

This remains one of the most cited and most relevant papers on the future of scientific evaluation. Its vision has aged remarkably well. The question is no longer whether open evaluation is desirable - this paper settled that - but whether anyone will build and adopt the systems to make it real.

This text was written with ClaudeAI.