Speaker
Description
Systemic reform in science continues to face a collective action problem: researchers agree that contributions such as data sharing, peer review, software development, and community engagement are essential, yet these remain structurally undervalued in current incentive systems. Although the Open Science movement has promoted greater transparency and expanded recognition, uptake of alternative evaluation tools remains low. Most systems rely on unpaid labor and lack meaningful incentives, creating a paradox where improved evaluation tools fail to attract enough evaluators to function effectively.
This talk introduces Metavaluation—an open-source, participatory evaluation protocol designed to overcome this barrier by embedding incentives directly into the evaluation process. Rather than treating peer review and contribution recognition as separate, unlinked processes, Metavaluation treats evaluations themselves as contributions, subject to the same collective valuation process. By feeding pairwise evaluations into a recursive feedback loop—where evaluations are used to value other evaluations—the system generates standardized, reproducible value metrics across multiple dimensions (e.g. Gratitude, Value to the Community and Mission). These metrics support rigorous, transparent, and inclusive science by informing recognition, funding, governance, and strategic planning.
Unlike conventional reputation systems, Metavaluation is designed from first principles to be open, inclusive, interoperable, and resistant to gaming. It avoids absolute scoring and popularity biases by relying on pairwise comparisons—a simple, accessible format proven to generate high-quality judgments with minimal cognitive load. It also controls for exposure and attention biases through random sampling. The key innovation is treating these comparisons as a “base unit” of community value, then embedding them within the same evaluative process. This allows all other contributions to be scaled in relative terms, making every value score comparable to the value of a single pairwise comparison. The result is a participatory, decentralized incentive mechanism that mitigates gatekeeping while encouraging broad engagement.
Since its inception, the framework has been prototyped and validated across multiple domains—academic conferences, community arts festivals, and decentralized projects—demonstrating its flexibility in valuing diverse contributions. These real-world experiments have shown how communities can use Metavaluation to make the invisible visible, recognize essential work, and begin aligning incentives with shared values. Its use of community-defined metrics and reward structures offers promising applications for open infrastructure, responsible AI, and sustainable funding models.
This presentation will:
• Share the conceptual foundations and system design of Metavaluation.
• Present results from recent prototypes in academic and grassroots settings.
• Reflect on learnings related to participation, data quality, and interoperability.
• Explore implications for open research infrastructures, value-aligned funding, and future data governance systems.
The session is especially relevant to attendees interested in reproducible science, decentralized governance, data stewardship, equity and inclusion, and cross-community infrastructure. It will complement a related workshop that introduces the hands-on use of the Metavaluation app for valuing contributions to the SciDataCon conference itself.
By centering reward and recognition on what communities truly value—rather than what is easiest to count—Metavaluation offers a scalable, interoperable, and fair approach to evaluation. It enables research communities to coordinate through shared metrics, reward engagement, and align practices with values—laying the groundwork for a more inclusive, resilient, and participatory data ecosystem.
