Speaker
Description
Introduction
The scientific community faces a critical challenge: essential contributions to research progress—including data sharing, code development, peer review, mentorship, and community engagement—remain systematically undervalued. Despite the Open Science movement’s efforts to broaden recognition beyond traditional publications, engagement with alternative evaluation systems remains persistently low. Researchers continue to prioritize activities that advance their careers within traditional reward structures, creating a paradox in which systems designed to improve evaluation cannot attract enough evaluators to function effectively.
This workshop introduces Metavaluation, a novel participatory framework that directly addresses this engagement challenge by treating evaluations themselves as valuable contributions. Through an innovative self-referential system—where evaluations are also evaluated—participants are incentivized to provide high-quality assessments, creating a virtuous cycle of participation. Unlike existing systems that struggle with low uptake, Metavaluation includes direct rewards for evaluators, making the process engaging, equitable, and sustainable.
Significance and Relevance to SciDataCon Themes
Metavaluation’s six-stage protocol aligns directly with several of SciDataCon 2025’s key themes:
-
Record – Open Research Through Interoperable Data. Contributions and evaluation dimensions are transparently recorded, forming an open registry that enables interoperable value metrics across research communities.
-
Review – Empowering Global Participation. Evaluations are conducted through simple pairwise comparisons, allowing anyone—regardless of expertise level—to meaningfully contribute. This reduces gatekeeping and fosters inclusive global participation.
-
Recognize – Supporting Rigorous, Responsible Science. A meta-evaluation process generates standardized, reproducible metrics for diverse contribution types, improving transparency and scientific integrity.
-
Reward – Aligning Incentives Toward Openness. Tokenized or symbolic rewards for both contributions and evaluations incentivize sustained participation, aligning individual and collective goals.
-
Respect – Building Responsible AI and Governance. The system tracks reliability over time, enabling merit-based governance models and providing value-aligned training data for future AI tools.
-
Research – Advancing Reproducible Meta-Research. Metavaluation supports continuous learning by documenting and analyzing the valuation process itself, contributing to open, reproducible meta-science.
Workshop Format and Structure
This 90-minute workshop will combine a concise overview of the Metavaluation model with a live, hands-on session where participants apply it in real time to value contributions to SciDataCon 2025. The session will be structured as follows:
1. Introduction and Framing (15 min)
Overview of the collective action problem in research evaluation and core blockers to cultural change. Introduction to Metavaluation’s six-stage protocol and its role as a participatory incentive layer for research communities. Overview of prototype data collected from diverse communities spanning the arts, sciences, and technology sectors, along with key learnings from three years of prototyping radical participatory governance models.
2. Live Demonstration (10 min)
Walkthrough of the newly released, open-source Metavaluation app. Demonstration of how to nominate contributions, conduct evaluations, and view output metrics.
3. Interactive Collective Valuation (40 min)
Participants will:
- Nominate real contributions from the conference (e.g. talks, organizing, registration fees).
- Evaluate them via pairwise comparisons across multiple dimensions (e.g. Gratitude, Value to the Community and Mission).
- Review system-generated metrics showing relative value rankings.
4. Discussion and Synthesis (20 min)
Group discussion on:
- What values emerged from the experiment
- Participant experience of the evaluation process
- Use cases in other communities (e.g. data stewardship, governance)
- Feedback and ideas for broader adoption
5. Wrap-Up and Invitation (5 min)
Invitation to continue using the system beyond the workshop. Encouragement to co-develop value taxonomies, nominate more contributions, and contribute to a growing commons of interoperable, community-generated data.
No technical background is required to attend this workshop — just your perspective, values, and voice.
