Statistical Genomics Foundry
Design docs design record

StatReviewer + SciScore

Deployed commercial automated manuscript referees — post-hoc, closed, reporting-level; their framing, strengths, and where they sit on our values.

statreviewer.com · sciscore.com — StatReviewer: "automated review of statistical and reporting integrity for scientific manuscripts"; SciScore: "the best methods review tool for scientific research." Two deployed, commercial, automated manuscript referees that run inside journal editorial pipelines. StatReviewer runs "thousands of algorithmic tests" over a submitted manuscript — numerical-consistency errors, quality of reporting, whether the reported test fits the data (e.g. "t-test for skewed data"), style, methodological reporting — and emits an editor/author checklist inside Editorial Manager. SciScore scans a methods section with NLP (published as conditional-random-field models plus regex named-entity recognition) against public rigor checklists — MDAR, ARRIVE, CONSORT, NIH rigor, RRIDs — and returns three reports plus a 1–10 score; journal-year averages form its Rigor and Transparency Index. Both are closed/commercial SaaS; SciScore lists Science, bioRxiv/medRxiv, AHA Journals, Charité, and CZI among clients. Verified 2026-06-27 against sciscore.com + the asedasciences "about" page, the SSP/Aries Editorial-Manager integration announcement, the BMC "peerless review" blog, and the Rigor-and-Transparency-Index papers (iScience / PMC). StatReviewer's own domain did not resolve at verification time; its claims are grounded in the integration and press sources.

Its framing

Both present as automated pre-review referees that augment, not replace, human peer review — explicitly analogized to how plagiarism software (CrossCheck) sits ahead of review. StatReviewer's stated ambition is not to judge the quality of the science but to check the reporting: it flags obvious numerical errors, missing replication detail, and mismatches between the data and the reported statistical test, then hands editors a checklist and author-ready feedback. SciScore's stated mission is "enhancing scientific rigor and transparency": it reads the materials-and-methods section across ten dimensions (ethics, group selection, demographics, reporting standards, replication, cell-line authentication, protocol/RRID identifiers, code, data, euthanasia) and scores how completely the manuscript states the rigor details the community checklists ask for. Both run in roughly a minute, at manuscript scale, integrated into real submission systems (Editorial Manager, eJournalPress).

Strengths relative to our approach

These are the closest thing to a deployed, commercial automated statistical referee — and on deployment they are far ahead of the Foundry:

  • Real market deployment at scale. Integrated into the editorial systems most journals actually run (Editorial Manager, eJournalPress), processing live submissions today, with named publisher and institutional clients. The Foundry is a design workspace; this pair is in production. The very fact that journals pay for an automated statistical/reporting referee is evidence the referee role has genuine demand — a point that supports our thesis.
  • The referee role, in spirit, done for real. StatReviewer goes beyond checklist presence: testing whether the reported method fits the data (a -test on skewed data) is a genuine assumption-style check, not just "is a field filled in." That is the closest external gesture toward refereeing method appropriateness rather than reporting completeness.
  • Public, community-owned rubrics. SciScore's checklists — MDAR, ARRIVE, CONSORT, NIH rigor — are open, human-authored, widely adopted standards. The judgment criteria (what a rigorous methods section must state) are inspectable even though the engine that applies them is not.
  • Published methodology and a derived metric. SciScore's approach (CRF + regex NER) and its Rigor-and-Transparency-Index validation are described in peer-reviewed papers — more methodological openness than a pure black box, and a real attempt to make the output measurable and comparable across journals.

Where it sits on our values

Value StatReviewer + SciScore
Produces skills No — both are hosted services, not portable skill artifacts. They emit reports and scores about a manuscript, not reusable units an agent installs and runs.
Progressive disclosure No — closed SaaS. SciScore layers its output (three reports + a score) and StatReviewer a checklist, but there is no progressive-disclosure surface from an authoritative source down to detail; the layering is report formatting, not a navigable knowledge stack.
Traceability Partial — SciScore grounds each judgment in a named public checklist item (MDAR/ARRIVE/CONSORT/NIH), so a human can see which community standard a flag derives from; StatReviewer cites the reporting/statistical issue it found. But the engine internals — which of the "thousands of tests" fired and the rule logic behind them — are closed. Judgments trace to a public standard, not to an inspectable rule.
CLI instructions derived from CLI docs N/A — neither is a CLI-tool skill system; there are no per-tool invocation instructions to derive.
Portability No skills produced — nothing to port; capability lives only behind the vendors' hosted services and editorial-system integrations.
Human scrutiny Partial, one level up — the checklists are human-authored, public, and community-correctable (MDAR/ARRIVE/CONSORT evolve in the open). The engine that applies them — StatReviewer's test battery, SciScore's CRF/regex models — is proprietary, not readable or contributable by users. You can argue with the standard; you cannot inspect or correct the referee.
Knowledge-base backed No (with a real caveat) — there is no inspectable knowledge base a human reads beneath the service. The caveat is genuine: SciScore's checklists are public standards and its method is published, so the criteria rest on an open, documented body of practice even though the applying engine is closed.

Where it diverges from the Foundry

Running automated statistical referees at scale is a real similarity in spirit, and this pair proves the role has paying demand — credit fully given. Three concrete axes separate them from the Foundry. Unit: they referee a written manuscript's reporting — is the right detail stated, is the reported test consistent with the reported data — whereas the Foundry's referee judges live method validity: assumptions actually met, no double-dipping, the named method real and appropriate, error rate empirically controlled (null/permutation calibration, simulation under known truth, power). StatReviewer's "t-test on skewed data" check is the closest gesture, but it infers from reported text, not from the data and the analysis as run. Timing: they run post-hoc on a finished paper at submission; the Foundry's gate runs in-loop, on an arbitrary live analysis, before a result is ever certified. Openness: they are closed, hosted, commercial engines; the Foundry's referee is an inspectable skill cast from a readable source, that partly constructs its own empirical check. The crux: they check whether a paper said the right things about an analysis already done; the Foundry checks whether the analysis is valid while it is still being done.

Sources