Assemble Galaxy workflow test fixtures and assertions.
/plugin marketplace add jmchilton/foundry
/plugin install foundry-skills@galaxy-workflow-foundryThen invoke as:
/foundry-skills:implement-galaxy-workflow-testProduces: 1 artifact.
Consumes: 3 artifacts.
Galaxy workflow test file (tests-format) with job inputs, expected outputs, assertions; passes static schema + label cross-check.
{
"id": "galaxy-workflow-test",
"kind": "yaml",
"default_filename": "galaxy-workflow-tests.yml",
"description": "Galaxy workflow test file (tests-format) with job inputs, expected outputs, assertions; passes static schema + label cross-check."
}Reviewable Galaxy test plan from a *-test-to-galaxy-test-plan Mold; profile, fixtures, snapshot/assertion provenance.
{
"id": "galaxy-test-plan",
"description": "Reviewable Galaxy test plan from a *-test-to-galaxy-test-plan Mold; profile, fixtures, snapshot/assertion provenance.",
"producers": [
"cwl-test-to-galaxy-test-plan",
"nextflow-test-to-galaxy-test-plan"
]
}gxformat2 workflow being tested; provides labels, outputs, and shapes the test must assert against.
{
"id": "galaxy-workflow-draft",
"description": "gxformat2 workflow being tested; provides labels, outputs, and shapes the test must assert against.",
"producers": [
"cwl-summary-to-galaxy-template",
"implement-galaxy-tool-step",
"nextflow-summary-to-galaxy-template",
"paper-summary-to-galaxy-template"
]
}Resolved test data references (URLs, paths, expected shapes) from paper-to-test-data or find-test-data.
{
"id": "test-data-refs",
"description": "Resolved test data references (URLs, paths, expected shapes) from paper-to-test-data or find-test-data.",
"producers": [
"find-test-data",
"paper-to-test-data"
]
}JSON Schema contract for the Galaxy workflow test format. Output of this Mold must validate against it.
references/schemas/tests-format.schema.jsonpackage://@galaxy-foundry/tests-format-schema#testsFormatSchema{
"$defs": {
"Collection": {
"additionalProperties": false,
"properties": {
"class": {
"const": "Collection",
"title": "Class",
"type": "string"
},
"collection_type": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Collection Type"
},
"elements": {
"anyOf": [
{
"items": {
"oneOf": [
{
"oneOf": [
{
"$ref": "#/$defs/LocationFile"
},
{
"$ref": "#/$defs/PathFile"
},
{
"$ref": "#/$defs/ContentsFile"
},
{
"$ref": "#/$defs/CompositeDataFile"
}
]
},
{
"$ref": "#/$defs/Collection"
}
]
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"title": "Elements"
},
"identifier": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Identifier"
},
"name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Name"
},
"rows": {
"anyOf": [
{
"additionalProperties": {
"items": {},
"type": "array"
},
"type": "object"
},
{
"type": "null"
}
],
"default": null,
"title": "Rows"
}
},
"required": [
"class"
],
"title": "Collection",
"type": "object"
},
"CollectionAttributes": {
"additionalProperties": false,
...Run the cheap static workflow-test validation and workflow-label cross-check before Planemo execution.
Trigger: After authoring or editing a Galaxy workflow test file and before Planemo invocation.
references/cli/validate-tests.jsoncontent/cli/gxwf/validate-tests.md{
"type": "cli-command",
"tool": "gxwf",
"command": "validate-tests",
"summary": "Validate Galaxy workflow test files and optionally cross-check labels against their workflow.",
"source_path": "content/cli/gxwf/validate-tests.md",
"source_revision": 3,
"body": "# `gxwf validate-tests`\n\nValidate a Galaxy workflow test file (`*-tests.yml` or `*.gxwf-tests.yml`) against the Galaxy tests schema. Optionally cross-check the test file against a workflow so missing input labels, missing output labels, and type mismatches fail before a slow Planemo run.\n\n`<file>` is a workflow test YAML file, usually named `<workflow>-tests.yml` or `<workflow>.gxwf-tests.yml`.\n\n## Output\n\nDefault output is human-readable validation diagnostics.\n\nWith `--json`, the command emits a structured report describing schema errors and, when `--workflow` is supplied, workflow-coherence errors such as missing labels or incompatible input values.\n\n## Examples\n\n```bash\ngxwf validate-tests workflow-tests.yml\ngxwf validate-tests workflow-tests.yml --json\ngxwf validate-tests workflow-tests.yml --workflow workflow.gxwf.yml --json\n```\n\n## Gotchas\n\n- This is the cheap static gate before Planemo. It does not execute the workflow and does not prove assertions pass on real outputs.\n- Use `--workflow` whenever the workflow file is available. Schema-valid tests can still reference stale input/output labels after workflow edits.\n- Run this before `planemo workflow_test_on_invocation` or full `planemo test`; it catches authoring mistakes without starting Galaxy.\n- The assertion vocabulary itself comes from [[tests-format]]; strategy for choosing assertions lives in [[planemo-asserts-idioms]]."
}
Revise workflow inputs, outputs, labels, checkpoints, and collection identifiers so meaningful tests can be authored.
Trigger: When test authoring reveals missing labels, omitted workflow-level outputs, unstable collection identifiers, weakly assertable final outputs, or fixture-shape pressure on workflow inputs.
references/notes/galaxy-workflow-testability-design.mdcontent/research/galaxy-workflow-testability-design.md---
type: research
subtype: component
tags:
- research/component
- target/galaxy
status: draft
created: 2026-05-03
revised: 2026-05-06
revision: 2
ai_generated: true
related_notes:
- "[[iwc-workflow-testability-survey]]"
- "[[iwc-test-data-conventions]]"
- "[[planemo-asserts-idioms]]"
- "[[iwc-shortcuts-anti-patterns]]"
- "[[planemo-workflow-test-architecture]]"
- "[[implement-galaxy-workflow-test]]"
- "[[gxformat2-schema]]"
- "[[gxformat2-workflow-inputs]]"
- "[[galaxy-datatypes-conf]]"
summary: "Design guidance for Galaxy workflow inputs, outputs, and checkpoints that make IWC-style workflow tests possible."
---
# Galaxy workflow testability design
Use this note when authoring or translating a Galaxy workflow **before** the `-tests.yml` file exists. It covers workflow structure choices that make later IWC-style tests meaningful: labels, promoted checkpoints, collection identifiers, and fixture-compatible inputs.
This is not a `content/patterns/` page. It is cross-cutting design guidance for Molds that need testable Galaxy workflows. Assertion syntax lives in [[planemo-asserts-idioms]]. Test YAML fixture shapes live in [[iwc-test-data-conventions]]. Accepted shortcut vs smell calls live in [[iwc-shortcuts-anti-patterns]]. Corpus evidence trail lives in [[iwc-workflow-testability-survey]].
## 1. Treat labels as API
Workflow input and output labels are not cosmetic. Planemo and IWC tests address workflow inputs and outputs by label, and the survey found exact label matches for every asserted output across 114 matched workflow/test pairs. A generated workflow should therefore pick stable, descriptive labels before test authoring starts.
Rules:
- Label every output that may need a test assertion.
- Treat input/output renames as breaking changes
...Flag assertion shortcuts that are acceptable in IWC versus shortcuts that should be avoided.
Trigger: When considering existence-only, size-only, image-only, checksum, output-label, or negative-test patterns.
references/notes/iwc-shortcuts-anti-patterns.mdcontent/research/iwc-shortcuts-anti-patterns.md---
type: research
subtype: component
tags:
- research/component
- target/galaxy
status: draft
created: 2026-04-30
revised: 2026-05-03
revision: 2
ai_generated: true
related_notes:
- "[[galaxy-workflow-testability-design]]"
- "[[iwc-test-data-conventions]]"
- "[[planemo-asserts-idioms]]"
- "[[implement-galaxy-workflow-test]]"
- "[[tests-format]]"
- "[[iwc-conditionals-survey]]"
- "[[iwc-map-over-lifecycle-survey]]"
- "[[iwc-tabular-operations-survey]]"
- "[[iwc-transformations-survey]]"
summary: "What IWC test suites cut corners on (accepted) vs what's a code smell — existence-only probes, sim_size deltas, image dim checks, label coupling."
---
# IWC test-suite shortcuts and anti-patterns
## Purpose
When an agent translates or authors a Galaxy workflow for IWC submission, the test suite it writes will be reviewed against IWC's *de facto* style — not against an idealized assertion ladder. That style routinely tolerates assertions that look weak in isolation. This note distinguishes the corner-cutting that is **normal and accepted** in the corpus from the patterns that an agent should treat as **smells** worth flagging.
This note owns accepted-vs-smell calls. For positive workflow-structure guidance behind label stability, checkpoint promotion, and collection identifier design, use [[galaxy-workflow-testability-design]].
Grounding: 115 `*-tests.yml` files under `workflow-fixtures/iwc-src/workflows/` (mirror of `galaxyproject/iwc`), prior synthesis in `galaxy-brain/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md`. Path citations below are relative to `iwc-src/workflows/` unless absolute.
## TL;DR rules of thumb
1. **Default to tolerant assertions.** `compare: sim_size` + `delta:`, `has_image_*` + `delta:`, `has_text` s
...Assemble job input fixtures, remote URLs, hashes, collection shapes, and test-data layout in IWC style.
Trigger: When writing or revising the job/input side of a Galaxy workflow test file.
references/notes/iwc-test-data-conventions.mdcontent/research/iwc-test-data-conventions.md---
type: research
subtype: component
tags:
- research/component
- target/galaxy
status: draft
created: 2026-04-30
revised: 2026-05-03
revision: 3
ai_generated: true
related_notes:
- "[[galaxy-workflow-testability-design]]"
- "[[iwc-shortcuts-anti-patterns]]"
- "[[planemo-asserts-idioms]]"
- "[[implement-galaxy-workflow-test]]"
- "[[tests-format]]"
- "[[iwc-tabular-operations-survey]]"
summary: "How IWC workflows organize and reference test data — Zenodo-first, SHA-1 integrity, collection shapes, CVMFS gotchas."
---
# IWC test data conventions
Reference for an agent implementing or editing a `<workflow>-tests.yml` in IWC style. All evidence cited from `/Users/jxc755/projects/repositories/workflow-fixtures/iwc-src/workflows/` (raw IWC clone) and `workflows/README.md`. Authoritative spec: [planemo.readthedocs.io/en/latest/test_format.html](https://planemo.readthedocs.io/en/latest/test_format.html). The companion analysis at `/Users/jxc755/projects/repositories/galaxy-brain/vault/projects/workflow_state/skills/COMPONENT_GALAXY_WORKFLOW_TESTING.md` is the synthesized source for several normative claims here.
This note owns **test YAML fixture shapes**. For workflow-structure choices that make those fixtures possible before the test file exists, use [[galaxy-workflow-testability-design]].
## 1. Where does test data live? Remote vs in-repo
Two storage patterns. They mix freely inside one job.
**Remote `location:` (default for any non-trivial input).** Strongly preferred for anything bigger than a toy fixture. Order of preference, observed in the corpus:
- **Zenodo** — overwhelming default, persistent DOI-backed URL.
- `read-preprocessing/short-read-qc-trimming/short-read-quality-control-and-trimming-tests.yml:13,17` — `https://zenodo.org/records/11484
...Choose assertion families, tolerance magnitudes, and the static/Planemo validation loop.
Trigger: When writing or revising output assertions for a Galaxy workflow test file.
references/notes/planemo-asserts-idioms.mdcontent/research/planemo-asserts-idioms.md---
type: research
subtype: component
tags:
- research/component
- target/galaxy
status: draft
created: 2026-04-30
revised: 2026-05-03
revision: 5
ai_generated: true
related_notes:
- "[[galaxy-workflow-testability-design]]"
- "[[iwc-test-data-conventions]]"
- "[[iwc-shortcuts-anti-patterns]]"
- "[[implement-galaxy-workflow-test]]"
- "[[tests-format]]"
- "[[planemo-workflow-test-architecture]]"
- "[[validate-tests]]"
- "[[iwc-tabular-operations-survey]]"
- "[[galaxy-discover-datasets]]"
summary: "Decision and idiom guide for picking planemo workflow-test assertions: which family per output type, how to size tolerances, when to validate."
---
# Planemo asserts: idiom and decision guide
Companion to [[iwc-test-data-conventions]] (input shapes), [[galaxy-workflow-testability-design]] (workflow structure before test YAML exists), and [[iwc-shortcuts-anti-patterns]] (what's accepted vs smell). This note is forward-looking: when authoring a new `<workflow>-tests.yml`, which assertion family fits which output, and what the recommended tolerances and operators are.
The **vocabulary itself is not restated here** — every assertion's parameter list, types, defaults, required fields, and Python docstring is rendered from the test-format JSON Schema at [[tests-format]]. Assertion names below deep-link into that page (e.g. [[tests-format#has_text_model|has_text]] jumps straight to that `$def`).
## 1. Choose by output type
The single most useful decision table. Pick the row that matches the file format the workflow emits; default to the recommended assertion family.
| Output type | Default assertion family | Why | Fallback |
|---|---|---|---|
| **Plain text reports / logs** (FastQC summary, MultiQC text section) | [[tests-format#has_text_model|has_text]] (su
...Write tests with stable labels and artifacts that Planemo can connect back to Galaxy invocations, jobs, and outputs.
Trigger: When adding or revising workflow tests that will be iterated with Planemo or generated from existing invocations.
references/notes/planemo-workflow-test-architecture.mdcontent/research/planemo-workflow-test-architecture.md---
type: research
subtype: component
title: "Planemo workflow-test architecture"
tags:
- research/component
- tool/planemo
- target/galaxy
status: draft
created: 2026-05-02
revised: 2026-05-03
revision: 2
ai_generated: true
related_notes:
- "[[galaxy-workflow-testability-design]]"
- "[[galaxy-tool-job-failure-reference]]"
- "[[galaxy-workflow-invocation-failure-reference]]"
- "[[planemo-asserts-idioms]]"
related_molds:
- "[[run-workflow-test]]"
- "[[debug-galaxy-workflow-output]]"
- "[[implement-galaxy-workflow-test]]"
sources:
- "~/projects/repositories/planemo/planemo/commands/cmd_test.py"
- "~/projects/repositories/planemo/planemo/commands/cmd_run.py"
- "~/projects/repositories/planemo/planemo/galaxy/activity.py"
- "~/projects/repositories/planemo/planemo/galaxy/invocations"
- "~/projects/repositories/planemo/planemo/galaxy/config.py"
summary: "Reference for Planemo workflow test/run architecture, Galaxy modes, API polling, and noisy failure boundaries."
---
# Planemo Workflow-Test Architecture
This note describes Planemo architecture relevant to workflow tests and workflow runs. It is reference material for Molds that need to run tests or interpret Planemo artifacts, not a command-selection recipe.
## Main Commands
| User action | Command | Core behavior |
|---|---|---|
| Full workflow test | `planemo test <workflow>` | Finds test definitions, starts or targets Galaxy, stages inputs, invokes workflow, checks assertions, writes reports. |
| Direct run | `planemo run <workflow> <job.yml>` | Runs one workflow/job pair and can download outputs without assertion checks. |
| Recheck assertions | `planemo workflow_test_on_invocation <tests.yml> <invocation_id>` | Runs test assertions against an existing invocation without rerunning the wo
...
# implement-galaxy-workflow-test
Follow the procedure below and use the artifact/reference sections as the runtime contract.
## When To Use
- Assemble Galaxy workflow test fixtures and assertions.
## Inputs
- Read artifact `galaxy-test-plan`. Produced by `cwl-test-to-galaxy-test-plan`, `nextflow-test-to-galaxy-test-plan`. Reviewable Galaxy test plan from a *-test-to-galaxy-test-plan Mold; profile, fixtures, snapshot/assertion provenance.
- Read artifact `galaxy-workflow-draft`. Produced by `cwl-summary-to-galaxy-template`, `implement-galaxy-tool-step`, `nextflow-summary-to-galaxy-template`, `paper-summary-to-galaxy-template`. gxformat2 workflow being tested; provides labels, outputs, and shapes the test must assert against.
- Read artifact `test-data-refs`. Produced by `find-test-data`, `paper-to-test-data`. Resolved test data references (URLs, paths, expected shapes) from paper-to-test-data or find-test-data.
## Outputs
- Write artifact `galaxy-workflow-test` as `galaxy-workflow-tests.yml`. Format: `yaml`. Galaxy workflow test file (tests-format) with job inputs, expected outputs, assertions; passes static schema + label cross-check.
## Required Tools
- **`gxwf`** (gxwf). `npm install -g @galaxy-tool-util/cli`.
Ephemeral run: `npx --package @galaxy-tool-util/cli gxwf`.
Check: `gxwf --version`.
Docs: https://github.com/jmchilton/galaxy-tool-util-ts/tree/main/packages/cli
## Load Upfront
- `references/schemas/tests-format.schema.json`: Schema file copied verbatim into the bundle. JSON Schema contract for the Galaxy workflow test format. Output of this Mold must validate against it.
## Load On Demand
- `references/cli/validate-tests.json`: CLI command reference packaged as a sidecar. Run the cheap static workflow-test validation and workflow-label cross-check before Planemo execution. Use when: after authoring or editing a Galaxy workflow test file and before Planemo invocation.
- `references/notes/galaxy-workflow-testability-design.md`: Research note copied verbatim into the bundle. Revise workflow inputs, outputs, labels, checkpoints, and collection identifiers so meaningful tests can be authored. Use when: test authoring reveals missing labels, omitted workflow-level outputs, unstable collection identifiers, weakly assertable final outputs, or fixture-shape pressure on workflow inputs.
- `references/notes/iwc-shortcuts-anti-patterns.md`: Research note copied verbatim into the bundle. Flag assertion shortcuts that are acceptable in IWC versus shortcuts that should be avoided. Use when: considering existence-only, size-only, image-only, checksum, output-label, or negative-test patterns.
- `references/notes/iwc-test-data-conventions.md`: Research note copied verbatim into the bundle. Assemble job input fixtures, remote URLs, hashes, collection shapes, and test-data layout in IWC style. Use when: writing or revising the job/input side of a Galaxy workflow test file.
- `references/notes/planemo-asserts-idioms.md`: Research note copied verbatim into the bundle. Choose assertion families, tolerance magnitudes, and the static/Planemo validation loop. Use when: writing or revising output assertions for a Galaxy workflow test file.
- `references/notes/planemo-workflow-test-architecture.md`: Research note copied verbatim into the bundle. Write tests with stable labels and artifacts that Planemo can connect back to Galaxy invocations, jobs, and outputs. Use when: adding or revising workflow tests that will be iterated with Planemo or generated from existing invocations.
## Validation
- None declared.
## Procedure
Stub. Replace with real skill content per MOLD_SPEC once first walks are done.
- **`planemo workflow_test_init --from_invocation <id>`** — preferred bootstrap for new test files; reviewer convention. See planemo-asserts-idioms §7.
- **`planemo workflow_test_on_invocation <tests.yml> <id>`** — fast assertion-iteration loop without re-running the workflow.
## Runtime Notes
- Do not read Foundry source files at runtime; use only files packaged in this skill bundle and user-supplied artifacts.
- Preserve declared artifact filenames unless the user or harness supplies explicit paths.
- Carry unresolved assumptions into the output artifact instead of silently inventing missing source evidence.