Cross-Project Workflow Linting: Testing Alignment Plan

Workflow linting exists in three places that evolved independently and need to be reconciled — not at the implementation level, but at the level of which rules exist and how they are tested. This document is the living plan + the rule inventory.

Projects in Scope

Note: the Galaxy Python wf_tool_state backend is not a separate linter — it wraps gxformat2’s lint and adds stateful tool-state validation. It is listed here because its CLI reports surface lint results and because it’s the source of truth for the lint_stateful fixtures under packages/schema/test/fixtures/workflow-state/.

Premise Check (Why Linting ≠ Validation ≠ Shared Impl)

• gxformat2 linting is older than VS Code linting. The lint.py / linting.py module predates the LSP server by years, so the two sets of rules evolved on separate tracks. There is no expectation that either project is a superset of the other.
• VS Code cannot consume gxformat2’s TS port directly. The galaxy-tool-util TS lintFormat2() returns {errors: string[], warnings: string[]} — a flat bag with no source positions. VS Code ValidationRules must emit Diagnostic objects with range: {start, end} pointing at specific nodes (see InputTypeValidationRule.ts:32-36). Sharing logic requires per-rule functions that can be wrapped by an AST walker; sharing string outputs isn’t workable.
• There is one precedent for shared primitives: VS Code’s ToolStateValidationService already calls validateFormat2StepStateStrict() from @galaxy-tool-util/schema and then remaps the errors onto source ranges itself. That’s the pattern — galaxy-tool-util supplies the check, VS Code supplies the positioning. This plan is not about lift-and-shift; it’s about aligning the rule inventory and the test fixtures.
• VS Code has two validation profiles (basic, iwc), selected via galaxyWorkflows.validation.profile. gxformat2 has an analogous split via --skip-best-practices. The table below tracks profile membership explicitly.

Plan Steps

Step 1 — Rule inventory table (this document)

Enumerate every lint operation anywhere in the three projects. For each: description, severity, implementation status per project, VS Code profile membership, and pointers to positive/negative declarative fixtures. Deliverable is the tables in the next section — see Rule Inventory below. Step 1 will not be fully filled in on first pass; the TODO cells are tracking debt that Step 3 iterates on.

Step 2 — Share declarative fixtures

Pick one of two approaches (open question):

• Option A — Publish fixtures from galaxy-tool-util: Ship packages/schema/test/fixtures/{workflows,expectations}/ as a consumable artifact (e.g. a small @galaxy-tool-util/workflow-fixtures package or a tarball attached to releases). VS Code server/*/tests/ adds a dev dep on it and runs a new declarative test harness modeled on packages/schema/test/declarative-normalized.test.ts.
• Option B — Migrate the sync targets into VS Code: Duplicate the sync-workflow-fixtures / sync-workflow-expectations Makefile targets into galaxy-workflows-vscode and keep three consumers of gxformat2’s examples. More divergence risk; rejected unless A proves infeasible.

Either way, VS Code needs a new test file that loads expectation YAMLs and asserts that (a) baseline validation + (b) each profile’s rules produce the expected diagnostics. The assertion shape differs from the TS side because diagnostics include ranges, so the YAML format may need a diagnostics: [{message_contains, range_hint}] extension.

Step 3 — Fill in the gaps

Document the architecture (this file) and iterate: for each rule in the table where any of the “Implemented” columns is no or any fixture column is —, open a targeted task. Most are either “add a missing rule to the weaker project” or “add an isolated fixture for a rule that currently only has a composite fixture”.

Rule Inventory

Legend for “Implemented” columns:

• Y — has a dedicated implementation
• baseline — runs automatically as part of schema/syntax validation (not a pluggable rule)
• — — not implemented
• ind — indirect (e.g. galaxy-python via gxformat2)

VS Code column format: basic|iwc per format (f2 = format2, ga = native) — e.g. iwc(f2+ga) means IWC profile, both formats.

A. Structural Lint (core correctness)

“Schema?” column: YES = the rule is fully redundant with JSON-schema/Effect-decode validation (candidate for deletion as a separate lint check); partial = schema enforces presence/type but not the specific value or cross-reference; NO = schema cannot express this check.

B. Best-Practices Lint (style / IWC)

Almost everything in this table is NO for “Schema?” — best-practice rules check for absence of optional fields, which schemas can’t express.

C. Stateful Lint (tool-state against cached tool defs)

Only present in galaxy-python lint_stateful.py and in galaxy-workflows-vscode ToolStateValidationService. galaxy-tool-util is the primitive both call — validateFormat2StepStateStrict. gxformat2 does not do stateful validation.

D. Baseline Syntax / Schema (always runs in VS Code)

These are not pluggable rules in VS Code but appear here because they overlap with explicit lint checks in gxformat2.

Pruning Candidates (Schema-Redundant Rules)

A rule that’s Schema? = YES is doing the same job as the baseline schema decode pass. The schema pass always runs (in gxformat2 via lint_pydantic_validation, in the TS port via Effect decode, in VS Code as the "Format2 Schema" / "Native Workflow Schema" baseline diagnostic, in galaxy-python via gxformat2). Keeping a hand-rolled lint rule for the same condition just means: (a) two error messages for the same input, (b) drift risk when the schema gets stricter, and (c) extra cells to maintain in this table.

Recommended drops:

A4 / A6 update (April 2026): Pydantic schemas in gxformat2 (schema/native_v0_1/workflow.yml) now use pydantic:type: "Literal['true']" and pydantic:type: "Literal['0.1']". Both gxformat2/schema/native.py and native_strict.py were regenerated. The single-quote form is deliberate: schema-salad-plus-pydantic’s codegen auto-injects a default= for any field whose type annotation matches the regex ^Literal\["(.+)"\]$ — i.e. double-quoted Literals only — and the auto-default would mask missing-field errors (A3/A5). Single-quoted Literal['true'] is the same Python type but doesn’t trip the regex, so no default is added and pydantic still rejects missing fields. The TS Effect-schema codegen needs the same treatment when this gets resynced (run make generate-schemas after make sync-schema-sources).

Recommended keep (still NOT schema-redundant):

• A7 (step key not integer) — depends on whether the native steps map is typed. Verify.
• A8 (subworkflow has empty steps) — schema enforces presence but not non-emptiness. Could be fixed with a Schema.NonEmptyArray/minProperties constraint; until then, keep.

Everything in Table B stays — best-practice rules check absence-of-optional, which schemas can’t express.

The cleanup pattern this implies:

1. Tighten the schemas (A4, A6, A8) so more rules become YES rather than partial.
2. Drop the redundant rules from lint.py and lint.ts once their schema replacement is in place.
3. The rule inventory shrinks; the test fixtures still exercise the schema baseline (the operation key in the YAML just changes from lint_format2 to validate_format2).

This is a behavior change for users who run gxwf lint --skip-best-practices and expect structural-only errors. After pruning, those errors come from the schema decode pass instead — same content, possibly different message text. Worth flagging in a changeset.

Test Harness Shape per Project

Architecture for Shared Fixtures (Step 2 target)

                            ┌──────────────────────────────┐
                            │        gxformat2             │
                            │  examples/ + expectations/   │  (source of truth)
                            └──────────────┬───────────────┘
                                           │ make sync-workflow-{fixtures,expectations}
                                           ▼
                    ┌────────────────────────────────────────┐
                    │   galaxy-tool-util/packages/schema     │
                    │   test/fixtures/{workflows,expectations}│  ← already wired
                    └──────────────┬─────────────────────────┘
                                   │   Step 2: publish as npm artifact
                                   │   (@galaxy-tool-util/workflow-fixtures)
                                   │   OR per-release tarball
                                   ▼
        ┌──────────────────────────┴───────────────────────────┐
        ▼                                                       ▼
┌──────────────────┐                        ┌────────────────────────────┐
│ galaxy-workflows │                        │   galaxy wf_tool_state     │
│    -vscode       │                        │ (already has its own       │
│                  │                        │  synced fixtures for       │
│ new declarative  │                        │  stateful checks)          │
│  test harness    │                        └────────────────────────────┘
│  loads fixtures  │
│  + asserts       │
│  diagnostics     │
└──────────────────┘

The assertion YAML may need to grow a VS-Code-specific extension for range hints, since a lint result in VS Code is {message, range, severity} not a flat string. Proposed extension (backwards compatible — ignored by pytest/vitest):

test_lint_format2_bad_int_default:
  fixture: synthetic-lint-bad-int-default.gxwf.yml
  operation: lint_format2
  assertions:
    - path: [error_count]
      value: 1
    - path: [errors, 0]
      value_contains: "invalid type"
  # new:
  diagnostics:
    - severity: error
      source: "Input Type"
      message_contains: "invalid type"
      at_path: ["inputs", 0, "default"]   # JSONPath-ish — resolved to range by VS Code harness

Branches to Hack On

Unresolved Questions

• Option A vs B for fixture distribution (npm pkg vs. in-tree sync)?
• Source-range extension format — at_path JSON-pointer, or richer? Who owns it?
• Should A17/A18 (pydantic schema strict/lax) be a lint rule in the TS port, or left to baseline Effect decode + a thin wrapper that categorizes the Effect failures into strict-extra vs structural?
• A16 (galaxy-directive markdown validation) — port to TS or drop from the inventory?
• B4 (release required) — backport to gxformat2 best practices? Currently VS-Code-only.
• Native-side structural rules A4/A7/A11/A12/A13 have no VS Code analogue — add them as Native validation rules, or accept that they’re CLI-only checks?
• Who maintains the rule table — is there a generator (parse the profiles.ts, lint.py, lint.ts and diff against this doc) or is it hand-curated?
• C3 (“tool not in cache”) is emitted by VS Code as an info diagnostic with an action, not as a lint error. Does it belong in the inventory or in a separate “diagnostics-only” section?
• Tighten a_galaxy_workflow and format-version to Schema.Literal Done in pydantic schema — TS Effect schemas need a resync (make sync-schema-sources && make generate-schemas).
• Confirm whether native steps keys are typed int in the schema (drives A7).
• Add a non-empty constraint to subworkflow steps (drives A8)?
• After dropping schema-redundant lint rules, do CLI exit codes need to keep mapping schema decode failures to EXIT_CODE_FORMAT_ERROR (today set by lint.py based on lint_context.found_errors)?