TypeScript CLI Layer Overhaul Plan

Date: 2026-04-03 Repo: jmchilton/galaxy-tool-util-ts (galaxy-tool-util) Goal: Bring TS CLI surface into spiritual parity with the Python gxwf-* namespace — expose cleaning, linting, format conversion, and roundtrip operations that already exist in @galaxy-tool-util/schema but lack CLI commands.

Current State

What the TS CLI exposes today

What the schema/core packages already implement (no CLI)

Python CLI commands (for parity reference)

gxformat2 (schema-free):

• gxwf-to-native — format2 -> native
• gxwf-to-format2 — native -> format2
• gxwf-lint — structural lint
• gxwf-viz — Cytoscape visualization (out of scope for TS)
• gxwf-abstract-export — abstract CWL export (out of scope for TS)

galaxy-tool-util workflow_state (schema-aware):

• gxwf-state-validate / gxwf-state-validate-tree
• gxwf-state-clean / gxwf-state-clean-tree
• gxwf-roundtrip-validate / gxwf-roundtrip-validate-tree
• gxwf-to-format2-stateful / gxwf-to-format2-stateful-tree
• gxwf-to-native-stateful / gxwf-to-native-stateful-tree
• gxwf-lint-stateful / gxwf-lint-stateful-tree
• galaxy-tool-cache (populate-workflow, add, add-local, list, info, clear, schema, structural-schema)

Plan

Step 0: Unify under galaxy-workflow CLI

Replace the standalone galaxy-workflow-validate binary with a single galaxy-workflow command with subcommands. This mirrors how galaxy-tool-cache already works and avoids name explosion as we add operations.

New binary: packages/cli/src/bin/gxwf.ts

gxwf validate <file>     # existing validate (structural + tool state)
gxwf clean <file>        # new
gxwf lint <file>         # new
gxwf convert <file>      # new (format conversion)

Deprecation: Drop galaxy-workflow-validate and do not worry about backward compatibility.

package.json bin entries:

{
  "galaxy-tool-cache": "./dist/bin/galaxy-tool-cache.js",
  "gxwf": "./dist/bin/gxwf.js",
}

Step 1: gxwf clean

Expose cleanWorkflow() via CLI.

gxwf clean <file> [--output <file>] [--diff] [--format native|format2]

Implementation: packages/cli/src/commands/clean.ts

Logic:

1. Read + parse file (JSON or YAML based on extension)
2. cleanWorkflow(data) — mutates in place
3. Output result or diff

Note: The Python version has stale key classification/policy (--preserve/--strip/--allow/--deny) and schema-aware cleaning (keys not in tool def). The TS cleanWorkflow() currently only strips hardcoded stale key sets. We should document this gap but not block the CLI on closing it — the hardcoded set covers the common case.

Tests:

• Unit test: clean a workflow with stale keys, verify they’re removed
• Unit test: clean a format2 workflow, verify passthrough
• CLI integration test: run command on fixture, check stdout/exit code

Step 2: gxwf lint

Unified lint combining structural checks, best practices, and tool state validation — equivalent to Python’s gxwf-lint-stateful. Tool state validation is on by default; the two skip flags let callers opt out of the expensive or opinionated phases.

gxwf lint <file> [--skip-best-practices] [--skip-state-validation] [--format native|format2]

Implementation: packages/cli/src/commands/lint.ts

Logic — three phases, each independently skippable:

1. Structural lint — lintWorkflow(data) (always runs)
2. Best practices — lintBestPracticesNative(data) or lintBestPracticesFormat2(data) based on format (skip with --skip-best-practices)
3. Tool state validation — reuse the existing validateNativeSteps() / validateFormat2Steps() logic from validate-workflow.ts (skip with --skip-state-validation). Requires tool cache; if cache is empty and --skip-state-validation not set, warn and degrade gracefully (report skipped steps, don’t hard-fail).

Combined exit code: 0 = clean, 1 = warnings only, 2 = structural/state errors.

This means gxwf lint subsumes gxwf validate — validate becomes a convenience alias for gxwf lint --skip-best-practices. We should note this in docs but keep validate as a command since it’s already established and the mental model (“I just want to check if this is valid”) is distinct from “lint me everything”.

Dependency: Step 2 should refactor the validation logic in validate-workflow.ts into a reusable function (returning StepValidationResult[]) that both lint.ts and validate-workflow.ts call. The current runValidateWorkflow() mixes validation with CLI output; split into validateWorkflow() (pure logic) + runValidateWorkflow() (CLI wrapper).

Tests:

• Lint a well-formed workflow with cached tools, verify all three phases pass
• Lint a workflow missing annotations/creators, verify best practice warnings
• Lint a workflow with broken output sources, verify structural errors
• Lint a workflow with invalid tool state, verify state validation errors
• --skip-best-practices suppresses those warnings but state validation still runs
• --skip-state-validation suppresses state checks but structural + best practices still run
• Both skip flags together = structural-only lint
• Empty tool cache + no skip flag = graceful degradation (skipped steps, not hard failure)

Step 3: gxwf convert

Expose toFormat2() and toNative() via CLI.

gxwf convert <file> [--to native|format2] [--output <file>] [--compact] [--json|--yaml]

Implementation: packages/cli/src/commands/convert.ts

Logic:

1. Read + parse
2. Detect format, determine target
3. toFormat2(data) or toNative(data)
4. Serialize (YAML for format2, JSON for native, overridable)

Note: This is the schema-free conversion path (like Python’s gxwf-to-format2 / gxwf-to-native). The Python project also has gxwf-to-format2-stateful and gxwf-to-native-stateful which use tool definitions for smarter state conversion. Those would be a follow-up feature requiring the tool cache integration.

Tests:

• Convert native -> format2, verify structure
• Convert format2 -> native, verify structure
• Round-trip: convert native -> format2 -> native, verify equivalence on key fields
• --compact strips position data
• Auto-detection of target format

Step 4: Enhance galaxy-tool-cache

Add missing subcommands from the Python version.

4a: galaxy-tool-cache populate-workflow

galaxy-tool-cache populate-workflow <path> [--cache-dir <dir>] [--galaxy-url <url>]

Scan a workflow file (or directory of workflows), extract all tool_id/tool_version pairs, fetch and cache each one. This is the TS equivalent of galaxy-tool-cache populate-workflow in Python.

Implementation: packages/cli/src/commands/populate-workflow.ts

Logic:

1. Read workflow, detect format
2. Normalize and collect all tool steps (with subworkflow recursion)
3. For each unique (tool_id, tool_version), call existing add logic
4. Report: cached N tools, M already cached, K failed

4b: galaxy-tool-cache structural-schema

galaxy-tool-cache structural-schema [--output <file>] [--strict]

Export the gxformat2 GalaxyWorkflow JSON Schema. This enables external tooling to validate workflow structure without the TS runtime.

Implementation: packages/cli/src/commands/structural-schema.ts

Logic:

1. JSONSchema.make(GalaxyWorkflowSchema) (Effect’s built-in JSON Schema generation)
2. Write to file or stdout

Step 5: Tree/batch mode

Mirror the Python -tree suffix as subcommands under gxwf. Each tree variant takes a directory instead of a file, discovers workflows, processes each, and aggregates results.

gxwf validate-tree <dir>
gxwf lint-tree <dir>
gxwf clean-tree <dir> --output-dir ./cleaned/
gxwf convert-tree <dir> --to format2 --output-dir ./converted/

Tree-specific options (not on single-file variants):

Shared logic in packages/cli/src/commands/tree.ts:

1. Discover workflow files (.ga, .gxwf.yml) recursively
2. Process each with the corresponding single-file logic
3. Aggregate results with summary counts (pass/fail/skip)
4. Exit code = worst exit code across all files

Maps directly to the Python _tree_orchestrator.py pattern. Subcommands under one binary instead of separate binaries.

Step 6: Documentation overhaul

6a: Update docs/packages/cli.md

Restructure to document both CLIs:

# CLI Reference

## galaxy-tool-cache
### add
### list
### info
### clear
### schema
### populate-workflow        # new
### structural-schema        # new

## gxwf
### validate / validate-tree
### clean / clean-tree           # new
### lint / lint-tree             # new
### convert / convert-tree       # new

6b: New guide: docs/guide/workflow-operations.md

Expand docs/guide/workflow-validation.md into a broader operations guide:

• Validation — structural + tool state, effect vs json-schema backends
• Cleaning — stale key removal, when to clean before validating
• Linting — structural checks + best practices, relationship to validation
• Format conversion — native <-> format2, when to use, limitations
• Batch processing — recursive mode, report formats
• Tool cache management — populating cache, schema export

6c: Update docs/architecture/overview.md

Add workflow operations to the architecture diagram showing the relationship between schema-level operations and CLI commands.

6d: Python parity table in docs

Add a table mapping Python CLI commands to TS equivalents:

Implementation Order

1. Step 0 — gxwf binary + migrate validate ✅ (ddc8da3)
2. Step 1 — clean command ✅ (ddc8da3)
3. Step 2 — lint command ✅ (ddc8da3)
4. Step 3 — convert command ✅ (ddc8da3)
5. Step 4a — populate-workflow ✅ (pending commit)
6. Step 4b — structural-schema ✅ (pending commit)
7. Step 5 — Tree variants (validate-tree, lint-tree, clean-tree, convert-tree) ✅ (947915b)
8. Step 6 — Documentation ✅ (pending commit)

Steps 1-3 are independent once step 0 is done and can be parallelized.

Post-implementation refactors (done)

After Steps 0–3, extracted shared abstractions to prepare for tree variants:

• writeWorkflowOutput() in workflow-io.ts — consolidated write-to-file-or-stdout pattern
• renderStepResults() in render-results.ts — shared step validation rendering
• lintWorkflowReport() exported from lint.ts — accepts pre-loaded ToolCache (avoids double-loading in tree mode)
• Test helpers: createCliTestContext() + shared fixtures (seedSimpleTool, seedAllTools)

Schema identifier annotations

structural-schema required identifier annotations on recursive Schema.suspend() calls for JSONSchema.make().

• Forward references: fixed in schema-salad-plus-pydantic 0.1.6 (#6)
• Backward circular references (e.g. WorkflowStep.run → GalaxyWorkflow): manually patched; tracked in #8 for code generator fix

Future Work (not in this plan)

These represent Python parity features that need deeper library work:

• Schema-aware conversion (--stateful flag on convert) — needs tool cache integration in conversion path, walker callbacks for state encoding/decoding
• Roundtrip validation (gxwf roundtrip) — needs schema-aware convert in both directions, structured diff classification
• Connection validation (--connections flag on validate) — needs connection graph builder, type propagation
• Stale key classification/policy — needs schema-aware key detection (keys not in tool def vs hardcoded set)
• Legacy encoding detection / prechecking — detection exists in schema package (scanToolState), could be exposed as gxwf precheck
• Report output modes (--report-json, --report-markdown) — structured output for CI integration

Unresolved Questions

• Should gxwf support reading from stdin for piping (cat wf.ga | gxwf lint -)?
• Should tree variants share all options with their single-file counterpart, or only the subset that makes sense?
• For clean --diff: use unified diff format or a custom format showing before/after per-step?
• populate-workflow: should it also accept a directory and auto-discover workflows (overlapping with tree mode)?
• Should we emit changesets per-step (0→4a, etc.) or batch into a single changeset for the whole overhaul?