TypeScript Tool State Meta-Models Plan

Goal

Create a TypeScript package (galaxy-tool-util) using Effect Schema that can:

1. Accept serialized ToolParameterBundleModel JSON (as produced by Python’s model_dump())
2. Dynamically generate validation schemas for each state representation
3. Validate tool state dicts against those schemas
4. Produce JSON Schema from those schemas
5. (Future) Cache tool objects, produce OpenAPI fragments, etc.

The correctness oracle is parameter_specification.yml from the Galaxy repo — the same YAML that drives the Python test suite.

Key Design Decision: Effect Schema over Zod

Effect Schema chosen because custom validation filters can declare their JSON Schema representation via jsonSchema annotation — Zod’s .refine() is opaque to JSON Schema generation. This matters because Galaxy API exposes these schemas and validators like in_range, regex, length need to appear in JSON Schema output.

Project Structure

galaxy-tool-util/
  package.json
  tsconfig.json
  vitest.config.ts
  src/
    index.ts                          # public API
    schema/                           # Effect Schema generation (this plan)
      index.ts
      state-representations.ts        # StateRepresentation type + registry
      parameters/                     # per-parameter-type schema generators
        index.ts                      # registry: parameter_type -> generator
        base.ts                       # shared types, DynamicSchemaInfo
        gx-integer.ts
        gx-float.ts
        gx-text.ts
        gx-boolean.ts
        gx-select.ts
        gx-data.ts
        gx-data-collection.ts
        gx-color.ts
        gx-hidden.ts
        gx-directory-uri.ts
        gx-drill-down.ts
        gx-data-column.ts
        gx-group-tag.ts
        gx-genomebuild.ts
        gx-baseurl.ts
        gx-rules.ts
        cwl-integer.ts
        cwl-float.ts
        cwl-string.ts
        cwl-boolean.ts
        cwl-file.ts
        cwl-directory.ts
        cwl-null.ts
        cwl-union.ts
      containers/                     # conditional, repeat, section
        conditional.ts
        repeat.ts
        section.ts
      validators/                     # Galaxy XML validators -> Effect filters
        index.ts
        regex.ts
        in-range.ts
        length.ts
        expression.ts
        empty-field.ts
        no-options.ts
      model-factory.ts                # createFieldModel() — assembles schemas
      connected-value.ts              # ConnectedValue union helper
    bundle/                           # ToolParameterBundleModel parsing (future)
      index.ts
    cache/                            # tool object caching (future)
      index.ts
    json-schema/                      # JSON Schema generation (future)
      index.ts
  test/
    parameter-specification.test.ts   # main spec-driven test suite
    fixtures/
      parameter_specification.yml     # copied from Galaxy repo
      parameter_models/               # serialized bundles per tool (generated by Python)
        gx_int.json
        gx_boolean.json
        ...

Input Data

The TypeScript side consumes two artifacts from the Python side:

1. parameter_specification.yml — the test oracle (valid/invalid payloads per tool per state representation)
2. Serialized ToolParameterBundleModel per tool — JSON files produced by running:

   # in Galaxy repo
   PYTHONPATH=lib python test/unit/tool_util/test_parameter_specification.py

   This dumps parameter_models.yml (or we write a small script to dump per-tool JSON files).

The TS test harness loads a tool’s serialized bundle, passes it to createFieldModel(bundle, stateRep), and validates each test case against the resulting Effect Schema.

Enumerated Scope

State Representations (12)

Parameter Types (28 distinct parameter_type values across ~80 tool entries)

Galaxy scalar: gx_integer, gx_float, gx_text, gx_boolean, gx_color, gx_hidden, gx_directory_uri, gx_genomebuild, gx_baseurl

Galaxy choice: gx_select (single + multiple), gx_drill_down, gx_data_column, gx_group_tag

Galaxy data: gx_data (single + multiple), gx_data_collection (many collection_type variants)

Galaxy containers: gx_conditional (boolean + select test), gx_repeat, gx_section

Galaxy special: gx_rules (rule builder with mappings — complex nested dict structure)

CWL: cwl_integer, cwl_float, cwl_string, cwl_boolean, cwl_file, cwl_directory, cwl_null, cwl_union

Validator Types

regex, in_range, length, expression, empty_field, no_options (others like metadata validators are dynamic/runtime-only — skip for now)

Per-Parameter Semantic Quirks

These are non-obvious behaviors that differ from naive assumptions:

• gx_float accepts StrictInt | StrictFloat (not just float) — 5 is valid, "5" is not
• gx_color has embedded validator logic beyond a simple regex — lowercase hex only, exactly 7 chars, with per-state-rep validator dispatch (validate_color_str, validate_color_str_if_value, validate_color_str_or_connected_value)
• gx_hidden forces optional=true for workflow_step (not _linked) even when the parameter is non-optional
• gx_text request_requires_value is unconditionally False (unlike gx_integer where it depends on optional and value)
• gx_hidden_data is NOT a distinct parameter_type — it’s gx_data with optional=true. Tools using it just work when gx_data is implemented.
• safe_field_name escaping: Parameters starting with _ get X prefix in the model field name with alias=original_name for JSON deserialization. Must replicate in TS.
• relaxed_request has per-parameter-type handling — e.g. gx_text allows null for non-optional params (py_type_relaxed_request), not just global “nulls allowed”

Phase 1: Infrastructure — COMPLETE (2026-03-29)

All steps implemented. Test results: 33 passed, 139 skipped.

What was built

• Step 1.1: Project scaffold — effect@3.21.0, @effect/schema@0.75.5, vitest@4.1.2, typescript@6.0.2, ESM module, strict tsconfig
• Step 1.2: src/schema/state-representations.ts — 12 state reps, 7 helper predicates (requiresAllFields, allowsAbsent, allowsBatching, usesStringIds, usesIntIds, allowsConnectedValue, allOptional, allowsUrlSources)
• Step 1.3: src/schema/bundle-types.ts — TypeScript interfaces for all 28 parameter types + container types, collectParameterTypes() and collectValidatorTypes() walkers for skip logic
• Step 1.4: src/schema/model-factory.ts + src/schema/parameters/base.ts + registries — createFieldModel(bundle, stateRep) assembles Schema.Struct from per-parameter generators, safeFieldName() handles _-prefix escaping
• Step 1.5: test/parameter-specification.test.ts — spec runner with:
  • YAML merge key support ({ merge: true })
  • Skip on unregistered parameter_type, validator_type, or state representation
  • Auto-inference of request_internal from request (when request_internal state rep is implemented)
  • Schema.decodeUnknownEither for validation
• Step 1.6: scripts/generate_fixtures.py — generates 77 Galaxy tool bundles (10 CWL skipped, needs cwltool). Copies parameter_specification.yml.

Learnings / deviations from plan

• Effect Schema S.optional() returns PropertySignature, not Schema — assembleStruct() needed S.Struct.Fields typing with a cast
• YAML merge: true option required — yaml library doesn’t resolve << merge keys by default
• Unknown parameter types → undefined return from createFieldModel() instead of permissive Schema.Unknown — cleaner for skip logic, the test harness checks registration before calling
• gx_integer already implemented as part of Phase 1 (needed to validate the infrastructure works) — covers request state rep for gx_int, gx_int_optional, gx_int_required, gx_int_required_via_empty_string
• CWL fixtures need cwltool — not available in system Python, deferred to Round 7
• S.Int from Effect Schema serves as strict integer — rejects strings, floats, booleans. Works well as Pydantic StrictInt equivalent.

Phase 2: First Parameter + First State Representation — COMPLETE (2026-03-29)

Implemented as part of Phase 1 infrastructure validation.

• src/schema/parameters/gx-integer.ts — handles request state rep
  • S.Int for strict integer validation
  • S.NullOr(S.Int) when optional: true
  • isOptional logic: requiresAllFields(stateRep) → false, allOptional(stateRep) → true, else based on p.optional || p.value !== null
• All 4 gx_int* tools passing for request: 33 tests green

Phase 3: Iterative Expansion

This phase is a structured loop. Each iteration adds either a parameter type or a state representation (or both) and fixes tests until green. The order is chosen to maximize coverage per iteration.

Critical ordering note: Validators must be implemented alongside the parameter types that use them. The skip logic checks both parameter_type registration AND validator support to prevent premature un-skipping.

Round 1 — Scalars, choice, all validators, request state rep — COMPLETE (2026-03-29)

Test results: 216 passed, 323 skipped (up from 33 passed).

Parameter types implemented (10 total):

• gx_integer — S.Int, implicit in_range from min/max, S.NullOr when optional
• gx_float — S.Number.pipe(S.finite()) (accepts int or float), same validator pattern as integer
• gx_boolean — S.Boolean strict, requestRequiresValue always false
• gx_text — S.String, nullable when optional, requestRequiresValue based on !optional && value === null
• gx_color — S.String.pipe(S.pattern(/^#[0-9a-fA-F]{6}$/)), always has default
• gx_hidden — S.String + validators, nullable when optional
• gx_select — literal union from options[], S.Array when multiple, S.NullOr when optional/multiple
• gx_directory_uri — S.String + gx*:// prefix filter + validators
• gx_genomebuild — S.String (dynamic options), S.Array when multiple, S.NullOr when optional
• gx_baseurl — S.String, nullable when optional

Validators implemented (6 total):

• in_range — min/max with exclude_min/exclude_max/negate
• regex — JS RegExp from expression string, negate support
• length — min/max string length, negate support
• expression — subset parser for 'literal' in value and value == 'literal' patterns; unrecognized expressions pass through
• empty_field — non-empty string check, negate support
• no_options — no-op at schema level (runtime option availability check)

Architecture established:

• Validator registry (validators/registry.ts) with applyValidators() dispatch — parameter generators call this to fold validators onto their base schema
• Validators applied before nullable wrapping (null bypasses validators for optional params)
• Implicit in_range synthesized from min/max fields on integer/float params

Post-Round 1 Review Fixes — COMPLETE (2026-03-29)

A subagent review identified P0/P1 issues. All fixed:

• P0: fromKey alias on non-optional fields — assembleStruct now wraps with S.propertySignature() before S.fromKey() for non-optional aliased fields (was silently broken for _-prefixed required params)
• P1-1: GeneratorContext for container recursion — ParameterSchemaGenerator signature now takes (param, stateRep, ctx). Context provides buildChildSchema(params, stateRep) callback so container generators can recurse without circular imports on model-factory.ts. All 10 generators updated.
• P1-2: BooleanParameterModel.truevalue/falsevalue — changed from string to string | null (fixtures have null)
• P1-3: Container type interfaces — ConditionalParameterModel, RepeatParameterModel, SectionParameterModel now extend ContainerBaseFields with hidden, label, help, argument, is_dynamic
• P1-5: collectValidatorTypes test_parameter — now checks validators on conditional’s test_parameter (was only walking when-branch children)

Tooling added:

• ESLint + typescript-eslint, Prettier
• Makefile with make lint, make format, make typecheck, make test, make check (all static), make fix

Round 2 — Container parameters — COMPLETE (2026-03-29)

Test results: 275 passed, 347 skipped (up from 216).

Extended GeneratorContext with buildChildSchemaInfos() and assembleStruct() methods to give containers access to raw field infos and struct assembly without circular imports.

Container types implemented (3):

• gx_conditional — discriminated S.Union of per-branch structs. Each branch has S.Literal(when.discriminator) on test_parameter + child params. Test_parameter is S.optional only in the is_default_when branch for non-requiresAllFields states. Handles boolean (true/false) and select (string literals) discriminators. Nested conditionals (conditional_conditional_boolean) work via natural recursion.
• gx_section — thin wrapper. Child struct via ctx.assembleStruct(childInfos). Section optional if all children optional (anyChildRequired drives requestRequiresValue).
• gx_repeat — S.Array(childStruct) with S.filter for min/max length. Repeat optional in request only when children DON’T require values OR min is 0/null. Formula: requestRequiresValue = anyChildRequired && min > 0.

Design decisions:

• Conditional uses S.optional(S.Literal(discriminator)) in default branch, not a separate absent-case struct. Works because S.Union tries all branches and only the default branch allows missing test_parameter.
• onExcessProperty: "error" at decode time (not struct definition) ensures wrong-branch fields are rejected.
• Repeat min/max as S.filter not S.minItems/S.maxItems (more portable across Effect Schema versions).

Round 3 — Choice, data, data_collection, data_column, group_tag — COMPLETE (2026-03-29)

Test results: 498 passed, 548 skipped (up from 275). All remaining skips are state rep coverage.

Parameter types implemented (5), completing ALL 18 Galaxy types:

• gx_drill_down — collects valid values from recursive options tree. hierarchy: "exact" = all nodes, hierarchy: "recurse" + single = leaf-only, hierarchy: "recurse" + multiple = all nodes. Literal union from values. No implicit default (requestRequiresValue = true).
• gx_data — HDA direct ({src: "hda", id: <string>}), URL source ({src: "url", url, ext}), Batch ({__class__: "Batch", values: [hdca|dce]}) with optional map_over_type. Multiple mode adds HDCA direct + array-of-sources variant. Uses usesStringIds(stateRep) and allowsBatching(stateRep) for state rep conditioning.
• gx_data_collection — HDCA ref + inline Collection ({class: "Collection", collection_type, elements: Array(Unknown)}). Elements permissive for now (recursive nesting).
• gx_data_column — S.Int (strict integer column index). Multiple = S.Array(S.Int). Default from value field.
• gx_group_tag — S.String, multiple = S.Array(S.String).

Bundle type updates: DrillDownParameterModel.hierarchy, DataColumnParameterModel.value, DataCollectionParameterModel.extensions/value.

Review findings (all addressed):

• URL source gating: allowsUrlSources() exists in state-representations but is incorrect (says request_internal only, but request also allows URLs). Deferred to state rep expansion.
• Shared data source schemas: minor DRY opportunity between gx-data and gx-data-collection (idSchema, hdcaSource). Not worth extracting yet.
• Repeat min/max logic: verified correct against spec comment in gx_repeat_boolean_min (“can skip the repeat all together”).

Round 4 — Expand state representations — COMPLETE (2026-03-29)

Test results: 2087 passed, 10 skipped (up from 498). All 12 state representations implemented. Remaining 10 skips are CWL parameter types (no fixture bundles).

All 12 state reps implemented in one pass:

1. request_internal + request_internal_dereferenced + landing_request + landing_request_internal — Scalars just worked via existing computeIsOptional. Data changes: allowsUrlSources fixed to cover all request-like reps (was wrong — only had request_internal). request_internal_dereferenced excludes URL sources AND inline collections for data_collection.
2. job_internal — All required (via computeIsOptional). Data: {src: "hda"|"dce", id: int} — added dce direct source. Data collection: {src: "hdca"|"dce", id: int}. No batch, no URL, no inline collection.
3. job_runtime — All required. Data: {class: "File", basename, location, path, nameroot, nameext, format, size, element_identifier?}. Data collection: recursive collection_type-aware validation — list → array elements, paired → {forward, reverse} record, nested types like list:paired validated recursively. Extra fields (column_definitions, has_single_item) as optional.
4. workflow_step + workflow_step_linked — Data params use S.Unknown.pipe(S.filter(() => false)) (rejects all values). workflow_step: data always optional (absent only). workflow_step_linked: central ConnectedValue wrapping in model-factory.ts via S.Union(schema, ConnectedValueSchema). Array types (select_multiple) use connectedValueHandled flag for item-level ConnectedValue instead of outer wrapping.
5. test_case_xml + test_case_json — Data: S.Union({class: "File", path}, {class: "File", location}). Data collection: recursive schema with typed File elements ({class: "File", identifier, path}), validates nested collection structure. test_case_xml accepts comma-separated strings for select_multiple and data_column_multiple.
6. relaxed_request — Only 6 tests, mostly same as request. Key difference: gx_text allows null for non-optional params (null coercion via stateRep === "relaxed_request" check).

Key design decisions:

• ConnectedValue wrapping is centralized in buildSchemaInfos (model-factory.ts) — added after each generator returns. Generators that handle it themselves (array types) set connectedValueHandled: true on DynamicSchemaInfo.
• S.Unknown.pipe(S.filter(() => false)) used instead of S.Never to avoid TypeScript variance issues with S.Schema.Any.
• Repeat min/max constraints skipped for allOptional state reps (landing_request allows single-element arrays even when min=2).
• isWorkflowStep() and isTestCase() helpers added to state-representations.ts for clean branching.

Round 5 — Special parameters

7. gx_rules — rule builder with mappings, complex nested dict structure

Round 6 — CWL parameters

8. cwl_integer, cwl_float, cwl_string, cwl_boolean — simple wrappers
9. cwl_file, cwl_directory — CWL-specific data refs
10. cwl_null — validates Literal[None]
11. cwl_union — union of other CWL types (recursive)

Development harness for each iteration

1. Pick next item from the iteration order
2. Run tests: make test — note which tools/reps are newly unskipped
3. Implement the parameter type or state representation
4. Run tests — iterate until all newly-unskipped tests are green
5. Verify no regressions: make check && make test
6. Commit

Phase 4: Drop All Skips

Step 4.1: Remove Skip Logic

• Remove the parameter_type registry check that skips unknown types
• Remove the state representation skip check
• All spec entries must now pass or explicitly be marked as expected_failure (if any Python/TS semantic differences exist)

Step 4.2: Full Green

• Run full suite, fix any remaining failures
• These should only be edge cases missed during iterative dev

Step 4.3: CI

• Add GitHub Actions workflow
• Run npx vitest run on PR
• Add fixture regeneration check (ensure committed fixtures match Python output)

Future Work (not in this plan, but project structure supports)

• JSON Schema generation: JSONSchema.make() from Effect Schema on generated models, with custom annotations for Galaxy-specific metadata
• Tool object caching: LRU cache keyed on tool ID + version, returns precomputed schemas for all state reps
• OpenAPI fragments: Generate OpenAPI 3.1 request/response schemas per tool
• Bundle validation: Validate incoming ToolParameterBundleModel JSON with Effect Schema (currently trusted)
• Conversion functions: decode(), encode(), runtimeify() — visitor pattern over parameter tree
• State representation auto-inference: Infer request_internal from request like the Python test runner does
• Galaxy client integration: Use generated schemas for client-side form validation in the Galaxy UI

Risks

Unresolved Questions

• Generate fixture JSON from Galaxy .venv in this repo or maintain a separate script in the Galaxy repo?
• Should the Python fixture generation script live in this repo (with Galaxy as a dev dependency) or in Galaxy itself?
• Pin to a specific Galaxy commit for fixtures, or always regenerate from Galaxy main?
• For relaxed_request null-to-default coercion Resolved: Simple stateRep === "relaxed_request" check in gx-text.ts. Only gx_text needed it.
• CWL parameters share some spec entries with Galaxy — share code or keep separate implementations?
• Should the __absent__ branch for conditionals use Schema.optional on the test field or omit it entirely from the struct? Resolved: S.optional(S.Literal(discriminator)) on the test field in the default when-branch.
• gx_rules has a complex nested dict structure (rules + mappings) — implement fully or treat as opaque Schema.Unknown initially?
• gx_data inline collection elements Resolved: test_case collections now have recursive typed validation (File + nested Collection). job_runtime collections validate collection_type + element structure recursively.
• Data source schema duplication Resolved: Not extracted. Each generator has its own state-rep-specific logic that would complicate sharing. Minimal overlap in practice.