Debrief: .ga export un-stringify + validate-gated clean

Date: 2026-06-30 Branch: wf_tool_state Trigger: IWC PR #1158 (mvdbeek: “Galaxy can just remove the extra JSON encoding on tool_state”)

What was asked

Make .ga exports drop the extra JSON encoding on tool_state (emit a JSON object, not a double-encoded string). Safe transform.
Clean workflows and, per step, only export the cleaned state if it validates. Scarier transform; validate is the safety net.

Decisions (confirmed with user)

Wire into both the Galaxy API download (workflow_to_dict) and the gxwf-state-clean CLI, sharing logic in clean.py.
Un-stringify = opt-in flag, not a new default. ~7 internal consumers do json.loads(step["tool_state"]) (editor, extraction, assertions); flipping the default is a separate sweep.
Validate gate = native schema validation, revert per step. Validate cleaned state via WorkflowStepNativeToolState.model_validate; if a step fails, revert that step’s tool_state to pre-clean, keep the rest.

What shipped

Area	Change
`clean.py`	`clean_stale_state(..., validate=False)`; `_revert_if_invalid()` per-step gate; `reverted_steps` on `CleanResult`; threaded through `clean_single`/`clean_tree`/`run_clean`/options.
`_report_models.py`	`CleanStepResult.reverted` + `revert_reason` (golden `clean_tree.json` regenerated).
CLI	`--validate` flag on `gxwf-state-clean` + `-tree` (dest `validate_state` to avoid shadowing Pydantic `validate`).
`managers/workflows.py`	`workflow_to_dict(..., clean_validate, tool_state_as_dict)`; `_clean_native_dict(..., validate)`; new `_normalize_native_tool_state()` (ga style only; recurses subworkflows).
`api/workflows.py`	`clean_validate` + `tool_state_as_dict` query params on `GET /download`.
`populators.py`	`download_workflow()` gained `clean`/`clean_strip`/`clean_preserve`/`clean_validate`/`tool_state_as_dict`.
Tests	`test_clean_validate_gate.py` (5 unit), `TestWfExportToolStateEncoding` (4 API).

Bug found & fixed (the real find)

clean=true never stripped anything through the API. _workflow_to_dict_export keys steps by int order_index, but clean_stale_state looked up raw_steps.get(str(step_id), raw_steps.get(step_id, {})) — both branches resolve to the same string-ish value, never the int — so every step fell through to {} and was silently skipped. Disk-loaded .ga (string keys, used by every existing test + the whole declarative/IWC corpus) masked it; no test had ever exercised the in-memory export dict. Fixed with _raw_step_def() (tries str then int key). Regression test: test_clean_with_integer_step_keys.

This means the prior CURRENT_STATE claim “clean via GET /download?clean=true works” was untested and false for the native path.

Verification

test/unit/tool_util/workflow_state/ — 720 passed, 16 skipped.
test/unit/workflow/test_workflow_state_tree.py — 19 passed.
API TestWfExportToolStateEncoding — 4 passed (default double-encodes; tool_state_as_dict emits dict + re-imports; clean strips bookkeeping; clean_validate keeps validating steps).
mypy clean on changed files; ruff: 0 new errors (pre-existing UP045 only).

Caveats / follow-ups

gxformat2 downgrade: run_tests.sh → common_startup downgraded the venv’s gxformat2 from the pinned parameter_models git branch to PyPI 0.27.0. Tests still passed under 0.27.0, but the venv is off-pin until reinstalled. (Agent-initiated reinstall of the external git branch was blocked by the permission classifier.)
Validate gate is native-path only; clean_format2_state ignores validate (format2 already carries schema-aware state).
Un-stringify default flip + updating the ~7 internal json.loads consumers remains a separate task.
clean=true working through the API is new as of this fix — worth a note wherever the API clean capability is advertised.