Galaxy Brain

MAP_OVER_EMPTY_EXTRACTION_TOOL_REQUEST_PLAN

Map-Over-Empty Workflow Extraction via Tool Request — Implementation Plan

Date: 2026-05-17 Branch: extract_issue_followups (off graph_workflow_extract). Depends-on: EXTRACT_TOOL_REQUEST_STATE_PLAN landed/being the base (the structured converter, shared ref-walk, and the source-neutral _structured_request_payload seam this plan extends all live there, commit a34923b7c9). Tracking issue: #21788 (empty collection → empty workflow); class also covers the #18484 follow-up and the extract.py:323 “track via tool request model” TODO. Related:

  • EXTRACT_TOOL_REQUEST_STATE_PLAN — the gate this builds on
  • CAPTURE_WORKFLOW_EXECUTION_STATE_PLAN — the pair: closes the literal repro by feeding the same seam
  • GRAPH_WORKFLOW_EXTRACTION_PLAN — consumes the new tool_request_ids selection primitive
  • QUEUED_EXECUTION_EXTRACTION_TOOL_REQUEST_PLAN — the second use case: same primitive, queued/running (grey) executions (#7003)
  • vault/research/Workflow Extraction Issues.md

At a glance

ProblemEmpty collection → map-over expands to zero jobs → job-based extraction has no representative job → step dropped → 0-step workflow (#21788).
Key insightThe ToolRequest is the abstract step description that exists with zero jobs. Extraction should source the step from the request, not from jobs.
This plan deliversA tool_request-sourced extraction path: select by tool_request_ids, synthesize the tool step from ToolRequest.request + ToolRequest.tool_source with no job. Closes the #21788 class on the async tool-request API path.
This plan does NOTClose the literal repro test_empty_collection_map_over_extract_workflow — that runs via workflow invocation (no ToolRequest). That remainder is CAPTURE_WORKFLOW_EXECUTION_STATE_PLAN, which feeds the same seam built here (one function, no re-architecture).
RiskLow. Every hard part (structured conversion, ref-walk, source-neutral seam, error mapping) already exists and is tested. New work is selection plumbing + dropping a job-dependency + ToolSource→Tool reuse.

Implementation status — SHIPPED (verified 2026-05-17, uncommitted on graph_workflow_extract)

StepStateEvidence
Spike (TOOL_FROM_SOURCE)✅test_tool_from_persisted_source_drives_workflow_step_state PASS
Schema tool_request_ids + _at_least_one_input✅unit + API
Validator branch (skip job/output gates; access-check)✅full ByIds class
Jobless extract path + severed job dep in _structured_step_inputs_by_id✅headline test PASS
Output wiring / chained jobless map-overs✅test_extract_chained_empty_map_over_tool_requests_by_ids PASS
Scope boundary (workflow-invocation repro stays red)✅untouched, SCOPE_TOOL_REQUEST_ONLY held

Verification: unit 99 passed, 1 skipped; API TestWorkflowExtractionByIdsApi 33 passed, 0 failed; black + isort clean. Headline test_extract_empty_map_over_tool_request_state_by_ids (#21788) green; was 500 before.

Decisions taken during impl (beyond the plan’s settled set):

Verified facts (probe, 2026-05-17 — empirical, not assumed)

A throwaway probe (tool_request_raw of cat1 Batch over a genuine empty list produced by the empty_list tool) established the real behavior on a running server:

  1. tool_request_raw over an empty collection returns 200 and creates a ToolRequest that resolves to a terminal/OK state (wait_on_tool_request → True).
  2. ToolRequest.request is preserved intact: {"input1": {"__class__": "Batch", "values": [{"src": "hdca", "id": <empty hdca>}]}}. The empty HDCA still carries its db id in the payload → the collection-input step and its connection are fully recoverable.
  3. jobs == [] — zero jobs, no representative job. Exactly the #21788 condition, reproduced on the tool-request API path (no workflow invocation).
  4. ToolRequest.implicit_collections == [{"src": "hdca", "id": <hdca>, "output_name": "out_file1"}] — an empty output HDCA + a ToolRequestImplicitCollectionAssociation exist even with zero jobs. Outputs are resolvable from the request, not jobs.
  5. Current POST /api/workflows/extract selecting that ICJ → HTTP 500 (unhandled crash at ImplicitCollectionJobs.representative_job, “lowest-order constituent job” over an empty set).
  6. The jobless ICJ’s ImplicitCollectionJobs.tool_request (the helper added in EXTRACT_TOOL_REQUEST_STATE_PLAN, derived from constituent jobs) returns None — there are no jobs to derive it from. The ToolRequest for a jobless execution is reachable only via the output HDCA’s tool_request_association, or by selecting the ToolRequest directly.

Fact 6 is the design pivot.

Why this exists

#21788 (and the copied/dynamic-collection cluster’s “nothing to trace” failures) all share one root: extraction needs a representative job, and an empty map-over has none. The whole point of ToolRequest (EXTRACT_TOOL_REQUEST_STATE_PLAN) is that it is the validated abstract description of an execution that exists before and independent of jobs. EXTRACT_TOOL_REQUEST_STATE_PLAN proved structured state can be synthesized from ToolRequest.request alone. This plan removes the last job-dependency in that path so a step with zero jobs still extracts — turning “0-step workflow” into the correct structurally-complete workflow.

The literal repro is workflow-invocation-produced (no ToolRequest); that remainder is owned by CAPTURE_WORKFLOW_EXECUTION_STATE_PLAN. Critical: because EXTRACT_TOOL_REQUEST_STATE_PLAN‘s _structured_request_payload(job, icj) seam is source-neutral (returns a payload dict, not a ToolRequest object), capture-state closes the remainder by extending that one function to also source WorkflowInvocationStep — no re-architecture, no consumer changes. This plan and the capture plan are a pair; this one ships the mechanism + a real slice now.

Settled decisions

Architecture / seam

 POST /api/workflows/extract
   payload.tool_request_ids = [R]                ← NEW selection primitive
        │
        ▼
 _validate_extract_by_ids_payload                ← NEW branch: tool-request
   (services/workflows.py)                         selection skips job/output
        │                                          gates; access-check the
        ▼                                          ToolRequest's history
 extract_steps_by_ids (workflow/extract.py)
   work item = (job=None,
                output_hdcas ← ToolRequest.implicit_collections,
                request_payload ← _structured_request_payload(...))   ← seam, reused
        │
        ▼
 step_inputs_by_id(trans, job=None, request_payload=…)
        │  request_payload is not None → structured path (unchanged dispatch)
        ▼
 _structured_step_inputs_by_id(trans, tool, request_payload)   ← job dependency REMOVED
   tool ← TOOL_FROM_SOURCE(ToolRequest.tool_source)            ← reuse executor recon
   state ← to_workflow_step_state(RequestInternalToolState(payload), bundle)  [exists]
   assoc ← request_internal_input_refs(payload)                              [exists]
        │
        ▼
 WorkflowStep(tool) + data_collection_input(empty hdca, type "list") wired
   outputs (empty HDCA) registered in id_to_output_pair via
   ToolRequest.implicit_collections[*].output_name  → downstream steps still wire

The only genuinely new logic is: the tool_request_ids selection + validator branch, the jobless work-item construction, and TOOL_FROM_SOURCE. Everything below the seam is reused unchanged from EXTRACT_TOOL_REQUEST_STATE_PLAN.

TOOL_FROM_SOURCE — RESOLVED (spike passing 2026-05-17)

The reconstruction function already exists and is the production tool-request path — confirmed by tracing the celery task:

Extraction reuses create_tool_from_representation directly (OPTION_RECONSTRUCT — the only option that handles uninstalled/dynamic tools and matches the executor for provenance fidelity). OPTION_TOOLBOX (toolbox.get_tool) is rejected: the whole point is decoupling from live toolbox state.

The narrowed residual question — RESOLVED by spike (2026-05-17). The persisted ToolSource model (model/__init__.py:1402-1408) stores only source + source_class — not tool_dir or tool_id/guid. The celery path supplies those from the live tool at request time (services/jobs.py:281-286); at extraction time we have only the blob. Spike test/unit/workflows/test_extract_tool_request_state.py::test_tool_from_persisted_source_drives_workflow_step_state (PASSING) pins it: a cat1-shaped tool reconstructed via create_tool_from_representation(app, original.tool_source.to_string(), tool_dir=None, tool_source_class="XmlToolSource", guid=None) yields correct id/version, non-None .parameters, and that bundle drives the exact _structured_step_inputs_by_id pipeline (ToolParameterBundleModel → RequestInternalToolState.validate → to_workflow_step_state) to {"input1": {"__class__": "ConnectedValue"}}. Independently corroborated by pre-existing test/unit/app/tools/test_tool_deserialization.py (XML/YAML reconstruct with no tool_dir, tool.inputs populated). tool_dir=None/guid=None is sufficient for the built-in slice — extract.py is unblocked. Installed-tool/macro edge (relative file/macro resolution) deferred, not on the built-in path; if it ever bites, fallback options unchanged: (a) toolbox.get_tool by source-derived id when installed, (b) typed error (boundedness invariant), (c) persist tool_dir/guid follow-up.

Files to touch (checklist)

lib/galaxy/schema/workflows.py

lib/galaxy/webapps/galaxy/services/workflows.py

lib/galaxy/workflow/extract.py

Model (lib/galaxy/model/__init__.py) — likely no schema change

Tests — see red-to-green.

Red-to-green test order

Project convention: red first, then green. One suite at a time; API suite needs the localhost-bind sandbox escalation.

  1. RED — API repro (the headline). test/…/api/test_workflow_extraction.py::TestWorkflowExtractionByIdsApi: produce an empty list via the empty_list tool, tool_request_raw cat1 Batch over it, POST /api/workflows/extract with tool_request_ids=[R]. Assert the desired green: a 2-step workflow — data_collection_input (collection_type == "list") + cat1 tool step (tool_state == {"input1": {"__class__":"ConnectedValue"}}), input connected. Currently 500 → fails. (Reuse _run_tool_request_get_request_and_jobs, _assert_collection_extract_state-style assertions, and the _icj_id_for_hdca/tool_request helpers already in the file.)
  2. Spike unit test. test/unit/tool_util/… or test/unit/workflows/test_extract_tool_request_state.py: ToolSource(source=…, source_class=…) reconstructs to a tool whose parameters bundle drives to_workflow_step_state for a representative tool (cat1). Pins TOOL_FROM_SOURCE.
  3. GREEN — schema + validator. tool_request_ids field + validator branch. Add a focused service/API test: tool-request selection with zero jobs is accepted (no 400/500 from the job/output gates).
  4. GREEN — jobless extraction path. extract_steps_by_ids tool-request work item + severed job dependency in _structured_step_inputs_by_id. Makes test 1 green. Add a non-empty tool-request tool_request_ids test too (parity with the existing job/ICJ-sourced structured tests — proves the new selection primitive is general, not empty-only).
  5. GREEN — output wiring. Two-step selection: an empty map-over feeding a downstream tool-request step, both selected by tool_request_ids; assert the downstream step connects to the empty producer’s output (the #21788 “structure preserved” goal in full).
  6. Regression guard — scope boundary. Keep test_empty_collection_map_over_extract_workflow (workflow-invocation) red/TODO unchanged; add a comment-free assertion in the new tests that this path is tool-request-only. Do not green the workflow-invocation repro here.

Run after each: ./run_tests.sh -api lib/galaxy_test/api/test_workflow_extraction.py and the touched unit files (tox -e unit -- …).

Out of scope (do not pull in)

Resolved questions

Unresolved questions

References (in-repo, file:line — read at extract_issue_followups)