Galaxy Workflow Testing: A Comprehensive Guide

Table of Contents

1. Overview
2. YAML-Based Workflow Framework Tests
3. Procedural API Workflow Tests
4. Shared Infrastructure: Populators and Fixtures
5. Integration Workflow Tests
6. Additional Workflow Test Suites
7. How to Run Tests
8. Comparison of Approaches
9. Key Files and Directories

Overview

Galaxy has two primary approaches to testing workflow execution correctness, plus several supplementary test suites. The two core approaches are:

1. YAML-based workflow framework tests — Declarative test definitions pairing workflow files (.gxwf.yml) with test specifications (.gxwf-tests.yml). These live in lib/galaxy_test/workflow/ and are the workflow analogue of Galaxy’s tool framework tests in test/functional/tools/.

2. Procedural API tests — Python test methods in lib/galaxy_test/api/test_workflows.py (and related files) that use the Galaxy API to imperatively create workflows, upload data, invoke workflows, and assert on results.

Both approaches ultimately drive a running Galaxy test server through its API, using a shared populator infrastructure (WorkflowPopulator, DatasetPopulator, DatasetCollectionPopulator) to abstract common operations. The intent documented in lib/galaxy_test/workflow/__init__.py is explicit:

This is meant to grow into the workflow based mirror of what the framework tests are for tools. api/test_workflows.py is still the place to test exceptional conditions, errors, etc… but tests of normal operation where semantics can be verified with simple inputs and outputs can now be placed in here.

YAML-Based Workflow Framework Tests

Architecture

The framework test system consists of three layers:

1. Workflow definitions — Standard Galaxy Format2 YAML workflow files with .gxwf.yml extension.
2. Test specifications — Companion YAML files with .gxwf-tests.yml extension that define inputs (jobs) and expected outputs.
3. Test runner — A single Python module test_framework_workflows.py that uses pytest parametrization to discover and execute all workflow/test pairs.

File Convention

For a workflow named flatten_collection, the files are:

lib/galaxy_test/workflow/
  flatten_collection.gxwf.yml           # The workflow definition
  flatten_collection.gxwf-tests.yml     # One or more test cases

The naming convention is strict — the test runner uses glob.glob("*.gxwf.yml") to discover workflows and derives the test file name by replacing .gxwf.yml with .gxwf-tests.yml.

Current Workflow Test Files (24 workflows)

default_values                          directory_index
default_values_optional                 empty_collection_sort
filter_null                             flatten_collection
flatten_collection_over_execution       integer_into_data_column
map_over_expression                     multi_select_mapping
multiple_integer_into_data_column       multiple_text
multiple_versions                       optional_conditional_inputs_to_build_list
optional_text_param_rescheduling        output_parameter
rename_based_on_input_collection        replacement_parameters_legacy
replacement_parameters_nested           replacement_parameters_text
subcollection_rank_sorting              subcollection_rank_sorting_paired
triply_nested_list_mapping              zip_collection

Workflow Definition Format

Workflow files use Galaxy’s Format2 YAML workflow syntax:

class: GalaxyWorkflow
inputs:
  required_int_with_default:
    type: int
    default: 1
outputs:
  out:
    outputSource: integer_default/out_file1
steps:
  integer_default:
    tool_id: integer_default
    tool_state:
      input1: 0
      input2: 0
    in:
      input3:
        source: required_int_with_default

Key elements:

• class: GalaxyWorkflow identifies Format2 workflows
• inputs defines workflow inputs with types (data, int, text, boolean, collection)
• outputs names workflow outputs with outputSource referencing step/output pairs
• steps defines tool invocations with tool_id, state/tool_state, and in connections
• Step connections use step_label/output_name syntax or positional $link references

Test Specification Format

Test files contain a YAML list of test cases. Each test case has:

- doc: |
    Human-readable description of what this test verifies.
  job:                          # Inputs to the workflow
    input_name:
      type: File                # File, raw, or collection
      value: test_file.bed      # Reference to test-data file
      content: "inline content" # Or inline content
  outputs:                      # Expected outputs
    output_name:
      class: File               # File or Collection
      asserts:
        - that: has_text
          text: "expected content"
  expect_failure: true          # Optional: test expects workflow to fail

Job Input Types

Simple string values — Uploaded as a dataset with the string as content:

job:
  my_input: "hello world"

File references — Reference files from Galaxy’s test-data/ directory:

job:
  my_input:
    type: File
    value: 1.bed
    file_type: bed

Raw parameter values — Non-dataset parameters (text, integer, boolean):

job:
  my_param:
    type: raw
    value: 42

Null values:

job:
  my_param:
    type: raw
    value: null

Collection inputs:

job:
  my_collection:
    type: collection
    collection_type: list
    elements:
      - identifier: el1
        content: "data1"
      - identifier: el2
        content: "data2"

Collections with specific extensions:

job:
  my_collection:
    type: collection
    collection_type: list
    elements:
      - identifier: item1
        content: '"ex2"'
        ext: 'expression.json'

Empty jobs — For workflows that need no external inputs:

job: {}

Output Assertions

Simple file content checks:

outputs:
  out:
    class: File
    asserts:
      - that: has_text
        text: "expected"
      - that: has_line
        line: "exact line match"

Metadata checks:

outputs:
  out:
    class: File
    metadata:
      name: 'expected_name'

Exact parameter output values:

outputs:
  out_int: 43

Collection outputs with element-level assertions:

outputs:
  out:
    class: Collection
    collection_type: list
    elements:
      'element_id_1':
        asserts:
          - that: has_text
            text: "A"
      'element_id_2':
        asserts:
          - that: has_text
            text: "B"

Nested collection outputs:

outputs:
  out:
    class: Collection
    collection_type: list:list
    elements:
      outer1:
        elements:
          inner1:
            asserts:
              - that: has_text
                text: "content"

Expected failure — Test that a workflow invocation fails:

- doc: Test that bad input causes failure
  expect_failure: true
  job:
    my_input:
      type: raw
      value: null
  outputs: {}

Test Discovery and Execution

The test runner in lib/galaxy_test/workflow/test_framework_workflows.py works as follows:

1. Discovery via pytest_generate_tests: A pytest hook function globs for *.gxwf.yml files in the script directory, parses each companion .gxwf-tests.yml file, and generates parametrized test cases. Each test gets an ID like flatten_collection_0, default_values_1, etc.

2. Server setup via conftest.py: A session-scoped fixture creates a GalaxyTestDriver that starts a Galaxy server with framework_tool_and_types = True (loading the sample tool configuration so test tools like cat1, random_lines1, collection_creates_pair, etc. are available).

3. Test execution in TestWorkflow.test_workflow: Each parametrized test case:

   • Loads the workflow YAML via gxformat2’s ordered_load
   • Creates a test history via dataset_populator.test_history()
   • Calls workflow_populator.run_workflow() with the workflow content and test job inputs
   • Verifies each declared output using _verify_output(), which handles both dataset and collection outputs
   • Supports expect_failure: true tests that assert the workflow fails

4. Rerun testing: The environment variable GALAXY_TEST_WORKFLOW_AFTER_RERUN (set to 1 in CI) causes each workflow to be rerun after initial execution, verifying that rerun produces the same results. This is accomplished via workflow_populator.rerun().

Verification Mechanism

Output verification in _verify_output delegates to Galaxy’s tool test verification infrastructure:

• Dataset outputs: Downloads the output content and calls verify_file_contents_against_dict() from galaxy.tool_util.verify, plus compare_expected_metadata_to_api_response() for metadata assertions.
• Collection outputs: Uses verify_collection() from galaxy.tool_util.verify.interactor, which recursively walks collection elements and applies per-element assertions.

This reuse of the tool verification layer means workflow framework tests support the same assertion vocabulary as tool tests (has_text, has_line, has_n_lines, has_text_matching, metadata checks, etc.).

Procedural API Workflow Tests

Architecture

The procedural API tests live in lib/galaxy_test/api/test_workflows.py — an ~8,850 line file containing approximately 290 test methods organized into several classes.

Class Hierarchy

FunctionalTestCase (lib/galaxy_test/base/testcase.py)
  -> ApiTestCase (lib/galaxy_test/api/_framework.py)
       -> BaseWorkflowsApiTestCase (test_workflows.py)
            -> TestWorkflowsApi (test_workflows.py) -- the main test class

Plus mixin classes:

• RunsWorkflowFixtures — reusable methods for running specific workflow patterns
• ChangeDatatypeTests — tests for datatype-related PJAs
• SharingApiTests — tests for workflow sharing via TestWorkflowSharingApi

Key Base Class: BaseWorkflowsApiTestCase

This class (defined in test_workflows.py at line 178) sets up the three core populators:

class BaseWorkflowsApiTestCase(ApiTestCase, RunsWorkflowFixtures):
    def setUp(self):
        super().setUp()
        self.workflow_populator = WorkflowPopulator(self.galaxy_interactor)
        self.dataset_populator = DatasetPopulator(self.galaxy_interactor)
        self.dataset_collection_populator = DatasetCollectionPopulator(self.galaxy_interactor)

And provides convenience wrappers:

• _upload_yaml_workflow() — uploads a Format2 YAML workflow string
• _run_jobs() / _run_workflow() — high-level workflow invocation + wait
• _setup_workflow_run() — creates workflow, datasets, and builds a request dict
• _invocation_details() — fetches invocation details
• _download_workflow() — downloads workflow in various formats
• import_workflow() — imports a native (.ga) format workflow

Test Patterns

The procedural tests use several common patterns:

Pattern 1: Inline YAML workflow + test_data string

The most common pattern. Workflows are defined as inline YAML strings (often imported from workflow_fixtures.py) and test data is passed as a YAML string or dict:

def test_run_subworkflow_simple(self) -> None:
    with self.dataset_populator.test_history() as history_id:
        summary = self._run_workflow(
            WORKFLOW_NESTED_SIMPLE,
            test_data="""
outer_input:
  value: 1.bed
  type: File
""",
            history_id=history_id,
        )
        content = self.dataset_populator.get_history_dataset_content(history_id)
        assert content == "expected..."

Pattern 2: Load native .ga workflow + manual setup

For testing native format workflow features:

def test_run_workflow(self):
    workflow = self.workflow_populator.load_workflow(name="test_for_run")
    workflow_request, history_id, workflow_id = self._setup_workflow_run(workflow)
    # ... invoke and assert

Pattern 3: Upload workflow then invoke separately

When testing invocation-specific behavior (pausing, canceling, scheduling):

def test_workflow_pause(self):
    workflow_id = self._upload_yaml_workflow("""
class: GalaxyWorkflow
steps:
  the_pause:
    type: pause
    ...
""")
    with self.dataset_populator.test_history() as history_id:
        invocation_id = self.__invoke_workflow(workflow_id, history_id=history_id)
        self._wait_for_invocation_non_new(workflow_id, invocation_id)
        # ... interact with pause step, verify state

Pattern 4: Testing error conditions and API responses

def test_cannot_run_inaccessible_workflow(self):
    workflow_id = self.workflow_populator.simple_workflow("test_for_run")
    with self._different_user():
        # Expect 403 when trying to invoke another user's workflow
        ...

Pattern 5: skip_without_tool decorator

Many tests depend on specific Galaxy tools being available:

@skip_without_tool("cat1")
def test_run_workflow(self):
    ...

This decorator checks the Galaxy server’s tool registry and skips the test if the required tool is not installed.

Categories of Tests in test_workflows.py

The ~290 tests cover these categories:

1. Workflow CRUD (~30 tests): show, delete, undelete, index, search, filtering, ordering, tagging, publishing, sharing

2. Import/Export (~25 tests): upload native/.ga format, Format2, URL import, TRS import (Dockstore, WorkflowHub), base64 import, export in various styles, round-trip format conversion

3. Basic Execution (~15 tests): run by step_id, step_index, UUID, name; URL inputs; deferred inputs; cached jobs; hash validation

4. Collection Handling (~40 tests): output collections, mapped output collections, dynamic collections, map-over, list:paired, subcollection mapping, empty lists, cross products, nested collections, collection type sources

5. Subworkflows (~25 tests): simple subworkflow execution, runtime parameters in subworkflows, replacement parameters, auto labels, mapping recovery, subworkflow outputs as workflow outputs

6. Conditional Steps (~20 tests): when expressions, boolean conditional steps, conditional subworkflows, map-over with conditionals, skipped steps with collection outputs

7. Pausing and Cancellation (~10 tests): pause steps, cancel invocations, resume from failed steps, job deletion on cancel

8. Post-Job Actions (PJAs) (~30 tests): rename, hide, delete intermediates, change datatype, add/remove tags, runtime PJAs, PJA import/export

9. Parameter Handling (~20 tests): integer parameters, text connections, numeric connections, validated parameters, default values, replacement parameters, runtime parameters, step parameters

10. Invocation Features (~20 tests): invocation reports, BioCompute Objects, RO-Crate export, invocation filtering, job metrics, invocation from store

11. Error Handling (~15 tests): inaccessible workflows, invalid IDs, immutable histories, tool state validation, missing tools, workflow output not found

12. Advanced Features (~20 tests): batch execution, stability tests, data column parameters, implicit conversion, dynamic tools, cached job reuse, deferred datasets

Related API Test Files

• lib/galaxy_test/api/test_workflows_from_yaml.py — Tests focused on Format2 YAML workflow upload/download round-tripping and execution. Extends BaseWorkflowsApiTestCase.

• lib/galaxy_test/api/test_workflows_cwl.py — CWL (Common Workflow Language) workflow tests. Uses CwlPopulator for CWL-specific workflow/tool loading and execution.

• lib/galaxy_test/api/test_workflow_extraction.py — Tests for extracting workflows from history (the “Extract Workflow” feature). Uses BaseWorkflowsApiTestCase.

• lib/galaxy_test/api/test_workflow_build_module.py — Tests for the workflow editor’s module building API.

Shared Infrastructure

WorkflowPopulator

Defined in lib/galaxy_test/base/populators.py, WorkflowPopulator is the central utility class. It extends BaseWorkflowPopulator and provides:

Workflow Lifecycle:

• upload_yaml_workflow(yaml_content) — Converts Format2 YAML to native format and uploads
• create_workflow(workflow_dict) — Uploads a native format workflow dict
• import_workflow(workflow) — Imports via the workflows API
• import_workflow_from_path(from_path) — Imports from a server-side path
• download_workflow(workflow_id, style) — Downloads in various formats

Invocation:

• run_workflow(has_workflow, test_data, history_id, ...) — High-level: upload + populate data + invoke + wait. Returns RunJobsSummary.
• invoke_workflow_raw(workflow_id, request) — Low-level invocation
• invoke_workflow_and_assert_ok(workflow_id, ...) — Invoke and assert 200

Waiting:

• wait_for_workflow(workflow_id, invocation_id, history_id) — Wait for invocation to schedule fully and all history jobs to complete
• wait_for_invocation(workflow_id, invocation_id) — Wait for invocation state
• wait_for_history_workflows(history_id) — Wait for all invocations in a history

Inspection:

• get_invocation(invocation_id) — Fetch invocation details
• workflow_invocations(workflow_id) — List invocations for a workflow
• download_invocation_to_store() — Export invocation as model store
• get_ro_crate() — Export as RO-Crate

The run_workflow method is the most important. Its flow:

1. Upload the Format2 YAML workflow via upload_yaml_workflow()
2. Parse test_data to extract step_parameters, replacement_parameters, and input data
3. Call load_data_dict() to upload test datasets/collections into the history
4. Build the invocation request with inputs mapped by name
5. Invoke the workflow and optionally wait for completion
6. Return a RunJobsSummary named tuple

RunJobsSummary

A NamedTuple returned by run_workflow():

class RunJobsSummary(NamedTuple):
    history_id: str
    workflow_id: str
    invocation_id: str
    inputs: dict
    jobs: list
    invocation: dict
    workflow_request: dict

DatasetPopulator and DatasetCollectionPopulator

These companion classes handle dataset and collection creation, waiting, and inspection. Key methods used in workflow tests:

• new_dataset(), new_history(), test_history() (context manager)
• get_history_dataset_content(), get_history_dataset_details()
• get_history_collection_details()
• wait_for_history(), wait_for_history_jobs()

Workflow Fixtures

lib/galaxy_test/base/workflow_fixtures.py contains ~50 reusable Format2 YAML workflow definitions as Python string constants. These are imported by both the API tests and integration tests. Examples include:

• WORKFLOW_SIMPLE_CAT_AND_RANDOM_LINES — Basic multi-step workflow
• WORKFLOW_NESTED_SIMPLE — Subworkflow example
• WORKFLOW_WITH_OUTPUT_COLLECTION — Collection output workflow
• WORKFLOW_WITH_RULES_1 — Apply rules workflow
• WORKFLOW_OPTIONAL_TRUE_INPUT_DATA — Optional input handling
• WORKFLOW_FLAT_CROSS_PRODUCT — Cross product operations
• WORKFLOW_KEEP_SUCCESSFUL_DATASETS — Error filtering
• WORKFLOW_WITH_CUSTOM_REPORT_1 — Custom invocation reports

Some fixtures include embedded test_data sections:

WORKFLOW_WITH_OUTPUT_COLLECTION = """
class: GalaxyWorkflow
...
test_data:
  text_input: |
    a
    b
    c
    d
"""

Test Data Loading (load_data_dict)

The load_data_dict() function in populators.py translates test data dictionaries into actual Galaxy history contents. It handles:

• String values -> uploaded as new datasets
• File references (type: File, value: filename) -> uploaded from test-data/ directory
• Raw values (type: raw) -> passed as literal parameter values
• Collections (collection_type + elements) -> created via the collections API
• URL references (type: url) -> fetched and uploaded

Integration Workflow Tests

Integration tests in test/integration/ can access Galaxy internals and customize server configuration. Several are workflow-focused:

• test/integration/test_workflow_refactoring.py — Tests workflow refactoring operations with direct database model access. Uses IntegrationTestCase to inspect WorkflowStep, WorkflowStepConnection, and PostJobAction models.

• test/integration/test_workflow_handler_configuration.py — Tests workflow handler assignment with custom job configurations.

• test/integration/test_workflow_scheduling_options.py — Tests workflow scheduling configuration options.

• test/integration/test_workflow_tasks.py — Tests Celery-based workflow task processing.

• test/integration/test_workflow_sync.py — Tests workflow synchronization behavior.

• test/integration/test_workflow_invocation.py — Tests invocation-specific features requiring custom Galaxy configuration.

Integration tests extend IntegrationTestCase and override handle_galaxy_config_kwds(cls, config) to customize the Galaxy server. They can also use all the same populators as API tests.

Additional Workflow Test Suites

Selenium Workflow Tests

Located in lib/galaxy_test/selenium/, these test the workflow UI:

• test_workflow_editor.py — Workflow editor interactions
• test_workflow_run.py — Running workflows through the UI
• test_workflow_management.py — Workflow listing, organizing
• test_workflow_sharing.py — Sharing workflows via UI
• test_workflow_landing.py — Workflow landing pages
• test_workflow_invocation_details.py — Invocation details page
• test_workflow_rerun.py — Rerunning workflows
• test_published_workflows.py — Published workflow browsing

Performance Tests

lib/galaxy_test/performance/test_workflow_framework_performance.py benchmarks workflow execution with configurable collection sizes and workflow depths. Uses WorkflowPopulator.scaling_workflow_yaml() to generate workflows of varying complexity.

Native Format Test Workflows (.ga files)

lib/galaxy_test/base/data/ contains .ga format test workflows used by tests that exercise native format import/export and specific legacy features:

test_workflow_1.ga                    test_workflow_2.ga
test_workflow_batch.ga                test_workflow_map_reduce_pause.ga
test_workflow_matching_lists.ga       test_workflow_missing_tool.ga
test_workflow_pause.ga                test_workflow_randomlines_legacy_params.ga
test_workflow_topoambigouity.ga       test_workflow_two_random_lines.ga
test_workflow_validation_1.ga         test_workflow_with_input_tags.ga
test_workflow_with_runtime_input.ga   test_subworkflow_with_integer_input.ga
test_subworkflow_with_tags.ga

How to Run Tests

Framework Workflow Tests

# Run all workflow framework tests
./run_tests.sh --framework-workflows

# Run a specific workflow test by name
./run_tests.sh --framework-workflows -id flatten_collection

# Direct pytest invocation
pytest lib/galaxy_test/workflow/test_framework_workflows.py

# Run a specific test
pytest lib/galaxy_test/workflow/test_framework_workflows.py -k flatten_collection_0

The CI workflow (.github/workflows/framework_workflows.yaml) runs these with:

• PostgreSQL database
• GALAXY_TEST_WORKFLOW_AFTER_RERUN=1 to verify rerun correctness
• Scheduled nightly runs additionally test extended metadata strategy

API Workflow Tests

# Run all API tests (includes workflow tests among others)
./run_tests.sh -api

# Run only workflow API tests
pytest lib/galaxy_test/api/test_workflows.py

# Run a specific test method
pytest lib/galaxy_test/api/test_workflows.py::TestWorkflowsApi::test_run_workflow

# Run related workflow API test files
pytest lib/galaxy_test/api/test_workflows_from_yaml.py
pytest lib/galaxy_test/api/test_workflows_cwl.py
pytest lib/galaxy_test/api/test_workflow_extraction.py

Integration Workflow Tests

# Run all integration tests
./run_tests.sh -integration

# Run a specific integration test file
pytest test/integration/test_workflow_refactoring.py

Environment Variables

Comparison of Approaches

When to Use YAML-Based Framework Tests

Use the declarative framework tests when:

• Testing normal workflow execution semantics — correct outputs given correct inputs
• The test can be expressed as simple input/output pairs without complex assertions
• Testing collection operations (flatten, zip, filter, sort, cross product, etc.)
• Testing parameter handling (defaults, optional inputs, replacement parameters)
• Testing mapping behavior (how collections flow through tools and subworkflows)
• You want the test to automatically participate in rerun verification
• The test does not need to inspect intermediate states, invocation scheduling, or error responses

Advantages:

• Minimal boilerplate — just two YAML files
• Automatically discovered and parametrized
• Rerun testing is built in
• Assertions reuse Galaxy’s tool verification infrastructure
• Easy to read and review
• Multiple test cases per workflow (the test file is a list)

Limitations:

• Cannot test error conditions or API error responses
• Cannot inspect intermediate workflow states (paused steps, scheduling)
• Cannot test workflow CRUD operations (import, export, share, delete)
• Cannot test interactions between multiple users
• Cannot customize Galaxy server configuration

When to Use Procedural API Tests

Use the procedural API tests when:

• Testing error conditions and API responses (403, 400, validation errors)
• Testing workflow lifecycle operations (CRUD, import/export, sharing, publishing)
• Testing invocation state management (pausing, resuming, canceling)
• Testing interaction between multiple users (with self._different_user())
• Testing complex scenarios requiring multiple sequential API calls
• Testing edge cases that need fine-grained control over invocation setup
• The test needs to inspect intermediate states during workflow execution
• Testing batch execution or cached job behavior

Advantages:

• Full control over every API interaction
• Can test error paths and exceptional conditions
• Can test multi-user scenarios
• Can inspect intermediate states during execution
• Can test any workflow API feature

Limitations:

• Significant boilerplate per test
• Tests are harder to read for someone unfamiliar with the populator API
• No automatic rerun verification
• Inline workflow YAML can become hard to maintain in large test files

Summary Decision Matrix

Key Files and Directories

Framework Workflow Tests

Procedural API Tests

Shared Infrastructure

Integration Tests

CI Workflows

Other