Plan: CWL Docker-Required Conformance Tests as Integration Tests

Problem

Some CWL conformance tests require Docker to function but Galaxy’s conformance test CI runs without docker_enabled in the job destination. These tests are permanently red — not because Galaxy’s CWL implementation is wrong, but because the test infrastructure doesn’t provide Docker.

Specifically, docker_entrypoint and dockeroutputdir across all three CWL versions can never pass without Docker:

docker_entrypoint: baseCommand: ["-c", "echo 'moo' > cow"] relies on the bash:4.4.12 image’s entrypoint providing bash. Without Docker, -c is executed as a bare command (exit 127).
dockeroutputdir: uses dockerOutputDirectory to control where outputs land inside the container. Meaningless without Docker.

Note: ~28 other conformance tests are tagged docker but pass without Docker because their baseCommand is something normal (e.g. echo, cat) and Docker is incidental. We do NOT touch those.

Approach

Introduce a DOCKER_REQUIRED list in the conformance test generator. Tests in this list are:

Skipped in the generated API conformance tests (with a reason pointing to the integration test)
Generated as explicit methods in a new integration test file that configures Galaxy with docker_enabled: true

This runs as part of the normal integration test suite — no separate CI workflow needed.

Changes

1. `scripts/cwl_conformance_to_test_cases.py`

Add `DOCKER_REQUIRED` dict

DOCKER_REQUIRED = {
    "v1.0": [
        "docker_entrypoint",
        "dockeroutputdir",
    ],
    "v1.1": [
        "docker_entrypoint",
        "dockeroutputdir",
    ],
    "v1.2": [
        "docker_entrypoint",
        "dockeroutputdir",
    ],
}

Remove from `RED_TESTS`

Remove docker_entrypoint and dockeroutputdir from RED_TESTS for all three versions. They’re not red — they’re structurally incompatible with the non-Docker test environment.

Modify conformance test generation

For tests in DOCKER_REQUIRED, emit a skip marker instead of the normal test body:

    @pytest.mark.skip(reason="Requires Docker job destination; tested in test/integration/test_containerized_cwl_conformance.py")
    def test_conformance_v1_0_docker_entrypoint(self):
        ...

Generate integration test file

After writing the conformance test files, generate test/integration/test_containerized_cwl_conformance.py.

During iteration, collect (version, version_simple, id_, doc) tuples for all DOCKER_REQUIRED tests. After the main loop, write the integration test file using a template.

The template for the integration test file:

"""Integration tests for CWL conformance tests that require Docker.

Generated by scripts/cwl_conformance_to_test_cases.py — do not edit manually.
"""

import os
import shutil
import unittest

from galaxy_test.base.populators import (
    CwlPopulator,
    DatasetPopulator,
    WorkflowPopulator,
)
from galaxy_test.driver.integration_util import IntegrationTestCase

CWL_TOOL_DIRECTORY = os.path.join(
    os.path.dirname(__file__),
    os.pardir,
    os.pardir,
    "test",
    "functional",
    "tools",
    "cwl_tools",
)


def _skip_unless_conformance_data():
    """Skip if CWL conformance test data has not been downloaded."""
    if not os.path.isdir(os.path.join(CWL_TOOL_DIRECTORY, "v1.0")):
        raise unittest.SkipTest(
            "CWL conformance test data not found; run: make generate-cwl-conformance-tests"
        )


class TestContainerizedCwlConformance(IntegrationTestCase):
    """CWL conformance tests that require a Docker-enabled job destination."""

    require_admin_user = True
    framework_tool_and_types = True
    container_type = "docker"

    @classmethod
    def handle_galaxy_config_kwds(cls, config):
        super().handle_galaxy_config_kwds(config)
        config["job_config"] = {
            "runners": {
                "local": {
                    "load": "galaxy.jobs.runners.local:LocalJobRunner",
                    "workers": 1,
                }
            },
            "execution": {
                "default": "local_docker",
                "environments": {
                    "local_docker": {
                        "runner": "local",
                        "docker_enabled": True,
                    },
                    "local_upload": {
                        "runner": "local",
                    },
                },
            },
            "tools": [
                {"id": "upload1", "environment": "local_upload"},
            ],
        }
        config["enable_beta_tool_formats"] = True
        config["enable_beta_workflow_modules"] = True
        config["conda_auto_init"] = False
        config["conda_auto_install"] = False
        config["tool_dependency_dir"] = "none"

    @classmethod
    def setUpClass(cls):
        if not shutil.which("docker"):
            raise unittest.SkipTest("docker not found on PATH")
        _skip_unless_conformance_data()
        super().setUpClass()

    def setUp(self):
        super().setUp()
        self.dataset_populator = DatasetPopulator(self.galaxy_interactor)
        self.workflow_populator = WorkflowPopulator(self.galaxy_interactor)
        self.cwl_populator = CwlPopulator(
            self.dataset_populator, self.workflow_populator
        )

    # --- generated test methods below ---
    $tests

Each test method:

    def test_conformance_v1_0_docker_entrypoint(self):
        """Test Docker ENTRYPOINT usage"""
        self.cwl_populator.run_conformance_test("""v1.0""", """Test Docker ENTRYPOINT usage""")

The doc string is pulled from conformance_tests.yaml during generation — it’s the lookup key that CwlPopulator.run_conformance_test() uses to find the test entry.

2. `.gitignore`

Add:

test/integration/test_containerized_cwl_conformance.py

This file is generated from downloaded (gitignored) conformance data, so it must also be gitignored.

3. Regenerate all test files

bash scripts/update_cwl_conformance_tests.sh
# or
make generate-cwl-conformance-tests

This will:

Download conformance test data (if needed)
Regenerate lib/galaxy_test/api/cwl/test_cwl_conformance_v1_?.py (with skips)
Generate test/integration/test_containerized_cwl_conformance.py (new)

4. No CI workflow changes

The conformance workflow (cwl_conformance.yaml) is unchanged. The docker-required tests are now @pytest.mark.skip so they silently pass in conformance runs.
The integration test runs wherever ./run_tests.sh -integration runs. Docker is available on GitHub Actions runners. If Docker is missing, setUpClass skips.
If conformance test data hasn’t been downloaded, setUpClass skips with a message.

Implementation order

Edit scripts/cwl_conformance_to_test_cases.py:
- Add DOCKER_REQUIRED dict
- Remove entries from RED_TESTS
- Add skip markers for DOCKER_REQUIRED in conformance test generation
- Add integration test file generation at end of main()
Add gitignore entry
Run make generate-cwl-conformance-tests (or manual invocations)

Run the integration test locally to verify:

. .venv/bin/activate
GALAXY_CONFIG_OVERRIDE_CONDA_AUTO_INIT=false \
pytest -s -v test/integration/test_containerized_cwl_conformance.py

Commit

Testing

Red-to-green: docker_entrypoint and dockeroutputdir currently fail in conformance CI (red, continue-on-error). After this change they are skipped in conformance and tested properly in integration with Docker enabled. The integration test is the green proof.

Future

If more conformance tests are identified as truly Docker-required (not just Docker-tagged), add their IDs to DOCKER_REQUIRED and regenerate.

DOCKER_REQUIRED_TESTING