Galaxy Pulsar Runner Code Sharing

Overview

Job-runner code crosses the Galaxy/Pulsar repository boundary through four distinct channels, not the two the folk description suggests. The two commonly-cited “paths” are really two Python import paths for the same logical source — galaxy.jobs.runners.util.* and pulsar.managers.util.* — and the important fact is that both are importable inside a single Galaxy process simultaneously, because Galaxy hard-depends on a Pulsar package that ships its own copy of the mirror. They have drifted: 20 of 23 shared files differ.

Verified against Galaxy origin/dev @ 08d4c8af5d and Pulsar origin/master @ 8e9d162953, on 2026-08-17.

Scope: the sharing mechanics — channels, sync procedure and drift, status-vocabulary translation, CI coupling. Not a Pulsar deployment guide, and not Pulsar’s runtime architecture (AMQP, staging, coexecution) except where it explains a seam.

Channel inventory

Channel 1 is the vendored mirror. Both __init__.py files carry the identical docstring — “This module should contain functionality shared between Galaxy and the Pulsar” — and no script, Make target, or CI check performs or verifies the sync.

Channel 2 makes Pulsar’s client mandatory for every Galaxy install: pyproject.toml:75 lists pulsar-galaxy-lib>=0.15.15 in the main dependency list, pinned to ==0.15.15 in lib/galaxy/dependencies/pinned-requirements.txt:189. See Component - Backend Dependency Management for the pinning machinery and Component - Worktree Bootstrapping for how it lands in a dev environment.

One codebase produces two PyPI identities. pulsar/setup.py:44 switches the package name on an env var (setup.py:12-13), and setup.py:29-30 strips every galaxy-* requirement when building the lib variant — a manual dependency-cycle break guarded by a single list comprehension with no test asserting it. Crucially setup.py:55-80 ships the same package list either way, including pulsar.managers.util and its subpackages. That is the root of the two-paths problem: pulsar-galaxy-lib delivers Pulsar’s entire copy of the mirror into every Galaxy environment.

Channel 3 runs the other way — Pulsar’s requirements.txt declares galaxy-job-metrics>=21.9.0, galaxy-objectstore>=19.9.0, galaxy-tool-util>=19.9.0, galaxy-util>=23.0. Floors of 19.9.0 in a repo read in 2026. The objectstore coupling connects to Component - Private Object Stores.

Channel 4 is Galaxy code executed on the remote node: lib/galaxy/tools/remote_tool_eval.py (see Component - YAML Tool Runtime), lib/galaxy/metadata/set_metadata.py, and lib/galaxy/job_execution/container_monitor.py. remote_tool_eval.py:11-33 imports across a wide surface of Galaxy internals.

Galaxy does have a deliberate seam here, which is more interesting than the absence of one would be. The entry points actually invoked on the Pulsar node live in lib/galaxy_ext/ — metadata/set_metadata.py, container_monitor/monitor.py, expressions/handle_job.py — a package whose __init__.py states the intent directly: things “loaded from outside Galaxy” and kept clear of the galaxy namespace, “which may be provided by other packages”. The libraries above are what those entry points call.

What is unconstrained is the version. lib/galaxy/jobs/command_factory.py:212-213 emits a command that runs remote_tool_eval.py out of $GALAXY_LIB, and $GALAXY_LIB comes from the Pulsar side — derived from the galaxy_home manager option or the GALAXY_HOME env var (pulsar/managers/base/__init__.py:138-148), prepended to PYTHONPATH by DEFAULT_JOB_FILE_TEMPLATE.sh, and read back by Galaxy at runners/pulsar.py:1066-1082. Pulsar’s extras_require['galaxy_extended_metadata'] governs a different path — metadata work Pulsar performs itself — and supplies none of galaxy.tools, galaxy.model.store, or galaxy.datatypes. So the remote Galaxy is whatever full checkout the Pulsar operator pointed galaxy_home at, constrained by nothing: no floor, no pin, no check.

Why the sync must be manual

The design is deliberate. pulsar/managers/util/cli/__init__.py:21-23 discovers plugins relative to whichever package root loaded the class:

module_prefix = self.__module__
self.cli_shells = plugins_dict(f"{module_prefix}.shell", "__name__")

The same source therefore works unmodified under either root. That path-agnosticism is what makes verbatim copying viable — and simultaneously removes the natural seam for factoring the code into one shared package.

Two files resolve the dual path at runtime, and they disagree with each other:

• util/cli/factory.py:1-10 tries galaxy.jobs.runners.util.cli and falls back to pulsar.managers.util.cli on ImportError. This file is byte-identical in both repos — written to be copied verbatim and resolve at load time.
• util/cli/shell/rsh.py:5 does from pulsar.managers.util.retry import RetryActionExecutor — unconditional, no fallback. Galaxy has no retry.py of its own. A file inside Galaxy’s vendored copy of the shared tree reaches back into the Pulsar package; the vendored tree is not self-contained. It resolves only because Channel 2 ships pulsar.managers.util.

Two files in the same directory take opposite positions on whether Pulsar is optional. That inconsistency is the sharpest symptom of the mess.

Full cross-repo import inventory

Grepping Galaxy’s lib/ for pulsar imports gives ten hits, and they are not confined to the Pulsar runner:

Two consequences. First, jobs/__init__.py:37 is the strongest evidence that Pulsar is genuinely mandatory rather than an optional runner — it is imported at module scope in the code path every job takes. Second, drmaa.py:90 means Galaxy’s DRMAA runner depends on pulsar.managers.util.drmaa, a directory that exists only in Pulsar’s copy of the mirror. Together with retry.py, that makes two of the seven “Pulsar-only” files actually load-bearing for Galaxy — the mirror is not two parallel copies so much as one copy plus a partial one that reaches across for the rest.

Drift inventory

Every path present in both trees was diffed. 23 shared files, 3 identical, 20 diverged.

Identical: cli/factory.py, cli/job/slurm_torque.py, external.py.

Behavioural divergences (not cosmetic):

Files unique to one side (5 Galaxy, 7 Pulsar) partly show independent cloud-batch development — Galaxy grew fork_safe_write.py and gcp_batch/; Pulsar grew aws_batch.py, cvmfsexec.py, gcp_util.py, tes.py. But “unique to Pulsar” does not mean “unused by Galaxy”: retry.py and drmaa/ are both Pulsar-only and imported by Galaxy (see above). The seventh, job_script/MEMORY_STATEMENT.sh, is the sharpest exhibit in the whole note — see below.

Three factors are visible in the diffs. The first plausibly masks the others, though that ordering is a hypothesis, not a measurement. Formatter/linter skew (blank lines, import ordering, docstring style) inflates every diff until a diff -r between the trees is unreadable without a normalisation pass. Underneath sit Python-version skew (Galaxy adopted walrus operators and inline annotations, Pulsar did not) and genuine feature divergence.

Job status translation

Pulsar’s vocabulary lives in pulsar/managers/status.py as module-level string constants, opening with # TODO: Make objects. — preprocessing, queued, running, complete, cancelled, failed, postprocessing, lost, plus is_job_done().

Galaxy consumes it in lib/galaxy/jobs/runners/pulsar.py:362-388, _update_job_state_for_status. Three findings:

1. Stringly-typed across the repo boundary. Galaxy compares against bare literals (["complete", "cancelled"], ["failed", "lost"], "running") and never imports pulsar.managers.status — despite hard-depending on the package that ships it. The shared enum exists, is reachable, and is unused. A rename on the Pulsar side is a silent runtime failure, not an ImportError.
2. Three statuses handled only by omission. preprocessing, queued, and postprocessing match no branch and fall through to “keep watching”. Correct today, but a new Pulsar status is indistinguishable from a transient one.
3. The STOPPED interjection is the compensating hack. Mid-handler, Galaxy consults its own database state rather than the incoming Pulsar status, and if the job was stopped Galaxy-side it calls client.kill(). Galaxy-initiated stop has no representation in Pulsar’s vocabulary, so reconciliation is bolted into the status handler. Its placement — complete/cancelled at :369-371, the STOPPED check at :372-376, failed/lost at :377-384 — is load-bearing and undocumented.

Structurally this is the same failure mode as Problem - YAML Tool Post-Hoc State Divergence: two sources of truth that are supposed to agree, with no test proving they do.

The sync was declared dead in 2015 and kept happening

From pulsar/HISTORY.rst, with release headers resolved:

The MEMORY_STATEMENT.sh exhibit

The last recorded sync illustrates the failure mode better than any row in the drift table. Both repos define MEMORY_STATEMENT_DEFAULT_TEMPLATE on line 27 of job_script/__init__.py, and load different filenames:

• Galaxy: resource_string(__name__, "MEMORY_STATEMENT_TEMPLATE.sh")
• Pulsar: resource_string(__name__, "MEMORY_STATEMENT.sh")

Pulsar ships both files; its MEMORY_STATEMENT_TEMPLATE.sh is loaded by nothing. Meanwhile Galaxy commit 5dde30a9db (2022-10-12), titled “Synchronize job script module with Pulsar fixes.”, deleted MEMORY_STATEMENT.sh from Galaxy and added MEMORY_STATEMENT_TEMPLATE.sh in its place. Pulsar’s 0.15.0 entry six months later (2023-04-13) reads “Re-import MEMORY_STATEMENT.sh from Galaxy” — importing, under that name, a file Galaxy no longer had.

That commit title is also independent evidence for the “bidirectional” label on Channel 1: Galaxy syncs from Pulsar as well.

Timeline reading

The 0.6.0 entry announces the retirement of the manual sync. Every subsequent entry is a manual sync. The packaging route never covered the runner util tree — it covered galaxy.util, galaxy.tool_util, and friends. The mirror survived the migration meant to eliminate it, without an owner.

Last recorded sync: 2023-04-13, roughly three years and four months before this analysis — consistent with the 21-of-24 drift. The scare quotes in Sync “recent” galaxy runner util changes suggest the 2019 pass was already understood to be partial.

CI coupling is one-directional

Galaxy breakage is caught — in Pulsar’s CI. pulsar/.github/workflows/galaxy_framework.yaml checks out galaxyproject/galaxy at both dev and master, builds a Pulsar wheel, swaps it into Galaxy’s pins via tools/replace_galaxy_requirements_for_ci.py, and runs Galaxy’s own framework suite against it across directory and extended metadata strategies. The catch happens on Pulsar’s PRs, not Galaxy’s. See Component - Workflow Testing for the suite being driven.

Galaxy → Pulsar is disabled. test_galaxy_packages_for_pulsar.yaml:20-21 carries if: false with the comment “disabled because it is currently redundant with test_galaxy_packages.yaml”. That claim is half right:

• Package coverage is genuinely subsumed. packages_for_pulsar_by_dep_dag.txt (tool_util_models, util, job_metrics, objectstore, tool_util) is a strict subset of the main list.
• Marker coverage differs, but is covered elsewhere. packages/test.sh:93-97 shows --for-pulsar drops the -m 'not external_dependency_management' exclusion, so only that variant ran the marked mulled/conda tests in test/unit/tool_util/mulled/ (see Component - Tool Testing Infrastructure). Those tests are not orphaned, however: .github/workflows/mulled.yaml is live and runs tox -e mulled, and tox.ini:52 sets mulled: marker=external_dependency_management — i.e. a dedicated workflow runs exactly that marked set on every push and PR. tox.ini:57 gives the plain unit env the mirror-image marker=not external_dependency_management.

So the “redundant” justification largely holds, and disabling the job did not open the coverage hole it first appears to. The one residual difference is narrow and unverified: --for-pulsar ran the marked tests inside each package’s isolated throwaway venv, installed from that package’s own pyproject.toml, whereas mulled.yaml runs them against a full Galaxy environment. Only the former would catch a missing or wrong dependency declaration in, say, galaxy-tool-util’s own metadata. Sharpening rather than blunting the point: mulled.yaml carries paths-ignore: packages/**, so changes under packages/ do not even trigger it — nothing now exercises the packaged mulled path. Whether that has ever caught a real bug is unknown; this is inference from reading the two harnesses, not a demonstrated regression.

Galaxy-side integration coverage exists — eight test/integration/test_pulsar_embedded*.py modules plus job-conf fixtures — but these exercise the embedded Pulsar path using the in-process manager, so they do not test the vendored-mirror drift or the two-import-path resolution.

Seams for de-messing

Ordered by cost, honest where there is no clean answer:

1. Import Pulsar’s status constants instead of literals. Galaxy already hard-depends on the package defining them. A few-line change converting silent drift into an ImportError. Smallest and most concrete item here.
2. Normalise formatting across both trees in one no-op commit. Removes the noise hiding the real divergence; a precondition for anything below.
3. Resolve the rsh.py / factory.py inconsistency. Either give Galaxy its own retry.py, or adopt the try/except pattern. Currently the directory contradicts itself.
4. Extract the mirror into a real package — the obvious answer, and there is no clean seam. The self.__module__-relative plugin discovery is designed around living under two roots; a shared package would have one root and would need that indirection removed, plus entry in the Pulsar package DAG and survival of the setup.py cycle-break. Real work, not a refactor.
5. A drift check in CI. The cheapest durable guard: diff the two trees and fail on unexpected changes with an allow-list for intentional divergence. Cannot live in Galaxy’s CI alone (it needs both checkouts) — Pulsar’s galaxy_framework.yaml already checks out both and is the natural host.

Notes

Job destination routing that selects the Pulsar destinations this runner services is covered in PR 20936 - Resource Requirements via TPV. Post-job processing on the Galaxy side of the same lifecycle is Component - Post Job Actions.

The relevant PRs for the sync history are Pulsar PRs (177, 297, 303), not Galaxy PRs, so related_prs is left empty rather than polluted with numbers that would resolve against the wrong repository.