Galaxy Backend Dependency Management

This document describes how Galaxy declares, pins, and updates its Python backend dependencies.

Architecture Overview

Galaxy uses uv as its dependency resolver and lockfile manager. Dependencies are declared in pyproject.toml, resolved into a uv.lock lockfile, and then exported to a set of flat pinned-*.txt files under lib/galaxy/dependencies/. These pinned files are what Galaxy actually installs from at runtime.

pyproject.toml                          # source of truth for all deps
    |
    v
uv lock --upgrade                       # resolves everything
    |
    v
uv.lock                                 # machine-managed lockfile
    |
    v
uv export --frozen ...                  # exports per-group pinned files
    |
    +-> lib/galaxy/dependencies/pinned-requirements.txt       (production)
    +-> lib/galaxy/dependencies/pinned-test-requirements.txt   (test)
    +-> lib/galaxy/dependencies/dev-requirements.txt           (dev, includes test)
    +-> lib/galaxy/dependencies/pinned-typecheck-requirements.txt (mypy stubs)

Dependency Declaration (pyproject.toml)

Production dependencies

Listed under [project].dependencies. These are the packages required to run a Galaxy server. Examples include SQLAlchemy, celery, fastapi, pydantic, cwltool, etc. Version constraints are specified inline with comments explaining why:

"SQLAlchemy>=2.0.37,<2.1,!=2.0.41",  # issue links
"celery>=5.4.0",  # prefer not downgrading this to upgrading typing-extensions

Dependency groups ([dependency-groups])

uv’s dependency-groups feature organizes non-production dependencies:

The dev group uses {include-group = "test"} to include all test deps.

Resolver configuration ([tool.uv])

[tool.uv]
constraint-dependencies = [...]     # global version constraints
default-groups = []                 # don't install any groups by default
extra-index-url = ["https://wheels.galaxyproject.org/simple"]
index-strategy = "unsafe-best-match"
package = false

Key settings:

• extra-index-url: Galaxy hosts its own wheel index at wheels.galaxyproject.org for packages not available on PyPI or that need special builds.
• index-strategy = "unsafe-best-match": Allows uv to pick the best matching version across all indexes, not just the first one that has the package.
• constraint-dependencies: Imposes additional version constraints during resolution without adding new dependencies.
• package = false: Galaxy itself is not installable as a package; uv only manages its dependencies.

Conditional Dependencies

Galaxy has many optional integrations (LDAP auth, S3 object stores, DRMAA job runners, etc.) that require additional packages. These are handled by a config-aware system at startup rather than through Python extras.

How it works

1. lib/galaxy/dependencies/conditional-requirements.txt lists all optional packages (psycopg2-binary, drmaa, boto3, various filesystem plugins, etc.)

2. lib/galaxy/dependencies/__init__.py defines ConditionalDependencies, which:

   • Parses Galaxy’s configuration files (galaxy.yml, job_conf.yml, object_store_conf.xml, auth_config.xml, etc.)
   • Extracts configured job runners, object stores, authenticators, file sources, vault backends, error reporters
   • For each optional package, has a check_<name>() method that returns True if the config requires it

3. scripts/common_startup.sh calls this logic during Galaxy startup:

   GALAXY_CONDITIONAL_DEPENDENCIES=$(python -c "import galaxy.dependencies; ...")
   echo "$GALAXY_CONDITIONAL_DEPENDENCIES" | pip install -r /dev/stdin \
       --constraint lib/galaxy/dependencies/conditional-constraints.txt

4. lib/galaxy/dependencies/conditional-constraints.txt provides platform-specific constraints for conditional deps (currently used for zeroc-ice wheel URLs needed by the OMERO file source plugin).

Examples of conditional checks

Lint Dependencies (Separate Pipeline)

Linting tools (flake8, ruff) often have dependency conflicts with Galaxy’s main dependency tree. They are managed separately:

• lib/galaxy/dependencies/lint-requirements.txt: Unpinned top-level lint deps (flake8, flake8-bugbear, ruff)
• lib/galaxy/dependencies/pinned-lint-requirements.txt: Pinned output

The update_lint_requirements.sh script resolves these in an isolated virtualenv (using Python 3.9) and freezes the result. This runs as part of make update-dependencies (the update-lint-requirements target runs first).

The requirements.txt Symlink

At the repository root, requirements.txt is a symlink to lib/galaxy/dependencies/pinned-requirements.txt. This is what common_startup.sh installs via pip install -r requirements.txt. The symlink provides a conventional entry point while keeping the actual file co-located with the rest of the dependency management code.

The Update Pipeline

make update-dependencies

The top-level Makefile target update-dependencies orchestrates the full update:

update-dependencies: update-lint-requirements
	$(IN_VENV) ./lib/galaxy/dependencies/update.sh

It first updates lint deps, then runs update.sh.

lib/galaxy/dependencies/update.sh

This script:

1. Ensures uv is available - uses the system uv if present, otherwise creates a temporary venv and pip-installs it
2. Resolves all dependencies - runs uv lock --upgrade which re-resolves the entire dependency tree to the latest compatible versions, updating uv.lock
3. Exports pinned files - runs uv export four times with different group selectors to produce each pinned requirements file:

UV_EXPORT_OPTIONS='--frozen --no-annotate --no-hashes'
uv export ${UV_EXPORT_OPTIONS} --no-dev > pinned-requirements.txt
uv export ${UV_EXPORT_OPTIONS} --only-group=test > pinned-test-requirements.txt
uv export ${UV_EXPORT_OPTIONS} --only-group=dev > dev-requirements.txt
uv export ${UV_EXPORT_OPTIONS} --only-group=typecheck > pinned-typecheck-requirements.txt

The --frozen flag means uv reads from the lockfile without re-resolving. --no-annotate omits the “via X” comments. --no-hashes omits integrity hashes.

Single-package updates

The script accepts -p <pkg> to upgrade a single package:

./lib/galaxy/dependencies/update.sh -p sqlalchemy

This runs uv lock --upgrade-package <pkg> instead of uv lock --upgrade, leaving other packages at their current versions.

Automated Weekly Updates (CI)

.github/workflows/dependencies.yaml

A GitHub Actions workflow runs every Saturday at 3:00 AM UTC (cron: 0 3 * * 6). It can also be triggered manually via workflow_dispatch.

The workflow:

1. Checks out the repository
2. Sets up Python 3.10 and installs uv
3. Runs make update-dependencies
4. Uses peter-evans/create-pull-request to:
   • Create a commit authored by galaxybot
   • Push to a fork (galaxybot/galaxy)
   • Open a PR titled “Update Python dependencies” on branch dev_auto_update_dependencies
   • Label the PR with area/dependencies and kind/enhancement
   • Delete and recreate the branch each week (so only one outstanding update PR exists at a time)

This gives maintainers a weekly PR to review, test through CI, and merge.

Runtime Installation (common_startup.sh)

When Galaxy starts (or when a developer runs common_startup.sh):

1. A virtualenv is created if needed (preferring uv venv if available, falling back to python -m venv)
2. pip install -r requirements.txt (the symlink to pinned-requirements.txt) installs all production deps at exact versions
3. If --dev-wheels is passed, dev-requirements.txt is also installed
4. Galaxy config files are parsed and conditional dependencies are installed with constraints from conditional-constraints.txt
5. The Galaxy client (JS/pnpm) is built if needed

File Summary