Galaxy Brain
Dashboard

Pr 4830 Workflow Resource Parameters

Administrators define workflow-level resource parameters users set when invoking workflows for scheduling

draft research/prgalaxy/workflowsgalaxy/tools/runtime
Raw
Revised:
2026-04-22
Revision:
2
GitHub PR:
#4830
Related Notes:
PR 20936 - Resource Requirements via TPV, PR 5378 - Tool Resource Requirements

PR #4830 Research Summary: Workflow-to-Job Scheduling Parameters

  • Title: Workflow-to-Job Scheduling Parameters
  • Author: JeffreyThiessen (Jeff Thiessen), with significant refactoring by jmchilton (John Chilton)
  • URL: https://github.com/galaxyproject/galaxy/pull/4830
  • State: MERGED (2018-03-20)
  • Merged by: jmchilton
  • Base branch: dev
  • Labels: kind/feature, area/workflows

High-Level Description

This PR adds the ability for administrators to define workflow-level resource parameters (CPU, memory, walltime, priority, etc.) that users can set when invoking a workflow. These parameters are then passed through to every job in the workflow, influencing job scheduling (particularly via Dynamic Tool Destination / DTD).

Key capabilities introduced:

  1. Admin-configurable XML file defining available resource parameters (processors, memory, time, priority)
  2. YAML-based mapper config that maps user groups to allowed resource parameter values
  3. Pluggable Python function alternative for custom mapping logic
  4. Resource parameters are serialized as WorkflowRequestInputParameter records with type RESOURCE_PARAMETERS
  5. Parameters are injected into job params as __workflow_resource_params__ and made available to DTD and dynamic job destinations
  6. Validation of select-type parameters (e.g., priority) against user’s allowed options

Files Changed and Current Status

PR File PathStatusCurrent Location
client/galaxy/scripts/mvc/tool/tool-form-composite.jsGONE (rewritten)Client rewritten in Vue.js; equivalent in client/src/components/Workflow/Run/model.js
config/galaxy.yml.sampleMOVEDconfig/galaxy.yml.sample still exists but config also at lib/galaxy/config/sample/galaxy.yml.sample
config/workflow_resource_mapper_conf.yml.sampleEXISTSSame path, unchanged
config/workflow_resource_params_conf.xml.sampleEXISTSSame path, unchanged
lib/galaxy/config.pyMOVEDSplit into lib/galaxy/config/__init__.py; workflow_resource_params_mapper config still present there
lib/galaxy/jobs/__init__.pyEXISTSStill uses parse_resource_parameters from galaxy.util
lib/galaxy/jobs/dynamic_tool_destination.pyEXISTSStill contains the __workflow_resource_params__ parameter extraction logic
lib/galaxy/jobs/mapper.pyEXISTSStill exposes workflow_resource_params to dynamic destination functions
lib/galaxy/managers/workflows.pyEXISTSStill uses get_resource_mapper_function and _workflow_resource_parameters
lib/galaxy/model/__init__.pyEXISTSRESOURCE_PARAMETERS = "resource" type and resource_parameters property still present
lib/galaxy/tools/execute.pyEXISTSStill handles workflow_resource_parameters kwarg and __workflow_resource_params__ injection
lib/galaxy/util/__init__.pyEXISTSparse_resource_parameters utility function still present
lib/galaxy/webapps/galaxy/config_schema.ymlMOVEDNow at lib/galaxy/config/schemas/config_schema.yml; still has workflow_resource_params_mapper schema
lib/galaxy/workflow/modules.pyEXISTSStill reads invocation.resource_parameters and passes to execute; sits next to CWL tool path
lib/galaxy/workflow/resources/__init__.pyEXISTSCore mapper logic unchanged
lib/galaxy/workflow/resources/example.py.sampleEXISTSUnchanged
lib/galaxy/workflow/run_request.pyEXISTSStill handles resource_params in WorkflowRunConfig and build_workflow_run_configs
test/unit/jobs/dynamic_tool_destination/mockGalaxy.pyMOVEDNow at test/unit/app/jobs/dynamic_tool_destination/mockGalaxy.py

Key Code Patterns and Current Locations

1. Resource Parameter Flow (workflow invocation -> job)

  • Entry point: lib/galaxy/workflow/run_request.py - build_workflow_run_configs() reads resource_params from payload, validates select options, stores in WorkflowRunConfig
  • Serialization: workflow_run_config_to_request() stores each resource param as a WorkflowRequestInputParameter with type RESOURCE_PARAMETERS
  • Deserialization: workflow_request_to_run_config() reads them back
  • Injection into jobs: lib/galaxy/workflow/modules.py reads invocation.resource_parameters and passes to lib/galaxy/tools/execute.py which injects as __workflow_resource_params__

2. Model Changes

  • lib/galaxy/model/__init__.py: WorkflowRequestInputParameter.types.RESOURCE_PARAMETERS = "resource"
  • WorkflowInvocation.resource_parameters property filters input_parameters by this type

3. DTD Integration

  • lib/galaxy/jobs/dynamic_tool_destination.py: Reads __workflow_resource_params__ from job parameters; workflow priority takes precedence over job-level priority

4. Dynamic Destination Integration

  • lib/galaxy/jobs/mapper.py: Exposes workflow_resource_params as a named argument to dynamic destination functions

5. Mapper Framework

  • lib/galaxy/workflow/resources/__init__.py: get_resource_mapper_function() supports three modes:
    • None -> null mapper (no resource params shown)
    • Python function reference (module:function)
    • YAML file with by_group mapping (group-based permission model)

CWL Relevance

This PR has minimal direct CWL impact but is architecturally relevant:

  1. In lib/galaxy/workflow/modules.py, the resource_parameters are extracted from the invocation and passed through to execute() in the same code path that handles both Galaxy native tools and CWL tools. The resource_parameters extraction happens before the use_cwl_path branch point, meaning CWL workflow steps also receive workflow resource parameters.

  2. The __workflow_resource_params__ parameter is injected alongside __workflow_invocation_uuid__ in lib/galaxy/tools/execute.py - both follow the same pattern of being workflow-level metadata passed to individual jobs.

  3. No CWL-specific handling of resource parameters was added; CWL tools in workflows get the same resource parameter treatment as Galaxy native tools.

Architectural Decisions

  1. Pluggable mapper pattern: Admin can use YAML config OR Python function for mapping users to resource parameters - extensible design
  2. Group-based permissions: Default implementation ties resource parameter availability to Galaxy user groups
  3. Workflow-level priority overrides job-level: When both workflow and job resource params specify priority, workflow wins (DTD integration)
  4. Validation at request time: Select-type parameters are validated when the workflow invocation is created, not at job execution time
  5. Reuse of job resource parameter definitions: If no workflow-specific resource params file is configured, falls back to job_resource_params_file

Incoming References (2)