PR #19377 Research Summary

Title: Empower Users to Build More Kinds of Collections, More Intelligently Author: John Chilton (jmchilton) URL: https://github.com/galaxyproject/galaxy/pull/19377 Status: MERGED into dev (branch: fixed_length_collections) Labels: area/UI-UX, kind/feature, area/API, area/dataset-collections, area/tool-framework, highlight Merge commit in dev: c212434dc8 In current branch’s merge base: Yes — all PR changes are already in the yaml_tool_harden_1 working tree.

Summary

Massive feature PR (~6700 additions across 130+ files) introducing:

1. paired_or_unpaired collection type — mixed paired/unpaired data in a single collection. Elements are either {forward, reverse} (paired) or {unpaired} (singleton). Tools declaring collection<paired_or_unpaired> also accept plain paired input. Lists of these (list:paired_or_unpaired) can match both list and list:paired.

2. record collection type — CWL-style heterogeneous tuples with named, typed fields. Generalization of paired. Fields defined via FieldDict ({name, type, format?}). Database stores fields JSON on DatasetCollection.

3. Collection Adapter framework (adapters.py) — wraps model objects to create ephemeral/pseudo collections for tool execution. Key adapters: PromoteCollectionElementToCollectionAdapter, PromoteDatasetToCollection, PromoteDatasetsToCollection. Adapter state serialized to adapter JSON column on job input association tables.

4. List Wizard UI — replaces PairedListCollectionCreator (deleted, -1257 lines) with wizard-based ListWizard.vue + PairedOrUnpairedListCollectionCreator.vue. “Auto Build List” and “Advanced Build List” options in history dropdown. Auto-detects pairing.

5. Rule-Based Import Activity — new standalone activity + wizard for seeding rule builder from remote files, pasted data, existing datasets, etc.

6. Collection Semantics Documentation (collection_semantics.yml + generated collection_semantics.md) — formal specification of mapping, reduction, sub-collection mapping, and paired_or_unpaired semantics with labeled examples tied to test cases.

7. __SPLIT_PAIRED_AND_UNPAIRED__ tool — new collection operation that splits list:paired_or_unpaired into homogeneous list + list:paired.

8. Database migration (ec25b23b08e2) — adds fields column to dataset_collection, adapter column to job_to_input_dataset, job_to_input_dataset_collection, job_to_input_dataset_collection_element.

Key Architectural Decisions

• Collection type matching is asymmetric: paired_or_unpaired consumes paired but not vice versa. can_match_type() in type_description.py implements this. list:paired_or_unpaired matches list and list:paired.
• Adapters bridge type gaps at runtime: When a paired collection is passed to a paired_or_unpaired input, an adapter wraps it. Adapter JSON is persisted on job input associations for provenance but isn’t re-used during job evaluation.
• single_datasets pseudo-type: Used as a subcollection type to allow flat collections (like paired_or_unpaired) to be split into individual DCE-level adapters.
• fields parameter threaded everywhere: From API payload -> collections_util -> collections.py -> builder.py -> type plugins. "auto" value triggers field guessing from element identifiers.
• Discriminated unions for adapter models in tool_util_models/parameters.py — AdaptedDataCollectionRequest and AdaptedDataCollectionRequestInternal use adapter_type discriminator.
• DatasetCollection.allow_implicit_mapping: Records return False — they don’t participate in implicit mapping.

Files Changed (130 files)

New Backend Files

New Frontend Files (selection)

Deleted Files

Significantly Modified Backend Files

Cross-References to Current Branch (yaml_tool_harden_1)

Files Modified by Both PR and Current Branch (40 files overlap)

The current branch focuses on structured tool state, YAML tool support, and collection runtime hardening. Key overlapping areas:

No Conflicts

Since PR #19377 is already merged into dev and the current branch is based on a commit after that merge (c212434dc8 is an ancestor of the merge base c89c5f0644), there are no merge conflicts. The current branch builds on top of all PR #19377 changes.

Architectural Alignment

The current branch’s work on collection runtime models with discriminated unions (commits 200d5b8fbc, cd04873951) directly extends the patterns established by PR #19377:

• PR #19377 introduced discriminated unions for adapter models (AdaptedDataCollectionRequestInternal discriminated on adapter_type)
• Current branch extends discriminated unions to collection runtime models (e.g., DataCollectionPairedRuntime, dynamic model factory for recursive collection types)
• PR #19377’s FieldDict type from tool_source.py is available for the current branch’s YAML tool work
• The collection_semantics.yml spec provides a reference for what test cases should exist for the new collection types

Key Types/Interfaces Introduced by PR Available in Current Branch

• CollectionAdapter and subclasses (adapters.py)
• FieldDict, CwlType, FieldType (tool_source.py)
• AdaptedDataCollectionRequest* models (parameters.py)
• PairedOrUnpairedDatasetCollectionType and RecordDatasetCollectionType (type plugins)
• DatasetCollection.fields and DatasetCollection.allow_implicit_mapping
• can_match_type() with paired_or_unpaired semantics
• SplitPairedAndUnpairedTool tool type