Galaxy Brain

markdown

Galaxy Markdown Architecture

Learning Questions

Learning Objectives

What is Galaxy Markdown?

Galaxy Markdown emerged to solve a fundamental portability problem. Traditional HTML-based documentation embedded instance-specific URLs—when workflows were exported and imported elsewhere, embedded links broke. Galaxy Markdown uses contextual addressing: references like output="results" or step="alignment" resolve at runtime to the appropriate objects in whichever Galaxy instance renders the document.

This enables truly portable documentation that travels with workflows, automatically adapting to new execution contexts while preserving the author’s intent.

Contextual Addressing

The diagram shows four input contexts converging to a single internal representation:

All contexts converge to Internal Galaxy Markdown with numeric IDs (history_dataset_id=12345). This conversion happens at context-appropriate boundaries— workflow refs resolve when an invocation is created, encoded IDs decode on import. The internal form then renders to HTML, PDF, or interactive UI.

Three Document Types

TypeEditableScopeUse Case
ReportsNoInvocationWorkflow output docs
PagesYesAnyUser documentation
Tool OutputNoJobTool-generated reports

Reports are auto-generated when a workflow completes. The WorkflowMarkdownGeneratorPlugin uses a default template containing invocation_inputs(), invocation_outputs(), and workflow_display() directives. Workflows can override this with custom reports_config.

Pages are user-created documents with full revision history—every save creates a new PageRevision record, enabling rollback and version comparison. Pages can embed any Galaxy object the user can access.

Tool Output markdown is generated by tools themselves, providing structured documentation of results. Like reports, these are read-only and scoped to a specific job execution.

Parser Components

lib/galaxy/managers/markdown_parse.py

The parser architecture separates concerns into two complementary files:

lib/galaxy/managers/markdown_parse.py (Syntactic Layer):

lib/galaxy/managers/markdown_util.py (Semantic Layer):

This separation enables the parser to be packaged independently, while Galaxy-specific resolution logic stays in the util module.

Directive Syntax

Block directive:

```galaxy
history_dataset_as_table(history_dataset_id=12345, title="Results")
```

Inline directive:

Text with ${galaxy history_dataset_name(output="results")} embedded.

lib/galaxy/managers/markdown_parse.py:GALAXY_MARKDOWN_FUNCTION_CALL_LINE

Galaxy Markdown supports two directive syntaxes:

Block directives appear in fenced code blocks with the galaxy language tag. The parser uses GALAXY_FLAVORED_MARKDOWN_CONTAINER_LINE_PATTERN to detect these blocks and GALAXY_MARKDOWN_FUNCTION_CALL_LINE to parse the directive call.

Inline directives use template syntax: ${galaxy directive(...)}. These are detected by EMBED_DIRECTIVE_REGEX and can appear anywhere in regular markdown text.

Arguments support both formats: unquoted values for IDs (history_dataset_id=12345) and quoted strings for display text (title="My Results"). The ARG_VAL_REGEX pattern handles both cases.

27 Directives

The 30+ directives fall into functional categories based on the Galaxy objects they render:

Each directive accepts specific arguments validated against ALLOWED_ARGUMENTS in markdown_parse.py. Invalid arguments trigger validation errors with line numbers.

Validation

def validate_galaxy_markdown(galaxy_markdown, internal=True):
    # Line-by-line fence tracking state machine
    for line, fenced, open_fence, line_no in _split_markdown_lines(markdown):
        if fenced and GALAXY_FUNC_CALL.match(line):
            _check_func_call(match, line_no)  # Validates args
    # Raises ValueError with line number on failure

lib/galaxy/managers/markdown_parse.py:validate_galaxy_markdown()

The validator implements a fail-fast strategy: it raises ValueError on the first invalid directive, reporting the exact line number for immediate feedback in the editor. The fence-tracking state machine handles edge cases like nested backticks and whitespace-only lines, ensuring only content within ```galaxy blocks is validated as directives.

Transformation Pipeline

The diagram shows the five-stage transformation pipeline:

  1. Import: ready_galaxy_markdown_for_import() decodes external IDs (base36) to internal numeric IDs for database storage.

  2. Validate: validate_galaxy_markdown() checks syntax without Galaxy dependencies.

  3. Resolve: resolve_invocation_markdown() expands workflow-relative references (output="results") to concrete IDs (history_dataset_id=12345).

  4. Export: Choose between lazy (ReadyForExportMarkdownDirectiveHandler) which preserves directives for frontend rendering, or eager (ToBasicMarkdownDirectiveHandler) which fully expands for PDF export.

  5. Render: Final output as HTML, PDF, or interactive Vue components.

Handler Pattern

lib/galaxy/managers/markdown_util.py

The handler pattern uses an abstract base class with two concrete implementations:

ReadyForExportMarkdownDirectiveHandler (lazy):

ToBasicMarkdownDirectiveHandler (eager):

This pattern enables adding new output formats by implementing the abstract interface without modifying existing handlers.

ID Encoding

# Storage (internal): numeric IDs
history_dataset_display(history_dataset_id=12345)

# Export (external): encoded IDs for URLs
history_dataset_display(history_dataset_id=a1b2c3d4e5f6)

# Regex patterns for conversion
UNENCODED_ID_PATTERN = r"(history_dataset_id)=([\\d]+)"
ENCODED_ID_PATTERN = r"(history_dataset_id)=([a-z0-9]+)"

Invocation Resolution

# Input: workflow-relative references
invocation_outputs(output="alignment_results")
job_metrics(step="bwa_mem")

# Output: instance-specific IDs
history_dataset_display(history_dataset_id=98765)
job_metrics(job_id=54321)

lib/galaxy/managers/markdown_util.py:resolve_invocation_markdown()

Invocation resolution is the key to portable workflow reports. When a workflow runs, resolve_invocation_markdown() maps abstract references to the specific objects created by that execution:

This happens via populate_invocation_markdown() which first attaches invocation_id to each directive, then resolution looks up the actual IDs from the invocation record. The same report template works across any invocation of the workflow.

PDF Export Pipeline

lib/galaxy/managers/markdown_util.py:internal_galaxy_markdown_to_pdf()

PDF export uses the eager handler to create self-contained documents:

  1. ToBasicMarkdownDirectiveHandler expands all directives to standard markdown
  2. markdown.markdown() converts to HTML
  3. HTML is sanitized for security
  4. WeasyPrint renders HTML to PDF
  5. Optional branding via markdown_export_prologue/epilogue and custom CSS

Instance administrators can customize PDF appearance per document type using markdown_export_css_reports, markdown_export_css_pages, etc. Large documents use async Celery tasks to avoid request timeouts.

Frontend Component Tree

client/src/components/Markdown/Markdown.vue

The frontend rendering hierarchy starts with Markdown.vue, which receives a MarkdownConfig containing the raw content and any validation errors from the backend.

parseMarkdown() splits content by triple-backtick delimiters into typed sections. SectionWrapper then dispatches each section to the appropriate renderer:

Unknown section types display an error alert rather than failing silently.

Section Parsing

// Triple-backtick splits content into typed sections
parseMarkdown(content) → [
  { type: 'markdown', content: '# Title...' },
  { type: 'galaxy', content: 'history_dataset_as_table(...)' },
  { type: 'vega', content: '{"$schema": "..."}' }
]

client/src/components/Markdown/parse.ts

The parser splits markdown content into discrete rendering units. Each fenced code block becomes a separate section with its language tag as the type:

This allows mixing standard markdown prose with embedded Galaxy directives, data visualizations, and interactive components in a single document.

Galaxy Directive Processing

client/src/components/Markdown/Sections/MarkdownGalaxy.vue

MarkdownGalaxy.vue processes Galaxy directives through a validation pipeline:

  1. Parse: getArgs() extracts the directive name and arguments
  2. Validate name: hasValidName() checks against the requirements.yml registry
  3. Validate object: hasValidObject() ensures required IDs are present (e.g., history_dataset_id for dataset directives)
  4. Validate label: hasValidLabel() checks workflow labels in report mode
  5. Render: Route to the appropriate Element component

Errors at any stage display user-friendly messages via Bootstrap alerts. The component uses Pinia stores (useInvocationStore, useWorkflowStore, useDatasetStore) to fetch and cache the data needed for rendering.

Element Components

21+ specialized renderers:

Each handles data fetching + rendering.

client/src/components/Markdown/Sections/Elements/

Each Element component handles a specific directive type with a consistent pattern:

Dataset elements detect content type and render appropriately: PDFs and HTML as iframes, images inline, tabular data as paginated tables. Job elements support both single jobs and implicit collection jobs via the useMappingJobs() composable.

Store-Centric Data Flow

Element components use Pinia stores as a caching layer between the UI and Galaxy APIs:

When multiple elements reference the same object, the store serves cached data instead of making redundant API calls. Stores use Vue’s reactivity system, so components automatically update when data changes.

Editor Architecture

client/src/components/Markdown/MarkdownEditor.vue

The markdown editor supports two editing modes:

TextEditor (always available):

CellEditor (workflow reports only):

Mode availability depends on context: pages get text mode only, workflow reports get both modes with a toggle.

Directive Registry

# directives.yml - metadata for editor UI
history_dataset_as_table:
  side_panel_name:
    page: "Dataset Table"
    report: "Output Table"
  help: "Embed dataset as formatted table..."

# templates.yml - insertion templates
history_dataset_as_table:
  template: 'history_dataset_as_table(history_dataset_id="%ID%")'

client/src/components/Markdown/directives.yml

The editor uses two YAML files to configure directive insertion:

directives.yml provides metadata for the editor UI:

templates.yml provides insertion templates:

Both files support mode-aware variants—the same directive can show as “Current Workflow” in report mode but “Display a Workflow” in page mode. The directives.ts module resolves these variants at runtime based on context.

Mode-Aware Design

Design Principles

The architecture follows five key principles:

Contextual Addressing: Multiple input formats (workflow labels, encoded IDs, output references) converge to internal numeric IDs at well-defined boundaries.

Lazy Resolution: Defer expensive operations until needed. The lazy handler preserves directives for frontend rendering; resolution happens on-demand.

Handler Extensibility: The abstract handler pattern enables new output formats without modifying existing code.

Mode Awareness: Pages, reports, and tool outputs have different rules for what can be referenced and how it’s displayed.

Separation of Concerns: Syntactic parsing (markdown_parse.py) is isolated from semantic resolution (markdown_util.py), and frontend rendering is independent of backend processing.

Adding a Directive

# 1. markdown_parse.py - add to ALLOWED_ARGUMENTS
ALLOWED_ARGUMENTS["new_directive"] = frozenset(["arg1", "arg2"])

# 2. markdown_util.py - add handler method
def handle_new_directive(self, ...):
    ...

# 3. Frontend - add element component
# client/src/components/Markdown/Sections/Elements/NewDirective.vue

# 4. directives.yml - add metadata

Adding a new directive requires changes across four layers:

  1. Backend parsing (markdown_parse.py): Add the directive name and allowed arguments to ALLOWED_ARGUMENTS. This enables validation without Galaxy dependencies.

  2. Backend handling (markdown_util.py): Implement handler methods in both ReadyForExportMarkdownDirectiveHandler (for frontend) and ToBasicMarkdownDirectiveHandler (for PDF export).

  3. Frontend rendering (Elements/): Create a Vue component that fetches data via Pinia stores and renders the appropriate UI.

  4. Editor integration (directives.yml, templates.yml): Add metadata for the editor sidebar and an insertion template.

The validation layer in MarkdownGalaxy.vue (requirements.yml) also needs updating to recognize the new directive and its required objects.

Architecture Overview

The architecture overview shows the complete flow:

Input: Contextual markdown from workflows, pages, or tools enters through format-specific APIs, each with its own addressing scheme.

Backend Processing: The four-stage pipeline (import → validate → resolve → export) transforms contextual references to internal IDs, then prepares content for the target format.

Frontend Rendering: Vue components parse sections, validate directives, fetch data through Pinia stores, and render interactive elements.

Output: The same source content produces interactive web UI, static PDF documents, or exportable markdown depending on the rendering path chosen.

Key Takeaways

Key Takeaways