# Galaxy Markdown Architecture

> 📊 <a href="markdown/slides.html">View as slides</a>

## Learning Questions
- What is Galaxy Markdown and how does it differ from regular HTML?
- How do contextual references get resolved to internal IDs?
- What is the transformation pipeline for Galaxy Markdown?
- How do frontend components render Galaxy directives?

## Learning Objectives
- Understand the contextual addressing system for Galaxy Markdown
- Learn the backend parsing and transformation pipeline
- Understand the handler pattern for export/rendering
- Know how frontend components process Galaxy directives
- Learn the editor architecture for pages and reports

## What is Galaxy Markdown?

- Portable, reproducible documentation with embedded Galaxy objects
- Alternative to HTML (instance-locked)
- Powers: workflow reports, pages, tool outputs


Galaxy Markdown emerged to solve a fundamental portability problem. Traditional HTML-based
documentation embedded instance-specific URLs—when workflows were exported and imported
elsewhere, embedded links broke. Galaxy Markdown uses contextual addressing: references
like `output="results"` or `step="alignment"` resolve at runtime to the appropriate
objects in whichever Galaxy instance renders the document.

This enables truly portable documentation that travels with workflows, automatically
adapting to new execution contexts while preserving the author's intent.


![Contextual Addressing](../_images/markdown_contextual_addressing.mmd.svg)


The diagram shows four input contexts converging to a single internal representation:

- **Workflow Markdown** uses step and output labels (`step="bwa_mem"`, `output="aligned_reads"`)
- **Tool Output Markdown** uses output references from the generating job
- **Page API** uses encoded IDs for URL safety (`history_dataset_id=a1b2c3d4`)
- **History Markdown** (planned) will use history-relative references

All contexts converge to **Internal Galaxy Markdown** with numeric IDs
(`history_dataset_id=12345`). This conversion happens at context-appropriate boundaries—
workflow refs resolve when an invocation is created, encoded IDs decode on import.
The internal form then renders to HTML, PDF, or interactive UI.


## Three Document Types

| Type | Editable | Scope | Use Case |
|------|----------|-------|----------|
| Reports | No | Invocation | Workflow output docs |
| Pages | Yes | Any | User documentation |
| Tool Output | No | Job | Tool-generated reports |


**Reports** are auto-generated when a workflow completes. The `WorkflowMarkdownGeneratorPlugin`
uses a default template containing `invocation_inputs()`, `invocation_outputs()`, and
`workflow_display()` directives. Workflows can override this with custom `reports_config`.

**Pages** are user-created documents with full revision history—every save creates a new
`PageRevision` record, enabling rollback and version comparison. Pages can embed any Galaxy
object the user can access.

**Tool Output** markdown is generated by tools themselves, providing structured documentation
of results. Like reports, these are read-only and scoped to a specific job execution.


![Parser Components](../_images/markdown_parser_components.mmd.svg)

`lib/galaxy/managers/markdown_parse.py`


The parser architecture separates concerns into two complementary files:

**lib/galaxy/managers/markdown_parse.py** (Syntactic Layer):
- Lightweight, self-contained parser with NO Galaxy dependencies
- Reusable in external tools (e.g., gxformat2)
- Recognizes directives within ` ```galaxy ` fenced blocks
- Validates directive syntax and arguments
- Reports errors with line numbers for debugging

**lib/galaxy/managers/markdown_util.py** (Semantic Layer):
- Galaxy-specific integration
- Resolves contextual references to internal IDs
- Implements handler pattern for export/rendering
- Handles ID encoding for URLs vs storage

This separation enables the parser to be packaged independently, while Galaxy-specific
resolution logic stays in the util module.


## Directive Syntax

**Block directive:**
````
```galaxy
history_dataset_as_table(history_dataset_id=12345, title="Results")
```
````

**Inline directive:**
```
Text with ${galaxy history_dataset_name(output="results")} embedded.
```

`lib/galaxy/managers/markdown_parse.py:GALAXY_MARKDOWN_FUNCTION_CALL_LINE`


Galaxy Markdown supports two directive syntaxes:

**Block directives** appear in fenced code blocks with the `galaxy` language tag. The parser
uses `GALAXY_FLAVORED_MARKDOWN_CONTAINER_LINE_PATTERN` to detect these blocks and
`GALAXY_MARKDOWN_FUNCTION_CALL_LINE` to parse the directive call.

**Inline directives** use template syntax: `${galaxy directive(...)}`. These are detected
by `EMBED_DIRECTIVE_REGEX` and can appear anywhere in regular markdown text.

Arguments support both formats: unquoted values for IDs (`history_dataset_id=12345`) and
quoted strings for display text (`title="My Results"`). The `ARG_VAL_REGEX` pattern
handles both cases.


## 27 Directives

- **Dataset:** `display`, `as_image`, `as_table`, `peek`, `info`, `link`, `name`, `type`
- **Workflow:** `display`, `image`, `license`, `invocation_inputs/outputs`
- **Job:** `parameters`, `metrics`, `tool_stdout/stderr`
- **Meta:** `generate_time`, `generate_galaxy_version`, `invocation_time`
- **Instance:** 6 `instance_*_link` variants


The 30+ directives fall into functional categories based on the Galaxy objects they render:

- **Dataset directives** display history items in various formats (tables, images, raw text)
- **Workflow directives** show workflow structure, licensing, and invocation summaries
- **Job directives** expose execution details like parameters, metrics, and tool output
- **Instance directives** generate links to Galaxy documentation and resources
- **Meta directives** insert generation timestamps and version information

Each directive accepts specific arguments validated against `ALLOWED_ARGUMENTS` in
`markdown_parse.py`. Invalid arguments trigger validation errors with line numbers.


## Validation

```python
def validate_galaxy_markdown(galaxy_markdown, internal=True):
    # Line-by-line fence tracking state machine
    for line, fenced, open_fence, line_no in _split_markdown_lines(markdown):
        if fenced and GALAXY_FUNC_CALL.match(line):
            _check_func_call(match, line_no)  # Validates args
    # Raises ValueError with line number on failure
```

`lib/galaxy/managers/markdown_parse.py:validate_galaxy_markdown()`


The validator implements a **fail-fast strategy**: it raises `ValueError` on the first
invalid directive, reporting the exact line number for immediate feedback in the editor.
The fence-tracking state machine handles edge cases like nested backticks and
whitespace-only lines, ensuring only content within ` ```galaxy ` blocks is validated
as directives.


![Transformation Pipeline](../_images/markdown_transformation_pipeline.mmd.svg)


The diagram shows the five-stage transformation pipeline:

1. **Import**: `ready_galaxy_markdown_for_import()` decodes external IDs (base36) to
   internal numeric IDs for database storage.

2. **Validate**: `validate_galaxy_markdown()` checks syntax without Galaxy dependencies.

3. **Resolve**: `resolve_invocation_markdown()` expands workflow-relative references
   (`output="results"`) to concrete IDs (`history_dataset_id=12345`).

4. **Export**: Choose between lazy (`ReadyForExportMarkdownDirectiveHandler`) which
   preserves directives for frontend rendering, or eager (`ToBasicMarkdownDirectiveHandler`)
   which fully expands for PDF export.

5. **Render**: Final output as HTML, PDF, or interactive Vue components.


![Handler Pattern](../_images/markdown_handler_pattern.mmd.svg)

`lib/galaxy/managers/markdown_util.py`


The handler pattern uses an abstract base class with two concrete implementations:

**ReadyForExportMarkdownDirectiveHandler** (lazy):
- Preserves directives in output for frontend processing
- Collects metadata (dataset names, types, peek data) in `extra_rendering_data`
- Used when rendering interactive Vue components

**ToBasicMarkdownDirectiveHandler** (eager):
- Fully expands directives to standard markdown
- Converts datasets to formatted tables, embeds images as base64 data URIs
- Used for PDF export where all content must be self-contained

This pattern enables adding new output formats by implementing the abstract interface
without modifying existing handlers.


## ID Encoding

```python
# Storage (internal): numeric IDs
history_dataset_display(history_dataset_id=12345)

# Export (external): encoded IDs for URLs
history_dataset_display(history_dataset_id=a1b2c3d4e5f6)

# Regex patterns for conversion
UNENCODED_ID_PATTERN = r"(history_dataset_id)=([\\d]+)"
ENCODED_ID_PATTERN = r"(history_dataset_id)=([a-z0-9]+)"
```


## Invocation Resolution

```python
# Input: workflow-relative references
invocation_outputs(output="alignment_results")
job_metrics(step="bwa_mem")

# Output: instance-specific IDs
history_dataset_display(history_dataset_id=98765)
job_metrics(job_id=54321)
```

`lib/galaxy/managers/markdown_util.py:resolve_invocation_markdown()`


Invocation resolution is the key to portable workflow reports. When a workflow runs,
`resolve_invocation_markdown()` maps abstract references to the specific objects created
by that execution:

- `output="aligned_reads"` becomes `history_dataset_id=98765`
- `step="bwa_mem"` becomes `job_id=54321`
- `input="reference_genome"` becomes `history_dataset_id=12345`

This happens via `populate_invocation_markdown()` which first attaches `invocation_id`
to each directive, then resolution looks up the actual IDs from the invocation record.
The same report template works across any invocation of the workflow.


![PDF Export Pipeline](../_images/markdown_pdf_export.mmd.svg)

`lib/galaxy/managers/markdown_util.py:internal_galaxy_markdown_to_pdf()`


PDF export uses the eager handler to create self-contained documents:

1. `ToBasicMarkdownDirectiveHandler` expands all directives to standard markdown
2. `markdown.markdown()` converts to HTML
3. HTML is sanitized for security
4. WeasyPrint renders HTML to PDF
5. Optional branding via `markdown_export_prologue/epilogue` and custom CSS

Instance administrators can customize PDF appearance per document type using
`markdown_export_css_reports`, `markdown_export_css_pages`, etc. Large documents
use async Celery tasks to avoid request timeouts.


![Frontend Component Tree](../_images/markdown_frontend_components.mmd.svg)

`client/src/components/Markdown/Markdown.vue`


The frontend rendering hierarchy starts with `Markdown.vue`, which receives a
`MarkdownConfig` containing the raw content and any validation errors from the backend.

`parseMarkdown()` splits content by triple-backtick delimiters into typed sections.
`SectionWrapper` then dispatches each section to the appropriate renderer:

- **MarkdownDefault**: Standard markdown via markdown-it, with KaTeX for math
- **MarkdownGalaxy**: Galaxy directives (the focus of this architecture)
- **MarkdownVega**: Vega-Lite data visualizations
- **MarkdownVisualization**: Galaxy's plugin-based visualizations
- **MarkdownVitessce**: Spatial and single-cell data viewers

Unknown section types display an error alert rather than failing silently.


## Section Parsing

```typescript
// Triple-backtick splits content into typed sections
parseMarkdown(content) → [
  { type: 'markdown', content: '# Title...' },
  { type: 'galaxy', content: 'history_dataset_as_table(...)' },
  { type: 'vega', content: '{"$schema": "..."}' }
]
```

`client/src/components/Markdown/parse.ts`


The parser splits markdown content into discrete rendering units. Each fenced code block
becomes a separate section with its language tag as the type:

- ` ```galaxy ` → `{ type: 'galaxy', content: '...' }`
- ` ```vega ` → `{ type: 'vega', content: '...' }`
- Regular markdown between blocks → `{ type: 'markdown', content: '...' }`

This allows mixing standard markdown prose with embedded Galaxy directives, data
visualizations, and interactive components in a single document.


![Galaxy Directive Processing](../_images/markdown_directive_processing.mmd.svg)

`client/src/components/Markdown/Sections/MarkdownGalaxy.vue`


`MarkdownGalaxy.vue` processes Galaxy directives through a validation pipeline:

1. **Parse**: `getArgs()` extracts the directive name and arguments
2. **Validate name**: `hasValidName()` checks against the `requirements.yml` registry
3. **Validate object**: `hasValidObject()` ensures required IDs are present
   (e.g., `history_dataset_id` for dataset directives)
4. **Validate label**: `hasValidLabel()` checks workflow labels in report mode
5. **Render**: Route to the appropriate Element component

Errors at any stage display user-friendly messages via Bootstrap alerts. The component
uses Pinia stores (`useInvocationStore`, `useWorkflowStore`, `useDatasetStore`) to
fetch and cache the data needed for rendering.


## Element Components

**21+ specialized renderers:**
- `HistoryDatasetAsTable`, `HistoryDatasetAsImage`
- `WorkflowImage`, `WorkflowDisplay`
- `JobMetrics`, `JobParameters`
- `ToolStdout`, `ToolStderr`

Each handles data fetching + rendering.

`client/src/components/Markdown/Sections/Elements/`


Each Element component handles a specific directive type with a consistent pattern:

- **Props**: Receive IDs and configuration from `MarkdownGalaxy`
- **Stores**: Fetch data via Pinia stores with automatic caching
- **Rendering**: Type-aware display (tables, images, iframes, formatted text)
- **Errors**: Graceful handling with `BAlert` components

Dataset elements detect content type and render appropriately: PDFs and HTML as iframes,
images inline, tabular data as paginated tables. Job elements support both single jobs
and implicit collection jobs via the `useMappingJobs()` composable.


![Store-Centric Data Flow](../_images/markdown_store_data_flow.mmd.svg)


Element components use Pinia stores as a caching layer between the UI and Galaxy APIs:

- **useDatasetStore**: Dataset metadata and content
- **useInvocationStore**: Workflow invocation records
- **useWorkflowStore**: Workflow definitions
- **useJobStore**: Job execution details
- **useDatatypesMapperStore**: Datatype hierarchy for rendering decisions

When multiple elements reference the same object, the store serves cached data instead
of making redundant API calls. Stores use Vue's reactivity system, so components
automatically update when data changes.


![Editor Architecture](../_images/markdown_editor_architecture.mmd.svg)

`client/src/components/Markdown/MarkdownEditor.vue`


The markdown editor supports two editing modes:

**TextEditor** (always available):
- Two-panel layout: MarkdownToolBox sidebar + textarea
- Direct markdown editing with syntax highlighting
- MarkdownToolBox provides directive insertion with dialogs
- 300ms debounced updates to prevent excessive re-renders

**CellEditor** (workflow reports only):
- Jupyter-like cell-based editing
- Each fenced code block becomes a discrete cell
- Visual preview of rendered content
- Cell operations: add, delete, clone, move, configure
- Lazy-loaded Ace editor for syntax highlighting

Mode availability depends on context: pages get text mode only, workflow reports
get both modes with a toggle.


## Directive Registry

```yaml
# directives.yml - metadata for editor UI
history_dataset_as_table:
  side_panel_name:
    page: "Dataset Table"
    report: "Output Table"
  help: "Embed dataset as formatted table..."

# templates.yml - insertion templates
history_dataset_as_table:
  template: 'history_dataset_as_table(history_dataset_id="%ID%")'
```

`client/src/components/Markdown/directives.yml`


The editor uses two YAML files to configure directive insertion:

**directives.yml** provides metadata for the editor UI:
- `side_panel_name`: Display label (can vary by mode)
- `side_panel_description`: Brief tooltip text
- `help`: Detailed usage guidance with `%MODE%` placeholder

**templates.yml** provides insertion templates:
- Default argument placeholders
- Pre-filled values for common patterns

Both files support mode-aware variants—the same directive can show as "Current Workflow"
in report mode but "Display a Workflow" in page mode. The `directives.ts` module
resolves these variants at runtime based on context.


## Mode-Aware Design

- **Page mode:** Reference any workflow/dataset in instance
- **Report mode:** Reference "this" invocation (step labels)
- Same directives, different presentation and validation


![Design Principles](../_images/markdown_design_principles.mindmap.mmd.svg)


The architecture follows five key principles:

**Contextual Addressing**: Multiple input formats (workflow labels, encoded IDs, output
references) converge to internal numeric IDs at well-defined boundaries.

**Lazy Resolution**: Defer expensive operations until needed. The lazy handler preserves
directives for frontend rendering; resolution happens on-demand.

**Handler Extensibility**: The abstract handler pattern enables new output formats without
modifying existing code.

**Mode Awareness**: Pages, reports, and tool outputs have different rules for what can be
referenced and how it's displayed.

**Separation of Concerns**: Syntactic parsing (markdown_parse.py) is isolated from semantic
resolution (markdown_util.py), and frontend rendering is independent of backend processing.


## Adding a Directive

```python
# 1. markdown_parse.py - add to ALLOWED_ARGUMENTS
ALLOWED_ARGUMENTS["new_directive"] = frozenset(["arg1", "arg2"])

# 2. markdown_util.py - add handler method
def handle_new_directive(self, ...):
    ...

# 3. Frontend - add element component
# client/src/components/Markdown/Sections/Elements/NewDirective.vue

# 4. directives.yml - add metadata
```


Adding a new directive requires changes across four layers:

1. **Backend parsing** (`markdown_parse.py`): Add the directive name and allowed arguments
   to `ALLOWED_ARGUMENTS`. This enables validation without Galaxy dependencies.

2. **Backend handling** (`markdown_util.py`): Implement handler methods in both
   `ReadyForExportMarkdownDirectiveHandler` (for frontend) and `ToBasicMarkdownDirectiveHandler`
   (for PDF export).

3. **Frontend rendering** (`Elements/`): Create a Vue component that fetches data via
   Pinia stores and renders the appropriate UI.

4. **Editor integration** (`directives.yml`, `templates.yml`): Add metadata for the
   editor sidebar and an insertion template.

The validation layer in `MarkdownGalaxy.vue` (`requirements.yml`) also needs updating
to recognize the new directive and its required objects.


![Architecture Overview](../_images/markdown_architecture_overview.mmd.svg)


The architecture overview shows the complete flow:

**Input**: Contextual markdown from workflows, pages, or tools enters through format-specific
APIs, each with its own addressing scheme.

**Backend Processing**: The four-stage pipeline (import → validate → resolve → export)
transforms contextual references to internal IDs, then prepares content for the target format.

**Frontend Rendering**: Vue components parse sections, validate directives, fetch data through
Pinia stores, and render interactive elements.

**Output**: The same source content produces interactive web UI, static PDF documents,
or exportable markdown depending on the rendering path chosen.


## Key Takeaways

- **Portable:** Context-specific refs → internal IDs at boundaries
- **Extensible:** Handler pattern, directive registry, new contexts
- **Multi-format:** Same internal form → HTML, PDF, interactive UI
- **27+ directives** for embedding Galaxy objects


## Key Takeaways
- Galaxy Markdown enables portable documentation via contextual addressing
- Multiple context-specific formats converge to internal IDs at boundaries
- Handler pattern enables lazy (frontend) vs eager (PDF) export
- 27+ directives for embedding Galaxy objects
- Frontend uses store-centric data flow for caching