Galaxy Brain

WF_VIZ_1_PLAN

WF_VIZ_1 — Cytoscape.js Port + Map/Reduce Encoding for Mermaid & Cytoscape

Goal: port gxformat2’s gxwf-viz (Cytoscape.js JSON + standalone HTML) into the TS monorepo as a new gxwf cytoscapejs subcommand, then thread connection-validation map-over / reduction info into both the Mermaid and the new Cytoscape emitters and encode it visually.

Two phases. Phase A is the port (no map/reduce yet — keep parity with Python first so fixtures are comparable). Phase B layers map/reduce encoding onto both emitters.

Status: both phases shipped. Phase A landed in e0ae7ec9 (Apr 27). Phase B landed in daa6d39c (Apr 27).

Discoveries log (2026-04-27)

Source-of-truth pointers

Naming: command is gxwf cytoscapejs (per ask). Internal package surface is cytoscape to match the Python module.

Phase A — Port gxwf-viz to TS as gxwf cytoscapejs ✅

Done in e0ae7ec9. A1–A7 all landed; A8 (file gxformat2 issue to switch gxwf viz from subprocess to in-process call) deferred — separate Python PR, optional.

A red→green workflow with fixtures imported from gxformat2 as the ground truth.

A1. Decide the package home

Options:

  1. New packages/workflow-viz/ (parallel to connection-validation).
  2. Add to existing packages/schema/src/workflow/cytoscape.ts next to mermaid.ts.

Decided: #2 — packages/schema/src/workflow/cytoscape.ts for the builder + models; packages/cli/src/commands/cytoscapejs/ for the HTML template + render fn. The Python module is ~150 LOC builder + ~130 LOC HTML template — no reason to spin a package up front. Co-locating with mermaid.ts keeps the “workflow → diagram” emitters in one place and reuses ensureFormat2 / resolveSourceReference. HTML render is a CLI-only concern; downstream consumers don’t need to ship the template.

A2. Port the Pydantic models → Effect Schema (or plain TS interfaces)

gxformat2/cytoscape/models.py defines CytoscapePosition, CytoscapeNodeData, CytoscapeEdgeData, CytoscapeNode, CytoscapeEdge, CytoscapeElements (with to_list()).

A3. Port the builder

Create packages/schema/src/workflow/cytoscape.ts:

Resolved: TS NF2 carries everything we need. position and tool_shed_repository are both Schema.optional in packages/schema/src/workflow/normalized/format2.ts (lines 60, 100, 114) and normalized/native.ts (lines 63, 66). doc and type_ (→ step_type) are also present. No NF2 enrichment commit is needed.

Edge-id format must match Python byte-for-byte: f"{step_id}__{input_id}__from__{ref.step_label}", and output is null when the resolved source’s output name is "output" (the default). Don’t camelCase these.

A4. Port the HTML render

A5. CLI wiring

Add packages/cli/src/commands/cytoscapejs.ts:

gxwf cytoscapejs <file> [output]
  --html       # force HTML output (default if [output] ends with .html)
  --json       # force JSON output (default if .json or stdout)

Mirror mermaid.ts shape. If output omitted: stdout JSON. If output ends .html: HTML render. Add to gxwf.ts like the existing mermaid command.

Decided: stdout-JSON when no output path is given. Document the divergence from Python (which writes .html next to input) in the changeset. Matches the mermaid CLI norm.

A6. Fixtures + tests (red→green, declarative)

Approach decided after gxformat2 PR #196 landed. No sidecar .cytoscape.json goldens, no new sync target. The Python side ships gxformat2/examples/expectations/cytoscape.yml (13 declarative cases) consumed by DeclarativeTestSuite. The TS port runs the same YAML against its own builder; parity is enforced by the assertions, not by byte-equal JSON files.

Steps:

  1. No new Makefile target. cytoscape.yml will sync automatically with the next make sync-workflow-expectations (the existing workflow-expectations group in scripts/sync-manifest.json already globs *.yml from gxformat2/examples/expectations/).
  2. Verify the synced cytoscape.yml lands at packages/schema/test/fixtures/expectations/cytoscape.yml and that the existing declarative harness (the one driving mermaid.yml) picks it up automatically.
  3. Implement TS counterparts for the three operations registered in tests/test_interop_tests.py:
    • cytoscape_elements_to_list — flat list of dicts in cytoscape.js shape
    • cytoscape_node_ids — [el.data.id for el in nodes]
    • cytoscape_edge_ids — [el.data.id for el in edges] These plug into the harness alongside workflow_to_mermaid etc.
  4. Hand-written: one tiny test for renderHtml that asserts the HTML contains </body> and cytoscape (mirrors Python’s test_render_html).

Limitation of the harness’s path navigation: the {field: value} finder uses getattr, which doesn’t work on plain dicts (only on Pydantic-style models with attribute access). Index-based paths like [2, "data", "id"] work fine on the list-of-dicts output. Plan accordingly when adding new cases.

Nullable fields: for keys like tool_id: null and edge output: null, use value: null (YAML’s null → Python None / JS null). value_absent: true checks whether the path resolves, not whether the value is null — these keys are always present in the dict.

Hard constraint: TS output shape must match Python byte-for-byte for the unannotated cases (snake_case fields, edge-id format, output: null convention). If they don’t, that’s a normalization bug to fix in NF2, not a reason to relax assertions.

A7. Docs + changeset

A8. Convergence note (optional but cheap)

The old GXWF_CLI_PLAN.md documents Python’s gxwf as a subprocess pass-through to gxwf-viz. Once the TS gxwf cytoscapejs lands, the Python gxwf viz should switch from subprocess pass-through to a direct in-process call to gxformat2.cytoscape.main(...) — that’s a separate small Python PR and not in scope here, but file an issue so we track convergence.

Phase B — Map / Reduce Encoding ✅

Done in daa6d39c. All steps B0–B6 implemented. Deviations and notes summarized at the end of this section.

Pre-req: Phase A merged so we have both emitters speaking the same node/edge model.

B0. Surface reductions in the connection-validation result

Today StepConnectionResult.mapOver exists, but reductions are detected and consumed inside _validateConnection without leaving a trace on ConnectionValidationResult. Add:

interface ConnectionValidationResult {
  ...
  reduction?: boolean;          // true if this connection reduced (list-like → multi=true)
  mapDepth?: number;            // 0 = scalar passthrough, 1 = list, 2 = list:paired, ...
}

mapDepth is derivable from mapping (count colons + 1 when set, 0 otherwise) — make it the canonical field for downstream emitters; mapping stays for human-readable text.

Tasks:

  1. Extend the type.
  2. Populate from connection-validator.ts:
    • At the multi-data reduction branch (line ~212), set reduction: true.
    • At map-over success path (line ~227), compute mapDepth from mapOver.collectionType.
    • Scalar matches: mapDepth: 0.
  3. Update fixtures under connection-validation/test/ so the report includes the new fields. Sync from the Python side if it grows the same fields; otherwise this is a TS-only enrichment for now (file a Python parity issue).
  4. Update report-builder.ts to emit the new fields.

B1. Common contract: an “edge annotation” type

Both emitters benefit from the same lookup. Add to packages/connection-validation/src/index.ts:

interface EdgeAnnotation {
  sourceStep: string;
  sourceOutput: string;
  targetStep: string;
  targetInput: string;
  mapDepth: number;
  reduction: boolean;
  mapping?: string | null;   // textual, e.g., "list:paired"
}

function buildEdgeAnnotations(report: WorkflowConnectionResult): Map<string, EdgeAnnotation>;
// key format: `${sourceStep}|${sourceOutput}->${targetStep}|${targetInput}`

This is what both emitters consume. No emitter imports the validator directly; they both take an optional EdgeAnnotation map / lookup fn.

B2. Mermaid: encode at source level

Edit packages/schema/src/workflow/mermaid.ts:

export interface MermaidOptions {
  comments?: boolean;
  edgeAnnotations?: Map<string, EdgeAnnotation>;
}

Encoding rules (start simple, document the chosen encoding):

AnnotationSource emitted
mapDepth === 0, no reductionA --> B (today’s behavior)
mapDepth >= 1, no reduction`A ==>
mapDepth >= 2A ==>|"list:paired"| B (thick + nested label)
reduction === trueA -. reduce .-> B (dotted, labeled)

Plus an optional linkStyle N stroke:#888,stroke-width:Npx line per edge keyed off its declaration index — tracked with a counter while emitting edges. Width grows with mapDepth (e.g. 2 + mapDepth).

Emission stability: declaration order = linkStyle index, so we can write one consolidated linkStyle block at the bottom. Don’t intermix.

Fixture additions:

B3. Cytoscape: encode at data + style level

Edit packages/schema/src/workflow/cytoscape.ts:

Edit the HTML template (cytoscape.html) to add styles:

{ selector: 'edge.mapover_1', style: { width: 4, 'curve-style': 'bezier' } },
{ selector: 'edge.mapover_2', style: { width: 6 } },
{ selector: 'edge.mapover_3', style: { width: 8 } },
{ selector: 'edge.reduction', style: { 'line-style': 'dashed', 'target-arrow-shape': 'tee' } },

For a true ribbon effect (multiple parallel strands), use cytoscape-edge-bundling or render N parallel bezier edges with stepped control-point-distances. Cleanest: for each mapDepth >= 1, emit N edges with the same source/target but different ids (<base>__strand_<k>) and offset control-point-distances. The first cut should ship with width-encoding only (single edge); add the multi-strand ribbon as a follow-up flag (--ribbons or in-template toggle) once the simple version is validated.

Tooltip update in template: include “Map depth: N” / “Reduction” in the edge-tooltip block.

Fixtures:

B4. Wire the validator into the CLI emitters

gxwf cytoscapejs and gxwf mermaid both currently load workflow → emit. Add a flag:

--annotate-connections    # run connection validator and encode mapDepth/reduction

Off by default to keep existing fixtures stable. When on:

  1. Build WorkflowGraph (same path gxwf validate --connections uses).
  2. Run validateWorkflowConnections.
  3. buildEdgeAnnotations(report) → pass to emitter via options.

Add to gxwf.ts:

.option("--annotate-connections", "Encode map-over / reduction info on edges (runs connection validator)")

Open Q: should gxwf validate --connections --report-html pipe its existing report through the cytoscape view as a richer alternative to the plain HTML report? Probably yes — but that’s a separate UI track (gxwf-report-shell).

B5. Tests

B6. Docs + changeset

Phase B — what actually shipped (daa6d39c)

Single squashed commit (not the 7-commit slice in the section below — pragmatic given the small surface).

Loose ends / follow-ups

Suggested commit/PR layout

Squashable into one PR per phase, but the natural commit slices:

Phase A

  1. cytoscape: port models + builder
  2. cytoscape: HTML render + template
  3. cli: add gxwf cytoscapejs command
  4. cytoscape: sync fixtures from gxformat2 + parity tests
  5. docs/changeset: cytoscapejs CLI

Phase B

  1. connection-validation: surface reductions + mapDepth on result
  2. connection-validation: buildEdgeAnnotations helper
  3. mermaid: encode mapDepth/reduction (option-gated)
  4. cytoscape: encode mapDepth/reduction on edge data + classes + style
  5. cli: --annotate-connections flag wires validator into both emitters
  6. fixtures: synthetic map/reduce coverage for both emitters
  7. docs/changeset: edge annotation encoding

Test strategy summary (matching the project’s red→green declarative norm)

Risks / things to verify before coding

  1. NF2 field coverage. Resolved: TS NF2 already carries tool_shed_repository, position, doc, and type_ (→ step_type).
  2. Snake_case vs camelCase. Cytoscape JSON must be snake_case to match Python and to interop with the existing HTML template. Be deliberate about not letting the project’s general camelCase convention bleed into the output models.
  3. Fixture drift. No longer applies: there is no committed .cytoscape.json golden set in gxformat2 (tests/examples/cytoscape/ is gitignored). Parity is enforced via the synced cytoscape.yml declarative cases instead.
  4. Reduction detection completeness. The validator detects multi-data reductions (connection-validator.ts:212); verify there’s no other reduction case (e.g. element-identifier, scalar-from-collection) currently silently handled. If there is, B0 must capture them all.
  5. Mermaid label escaping. Adding |"list:paired"| labels — confirm sanitizeLabel is applied, since : and " matter to Mermaid parsers.
  6. HTML template CDN. Python pins to old versions (cytoscape 3.9.4, tippy 4.0.1, popper 1.14.7, cytoscape-popper 1.0.4). Bumping is a separate gxformat2 PR — the TS port should sync the template as-is via scripts/sync-manifest.json (new group cytoscape-template pointing at gxformat2/cytoscape/cytoscape.html) so the template never forks.
  7. Nullable-key gotcha in the declarative harness. value_absent: true checks whether the path resolves, not whether the value is null. For tool_id/output/doc/repo_link (always present, sometimes null), use value: null not value_absent.

Resolved questions (post-discovery)

Unresolved questions