How It Works

Hub workflows are dxflow workflow definitions that dxflow deploys on Docker, Podman, Singularity, or Apptainer. This page covers how they're structured, deployed, and managed.

Workflow structure

A workflow has a name, optional tags, and one or more steps (containers):

name: app
tags:
  - example
steps:
  - name: app
    runtime: docker
    mode: sequential
    image: workflow-image:latest
    command:
      - app
      - --serve
    env:
      - PARAM1=value1
    volumes:
      - host: ./data
        container: /data
      - host: ./results
        container: /results
    ports:
      - host: "8080"
        container: "8080"
    resources:
      cpu: "4"
      memory: 8G

Each step field:

• runtime — the container runtime that runs the step: docker, podman, singularity, or apptainer (default docker).
• mode — sequential (steps run in order, one after another; requires a command) or parallel (steps run at the same time). Default sequential.
• image — the container image to run.
• command — runs inside the container as raw arguments, with no shell; wrap it in /bin/sh -c "…" when you need globs, pipes, or variable expansion.
• env — environment variables as KEY=value, to customize behavior without editing the file.
• volumes — map local dirs to container paths for I/O; add mode: ro for read-only.
• ports — expose web interfaces or APIs as host/container pairs.
• resources — cpu, memory, and gpu limits.

Deploy a workflow

Find a workflow in a category, then deploy it via the web console or the CLI.

Web console: open http://localhost, go to Apps & Pipelines → Hub / Templates, select a workflow, configure parameters, and click Deploy.

CLI:

dxflow workflow create --identity my-workflow hub://fastqc
dxflow workflow start my-workflow
dxflow workflow logs --live my-workflow

Each workflow page lists its workflow definition, configuration options, usage steps, and system requirements.

The hub:// source

dxflow workflow create takes a source locator, and hub://<name> resolves it against this catalog. <name> is the workflow's folder name with the numeric prefix dropped — 02.genomics/01.fastqc/ is hub://fastqc. The engine looks the entry up in dxflow-ai/hub (main branch) at deploy time, extracts the yaml block from its ## Configuration section, and creates the workflow from it — so the file stays on the hub, not on your machine, and each deploy picks up the current definition.

Browse and read entries with the same names:

# Search across name, description, tags, and category (empty query lists everything)
dxflow workflow hub search genomics

# Print the full entry document, or one section
dxflow workflow hub inspect fastqc
dxflow workflow hub inspect fastqc --yaml
dxflow workflow hub inspect fastqc --ini

The other sources create accepts:

• hub://<name> — a workflow from this catalog, by name.
• file://<path> — a definition on the engine host; a bare path is read as this too.
• http://… / https://… — a definition fetched over HTTP.

Take a hub definition local when you want to edit it — inspect it into a file, change it, and create from the file:

dxflow workflow hub inspect fastqc --yaml > fastqc.yml
dxflow workflow create --identity my-fastqc fastqc.yml

Manage data

Upload input before running and download results after:

# Upload input (destination path is created automatically)
dxflow artifact upload /local/data.csv /data/input/

# List and download output
dxflow artifact list /data/output
dxflow artifact download /data/output/ /local/results/

Volume mounts persist data across restarts, and shared volumes let multiple workflows read the same data.

Lifecycle

A workflow moves through created → started → stopped/exited (or failed on error — check the logs). Manage it with:

dxflow workflow start my-workflow
dxflow workflow stop my-workflow
dxflow workflow remove my-workflow

Common patterns

Interactive (web UI) — expose a port to reach a browser interface at http://localhost:8888:

steps:
  - name: jupyter
    image: jupyter/scipy-notebook
    command:
      - start-notebook.sh
    ports:
      - host: "8888"
        container: "8888"

GPU — request an NVIDIA GPU for ML or scientific computing:

steps:
  - name: ml-training
    image: tensorflow/tensorflow:latest-gpu
    command:
      - python
      - train.py
    resources:
      gpu: nvidia

Multi-step pipeline — several steps that run in order; each runs to completion before the next starts:

steps:
  - name: preprocess
    mode: sequential
    image: preprocessor
    command:
      - preprocess
      - --input
      - /data
    volumes:
      - host: ./data
        container: /data
  - name: train
    mode: sequential
    image: trainer
    command:
      - train
      - --data
      - /data
    volumes:
      - host: ./data
        container: /data