Core Concepts
This page explains how dxflow is put together: its layered architecture, how it sits alongside your existing infrastructure, the resources every engine manages, and the permission model.
Architecture
dxflow follows a compute-first design: the engine runs close to your workloads, so no central control layer moves data around. It's organized in four layers:
Infrastructure
Physical or virtual machines, cloud instances, edge devices
Engine
The dxflow daemon — core services, CLI, and APIs
Runtime
Docker, Podman, Singularity, or Apptainer
Application
Your workloads, models, and data pipelines
Deployment
Every compute node runs its own engine, with no control layer in between. That keeps the engine on the same host as the compute, so there are no network hops for logs, stats, or shells.
Because dxflow is a self-contained binary, it runs alongside any scheduler you already have rather than replacing it:
- Slurm / PBS / LSF — start dxflow from a prolog script when a job allocation begins
- Kubernetes / Nomad — run the engine as a DaemonSet or side-car
- Spark / Ray / AWS Batch — bundle the engine via user-data on each executor
To drive several engines from one terminal, keep a profile per host rather than a coordinating server.
Interfaces
Every engine exposes the same three interfaces over the same API:
| Interface | Access | Purpose |
|---|---|---|
| REST API | /api on the configured port | Programmatic access, with streaming responses for live data |
| CLI | dxflow <cmd> | Local and remote command execution, scriptable for CI/CD |
| Web console | / on the configured port | Visual management and live dashboards |
Resources
An engine manages four kinds of resource:
- Workflows — containerized apps defined in dxflow YAML, with lifecycle control, resource limits (CPU/memory/GPU), ports, and volumes. See Workflows.
- Shells — interactive terminal sessions on the engine host, with TTY support and live reconnect. See Shells.
- Artifacts — file storage on the engine volume: upload/download, directories, zip/unzip, and signed sharing links. See Artifacts.
- Links — publish a workflow step on a licensed subdomain, served over HTTPS by the engine. See Links.
Identities
Workflows, shells, and agent sessions are addressed by an identity: 3 to 12 characters, opening with two letters, then letters and digits. Leave it out and the engine generates one.
One you supply is folded to that shape — lowercased, everything outside a-z0-9 dropped, cut to 12 — and every call accepts either form afterwards:
dxflow workflow create hub://jupyter --identity open-ui # creates openui
dxflow workflow start open-ui # either form works
Folding is not reversible, so open-ui and Open UI both name openui, and a second create answers 409. Input that folds to nothing usable is refused with the reason — 2024-run folds to 2024run, which does not open with two letters.
Artifacts are the exception: an artifact identity is a path, kept as given.
Permission model
Access is granted per key, not per user. Each entry in ~/.dxflow/authorized_keys carries its own set of permissions, and a pairing session carries whatever dxflow engine pair --permissions gave it.
Functional (combinable): SHELL, ARTIFACT, WORKFLOW, RUNTIME, AGENT
Modifiers:
READ_ONLY— blocks create, modify, and delete; viewing and monitoring only. Combinable with any functional permission.MASTER— full administrative access, including key management and engine restart.
| Role | Permissions | Can do |
|---|---|---|
| Developer | SHELL + ARTIFACT + WORKFLOW | Shells, files, and workflows — no runtime or admin |
| Read-only monitor | WORKFLOW + SHELL + READ_ONLY | View status and logs; no changes |
| Administrator | MASTER | Everything, including key management |
SHELL + ARTIFACT + RUNTIME + WORKFLOW + AGENT — full functionality without admin privileges or read-only restrictions.Which of these permissions the engine can serve at all is decided by the active license.
Transport
Local CLI commands reach the engine over a Unix domain socket, where OS file permissions are the trust boundary and no token is involved. Network listeners are opt-in: --http and --https are what open a port, and every connection arriving on one authenticates. Use --https for anything beyond localhost. See Authentication.