Core Concepts

This page explains how dxflow is put together: its layered architecture, the two deployment patterns, the components every engine exposes, and the permission model.

Architecture

dxflow follows a "compute-first" design: the engine runs close to your workloads, so there's no central control layer moving data around. It's organized in four layers:

Deployment patterns

dxflow supports two patterns:

Node-Embedded is the most common pattern — it eliminates network hops between the engine and the compute. Because dxflow is a self-contained binary, it runs alongside any scheduler you already have rather than replacing it:

• Slurm / PBS / LSF: start dxflow from a prolog script when a job allocation begins
• Kubernetes / Nomad: run the engine as a DaemonSet or side-car
• Docker Compose / Podman: include the dxflow container in your compose file
• Spark / Ray / AWS Batch: bundle the engine via user-data on each executor

In the Federated Master pattern, a single instance acts as a control plane that orchestrates the agents running on each node.

Core components

Every engine — regardless of deployment pattern — exposes the same interfaces:

It manages four kinds of resources:

• Workflows — containerized apps defined in Docker Compose-style YAML, with full lifecycle control, resource limits (CPU/memory/GPU), networking, and volumes. See Workflows.
• Shells — interactive terminal sessions on the engine host, with TTY support and live reconnect. See Shells.
• Artifacts — file storage on the engine volume: upload/download, directories, zip/unzip, and signed sharing links. See Artifacts.
• Bridges & proxies — connect engines and expose internal services through authenticated WebSocket tunnels. See Tunneling.

Security model

Authentication

dxflow uses RSA key-pair authentication: generate a key, register its public key with the engine, and sign in with the private key. Multiple keys can be registered, each with its own permissions. See Authentication.

Permissions

Permissions are composable — combine functional permissions with modifiers to build custom access levels.

Functional (combinable): SHELL, ARTIFACT, WORKFLOW, PROXY, BRIDGE, RUNTIME

Modifiers:

• READ_ONLY — blocks create/modify/delete; viewing and monitoring only. Combinable with any functional permission.
• MASTER — full administrative access, including key and user management. Includes all other permissions.

Examples:

Network

All communication is TLS-encrypted, with automatic certificate generation. Local communication uses Unix domain sockets, and the engine works alongside your existing firewall and network policies.