Core Concepts

Understanding these core concepts and architecture patterns will help you leverage dxflow's full potential for distributed computing and workflow orchestration.

Architecture Overview

dxflow is designed as a lightweight, distributed computing platform that enables seamless orchestration of data and compute workflows across heterogeneous computing environments.

Layered Architecture

dxflow follows a layered, modular architecture designed for flexibility and scalability:

This layered approach allows dxflow to integrate seamlessly with existing infrastructure while providing a consistent interface across diverse environments.

Deployment Patterns

dxflow supports two primary deployment patterns, each optimized for different use cases and organizational needs:

Node-Embedded Deployment

The Node-Embedded pattern places a dxflow engine directly on every compute target—whether it's an EC2 spot instance, an on-premises Slurm node, or a Docker container on a lab workstation.

Key Characteristics

Integration with Schedulers

Because the runtime is embedded at the node level, dxflow cohabits peacefully with any resource scheduler already present:

• Slurm / PBS / LSF: The prolog script starts dxflow when a job allocation begins
• Kubernetes / Nomad: Deploy the agent as a DaemonSet or side-car
• Docker Compose / Podman: Include the dxflow container in the same docker-compose.yml
• Spark / Ray / AWS Batch: Each executor or EC2 instance bundles the agent via user-data

This design mirrors the side-car model popularised by Dapr: place a lightweight process next to every workload to provide cross-cutting capabilities.

Federated Master Deployment

In the federated master deployment, a single dxflow instance acts as a control plane for the entire fleet of compute nodes. This master instance orchestrates the dxflow agents running on each node or container, providing a centralized interface for managing and monitoring computational tasks.

Core Components

Engine

The dxflow engine is the heart of each compute node, providing:

Unified Interface Components

Regardless of deployment pattern, every dxflow engine provides a consistent set of interfaces:

Access Points

Each engine immediately exposes multiple access methods:

API

http://localhost:<port>/api — REST

CLI

dxflow <cmd> — local CLI passthrough

Console

http://localhost:<port>/console — web console

Workflows

Workflows are containerized applications managed through industry-standard Docker Compose:

Key Features

• Docker Compose Integration: Use your existing compose files
• Lifecycle Management: Full control over container states
• Resource Constraints: CPU, memory, and GPU limits
• Network Management: Custom networks and port mappings
• Volume Handling: Persistent data and bind mounts
• Multi-container Coordination: Complex application stacks

Shell Sessions

Secure, interactive terminal access for debugging and administration:

Session Features:

• Multi-session Support: Create and manage multiple concurrent shells
• Cross-platform Compatibility: Consistent experience on Linux, macOS, and Windows
• Session Persistence: Long-running sessions survive network disconnections
• Secure Access: Authentication-protected with audit logging

Common Use Cases:

# System debugging
dxflow shell create debug-session
dxflow shell connect debug-session

# Interactive development
dxflow shell create dev-environment
# Install dependencies, test code, debug issues

Object Storage (File System)

Integrated file management with enterprise-grade features:

Proxy Services

Advanced network proxy capabilities for secure service exposure:

Proxy Types

• HTTP/HTTPS Proxy: Web application and API exposure
• TCP Proxy: Database and service connections
• WebSocket Proxy: Real-time application support
• Load Balancing: Distribute traffic across multiple backends

Security Features

• Access Control: IP allowlists and authentication
• TLS Termination: SSL certificate management
• Rate Limiting: Protect against abuse
• Audit Logging: Track all proxy access

Bridges

Connect and orchestrate multiple dxflow instances for distributed computing:

Bridge Capabilities:

• Multi-node Orchestration: Coordinate workloads across engines
• Resource Federation: Share compute and storage resources
• Hybrid Cloud Support: Bridge on-premise and cloud environments
• Load Distribution: Optimize resource utilization

Security Model

Authentication & Authorization

dxflow implements a comprehensive security model:

RSA Key-Pair Authentication

• Public Key Infrastructure: Secure, scalable authentication
• Key Management: Generate, register, and rotate keys easily
• Multi-key Support: Different keys for different access levels

Composable Permission System

dxflow implements a flexible permission model where permissions can be combined to create custom access levels:

Permission Examples

Permissions: SHELL + OBJECT + WORKFLOW

Can:
- Create and connect to shell sessions
- Upload/download files and manage directories
- Create, start, stop workflows
- Execute commands in containers

Cannot:
- Manage proxy services or bridges
- Access Docker platform operations
- Perform administrative tasks

Permissions: WORKFLOW + SHELL + READ_ONLY

Can:
- View workflow status and logs
- List shell sessions
- Monitor system health
- Access read-only endpoints

Cannot:
- Create or modify workflows
- Create shell sessions or execute commands
- Upload/download files

Permissions: OBJECT + BRIDGE

Can:
- Full file system access
- Connect to remote engines
- Transfer files between engines
- Generate sharing links

Cannot:
- Create workflows or containers
- Access shell sessions
- Manage proxy services

Permissions: MASTER

Can:
- All system operations
- User and key management
- Engine configuration
- Complete administrative control

Note: MASTER includes all other permissions

Network Security

• TLS Encryption: All communications secured with industry-standard encryption
• Certificate Management: Automatic certificate generation and renewal
• Network Isolation: Unix domain sockets for local communication
• Firewall Integration: Works with existing network security policies

Best Practices

Engine Deployment

• Resource Planning: Size engines based on expected workloads
• High Availability: Deploy multiple engines for redundancy
• Monitoring: Set up health checks and alerting
• Backup: Regular configuration and data backups

Workflow Design

• Resource Limits: Always specify CPU and memory constraints
• Health Checks: Implement container health checks
• Graceful Shutdown: Handle SIGTERM signals properly
• Logging: Use structured logging for better debugging

Security Hardening

• Least Privilege: Grant minimum necessary permissions
• Key Rotation: Regularly rotate authentication keys
• Audit Logging: Monitor all access and operations
• Network Segmentation: Use firewalls and VPNs appropriately

Advanced Topics

Integration Patterns

• CI/CD Integration: Automate deployments with dxflow APIs
• Monitoring Integration: Connect with Prometheus, Grafana, etc.
• Service Mesh: Deploy alongside Istio or Linkerd
• GitOps: Manage configurations with Git workflows

Scaling Strategies

• Horizontal Scaling: Add more engine nodes as needed
• Vertical Scaling: Increase resources on existing nodes
• Auto-scaling: Dynamic scaling based on workload demand
• Edge Deployment: Distribute engines to edge locations

This comprehensive understanding of dxflow concepts and architecture will enable you to design and implement robust, scalable distributed computing solutions.