Advanced Topics

Remoting

Control distributed dxflow infrastructure from a single CLI using secure authentication and profile-based management

dxflow provides comprehensive remote management capabilities that allow you to control distributed server infrastructure from a single local CLI, enabling centralized operations across multiple environments.

Two-Instance Architecture: Remoting requires two dxflow instances - one running as a server (with API endpoints) and another as a client (CLI) that connects to control the remote server.

Overview

The dxflow CLI provides comprehensive remote management capabilities for distributed dxflow server instances. The remote architecture enables centralized control plane operations across distributed environments without requiring direct server access.

Key Benefits:

Multi-Server Control: Manage multiple dxflow servers from single CLI
No Direct Access: Control servers behind firewalls without SSH
Multi-Cloud Support: Uniform management across AWS, GCP, Azure
Profile Isolation: Separate dev/staging/production environments

How It Works

Remote Architecture

dxflow remoting requires two separate dxflow instances communicating over HTTP/HTTPS:

dxflow Client

Local CLI Instance

Sends CLI commands
Manages profile configs
Handles authentication
Converts commands to API calls

dxflow Server

Remote Target Instance

Runs REST API server
Executes workflow engine
Manages resources
Processes client requests

Remote dxflow Server

The target machine runs dxflow as a service:

# Start dxflow server on remote machine
dxflow boot up --daemon --port 8080

# Server exposes API endpoints
# http://your-server:8080/api/engine/ping
# http://your-server:8080/api/workflow/*
# http://your-server:8080/api/shell/*

Local dxflow Client

Your local CLI connects to the remote server:

# Configure connection to remote server
dxflow config set connection-address "http://your-server:8080"
dxflow config set connection-key "/path/to/server-key.pem"

# All commands now target remote server
dxflow engine ping
dxflow workflow list

API Translation

CLI commands are automatically converted to HTTP API calls:

# CLI command
dxflow workflow list

# Becomes HTTP request
GET /api/workflow/list
Authorization: Bearer <jwt-token>

The local CLI establishes secure connections to remote servers, handles authentication automatically, and provides transparent access to all remote functionality.

Authentication Process

Authentication Flow

The remoting system uses a sophisticated authentication process:

Connection Type Check: TCP/IP connections require authentication for remote server access
Token Discovery: CLI searches for existing connection-token in configuration
Two-Stage Validation:
- Quick Inspect: Local JWT validation (subject + expiration check)
- Deep Inspect: Remote server validation via /api/auth/inspect/
Challenge-Response Authentication: If token invalid/missing:
- CLI requests challenge from /api/auth/challenge/ (receives identity + nonce)
- CLI signs nonce with private key using RSA-PKCS1v15 + SHA256
- CLI sends signature to /api/auth/verify/ (receives new JWT token)
Token Persistence: New tokens automatically stored in profile configuration
Automatic Recovery: On 401 responses, tokens are automatically reset and regenerated

Authentication Methods

Method	Process	Use Case
Private Key	RSA challenge-response authentication with automatic token lifecycle	Production environments
Direct Token	Manual JWT token extraction from remote server instance	Quick setup/testing

Getting Started

Step 1: Configure Server Connection

Set the target remote server address:

# HTTPS for production (recommended)
dxflow config set connection-address "https://your-server.com"

# HTTP for development/testing
dxflow config set connection-address "http://192.168.1.100:8080"

Step 2: Set Up Authentication

Configure RSA private key for secure authentication:

# Use server's private key for authentication
dxflow config set connection-key "/path/to/server-private-key.pem"

Step 3: Verify Connection

Test the remote connection:

# Verify connectivity and authentication
dxflow engine ping

Success: Once authenticated, all dxflow commands automatically target the remote server.

Profile Management

Multiple Environment Support

dxflow uses configuration profiles to manage multiple remote servers independently:

Environment Isolation

Separate Configurations

Each profile maintains its own connection settings
Independent authentication credentials per profile
No cross-contamination between environments

Profile Switching

Flexible Operations

Switch between profiles with simple flags
Run commands on different environments simultaneously
Profile-specific command history and settings

Setting Up Profiles

Development Environment Setup

# Configure development server
dxflow --config-profile dev config set connection-address "http://dev-server:8080"
dxflow --config-profile dev config set connection-key "/keys/dev-key.pem"

# Test development connection
dxflow -C dev engine ping

Production Environment Setup

# Configure production server
dxflow --config-profile prod config set connection-address "https://prod-server.com"
dxflow --config-profile prod config set connection-key "/keys/prod-key.pem"

# Test production connection
dxflow -C prod engine ping

Multi-Cloud Environment Setup

# AWS environment
dxflow --config-profile aws config set connection-address "https://dxflow.us-west-2.amazonaws.com"
dxflow --config-profile aws config set connection-key "/keys/aws-key.pem"

# GCP environment
dxflow --config-profile gcp config set connection-address "https://dxflow.us-central1.gcp.com"
dxflow --config-profile gcp config set connection-key "/keys/gcp-key.pem"

# Azure environment
dxflow --config-profile azure config set connection-address "https://dxflow.eastus.azure.com"
dxflow --config-profile azure config set connection-key "/keys/azure-key.pem"

Authentication Process

RSA Challenge-Response Authentication

dxflow uses the same secure authentication system for remote connections as described in the Authentication documentation:

Automatic Authentication (Recommended)

# Configure private key for automatic authentication
dxflow config set connection-key "/path/to/server-private-key.pem"

How it works:

CLI requests authentication challenge from remote server
CLI signs challenge using your private key (RSA-PKCS1v15 + SHA256)
Server verifies signature and issues JWT token
Token stored automatically and refreshed as needed

Direct Token Configuration

# Generate token on remote server
dxflow engine token  # (run on remote server)

# Configure token locally
dxflow config set connection-token "eyJhbGciOiJSUzI1NiJ9..."

# Test connection
dxflow engine ping

Use cases:

Quick testing and debugging
Temporary access scenarios
Environments without private key access

Override with Environment Variables

# Temporary connection override
export CONNECTION_ADDRESS="https://temp-server.com"
export CONNECTION_KEY="/temp/keys/temp-key.pem"

# Run commands with overridden settings
dxflow engine ping
dxflow workflow list

Benefits:

No permanent configuration changes
Perfect for automation and CI/CD
Quick environment switching

Common Use Cases

Develop locally while testing on remote infrastructure:

# Deploy to development environment
dxflow -C dev workflow compose deploy ./my-workflow.yaml
dxflow -C dev workflow logs my-workflow --follow

# Promote through environments
dxflow -C staging workflow compose deploy ./my-workflow.yaml
dxflow -C prod workflow compose deploy ./my-workflow.yaml

Manage workloads across multiple cloud providers:

# Monitor all cloud environments
dxflow -C aws engine stat
dxflow -C gcp engine stat
dxflow -C azure engine stat

# Deploy to multiple clouds
dxflow -C aws workflow compose deploy ./production-workflow.yaml
dxflow -C gcp workflow compose deploy ./production-workflow.yaml

Integrate into CI/CD pipelines:

# Environment-based deployment
export CONNECTION_ADDRESS="https://staging-server.com"
export CONNECTION_KEY="/ci/keys/staging-key.pem"

dxflow workflow compose deploy ./build-artifacts/workflow.yaml
dxflow engine healthcheck || exit 1

Implement failover strategies:

# Check primary health and failover
if ! dxflow -C primary engine ping; then
  dxflow -C backup workflow compose deploy ./critical-services.yaml
fi

Configuration Storage

Profile Configuration Files

Each profile stores its configuration in a dedicated file:

Profile	Configuration File
Default	`~/.dxflow/default_config.yaml`
Custom	`~/.dxflow/{profile-name}_config.yaml`

Configuration Structure

# Example: ~/.dxflow/production_config.yaml
connection-address: "https://prod.company.com"  # Base URL (no /api suffix)
connection-key: "/secure/keys/prod-key.pem"     # RSA private key path
connection-token: "eyJhbGc..."                  # Auto-generated JWT token

Environment Variable Overrides

Override configuration temporarily using environment variables:

Environment Variable	Configuration Key
`CONNECTION_ADDRESS`	`connection-address`
`CONNECTION_KEY`	`connection-key`
`CONNECTION_TOKEN`	`connection-token`

Troubleshooting

Cannot connect to remote server

# 1. Verify server address configuration
dxflow config get connection-address

# 2. Test basic HTTP connectivity
curl -k "$(dxflow config get connection-address)/api/engine/healthcheck/"

# 3. Check DNS resolution and firewall rules
ping your-server.com
telnet your-server.com 443

Connection timeouts or network errors

Verify server is running and accessible
Check corporate firewall and proxy settings
Ensure proper HTTPS certificates are configured
Try HTTP instead of HTTPS for testing

Authentication failures or invalid tokens

# 1. Verify private key file permissions
chmod 600 /path/to/your-key.pem
ls -la /path/to/your-key.pem

# 2. Force token refresh
dxflow config unset connection-token
dxflow engine ping

# 3. Test with manual token (temporary)
dxflow config set connection-token "new-token-from-server"

Key-related problems

Ensure private key corresponds to public key on server
Verify key file is not corrupted or modified
Check system clock synchronization (affects token timing)

Commands targeting wrong environment

# Always verify active configuration
dxflow config get connection-address

# Use explicit profile specification
dxflow --config-profile correct-env engine ping

# List all available profiles
ls ~/.dxflow/*_config.yaml

Profile configuration conflicts

Ensure each profile has unique configuration
Check for typos in profile names
Verify profile-specific credentials are correct

Security Best Practices

Key Protection: Store private keys with chmod 600 permissions and separate keys per environment
HTTPS Only: Always use HTTPS for production environments
Profile Isolation: Create separate profiles for each team member and environment
Regular Audits: Monitor authentication logs and rotate keys periodically

Common Commands

Connection Management

# View current configuration
dxflow config
dxflow config get connection-address

# Update connection settings
dxflow config set connection-address "https://new-server.com"
dxflow config set connection-key "/path/to/new-key.pem"

# Test connection
dxflow engine ping
dxflow engine healthcheck

Profile Operations

# Use specific profile
dxflow --config-profile aws engine stat
dxflow -C gcp workflow list

# Profile-specific configuration
dxflow -C dev config set connection-address "https://dev-server.com"

# Force token refresh for profile
dxflow -C prod config unset connection-token
dxflow -C prod engine ping

Remote Management Ready: With proper configuration, all dxflow commands work seamlessly with remote servers, providing the same functionality as local installations while maintaining security and performance.

Tunneling

Create secure tunnels and proxy connections for exposing local services through WebSocket bridges

FAQs

Frequently Asked Questions about dxflow, covering common queries and troubleshooting tips to help users navigate