Advanced Topics

Streaming

Enable high-performance data streaming for large datasets and real-time updates

dxflow automatically optimizes data delivery using streaming or buffered responses depending on your needs, ensuring fast response times and efficient resource usage.

Smart Performance: dxflow automatically chooses the best response method - streaming for large data or real-time updates, buffering for small quick responses.

How It Works

Automatic Mode Selection

dxflow intelligently chooses between two response modes:

Streaming Mode

For Large Data

Sends data as it becomes available
Immediate response start
Memory efficient for big lists
Perfect for file listings and logs

Buffered Mode

For Quick Responses

Waits for complete data
Single fast transmission
Better for small queries
Standard HTTP behavior

When Streaming Activates

Streaming mode activates automatically for:

Large file directory listings
Workflow execution logs
Real-time status updates

Using Streaming

Enable Streaming Manually

Add the stream parameter to any API request:

# Enable streaming for file listings
curl "http://localhost/api/object/fs/?stream=true"

# Enable streaming for workflow logs
curl "http://localhost/api/workflow/logs?stream=true"

In the Web Interface

The web console automatically uses streaming for:

Large directory browsing
File search results
Bulk file operations
Archive contents

Performance Benefits

Streaming vs Buffered

Aspect	Streaming	Buffered
Response Start	Immediate	After completion
Memory Usage	Very Low	Full dataset size
Best For	Large data, Live updates	Small queries, Simple data

Real-World Performance

Example: A directory with 10,000 files starts showing results in ~50ms with streaming vs ~2000ms with buffering, while using 95% less server memory.

Common Use Cases

File Management

# Large directory listings automatically use streaming
dxflow object ls /large-directory

# File uploads show real-time progress
dxflow object upload /path/to/large-file

Workflow Monitoring

# Live workflow logs automatically stream
dxflow workflow logs my-workflow --follow

# Workflow status updates in real-time
dxflow workflow status

System Administration

# Network connectivity testing with streaming
dxflow ping

# Bridge connection listings
dxflow bridge list

Best Practices

When to Use Streaming

File directory with >100 items
Live log monitoring
Long-running operations
Real-time dashboards
Large data exports

When to Use Buffered

Quick status checks
Small configuration queries
Single file information
Simple API calls
Mobile applications

Response Format

Both streaming and buffered responses use the same structured JSON array format with three types of chunks:

Chunk Types

Status Chunk

Always First

Contains HTTP status code and message
Indicates success or error state
Required in every response

Total Chunk

Optional Count

Provides total number of items
Helps with progress tracking
Useful for pagination

Entity Chunk

Actual Data

Contains the real response data
Multiple entities for lists
Each item wrapped individually

Example Response Structure

[
  {
    "kind": "status",
    "payload": {
      "code": 200,
      "message": "OK"
    }
  },
  {
    "kind": "total",
    "payload": 1500
  },
  {
    "kind": "entity",
    "payload": {
      "name": "file1.txt",
      "size": 1024,
      "type": "file"
    }
  },
  {
    "kind": "entity",
    "payload": {
      "name": "file2.txt",
      "size": 2048,
      "type": "file"
    }
  }
]

Response Flow

Progressive Delivery

Status chunk sent immediately
Total chunk sent if known
Entity chunks sent as processed
Each chunk flushed to client instantly

Payload Contents

The payload field contains different data depending on the chunk type:

Chunk Kind	Payload Structure	Example
status	`{code: number, message: string}`	`{code: 200, message: "OK"}`
total	`number`	`1500`
entity	`object`	`{id: 1, name: "item"}`

Error Handling

Error responses maintain the same structure:

[
  {
    "kind": "status",
    "payload": {
      "code": 404,
      "message": "File not found"
    }
  }
]

Client Friendly: The consistent structure works identically in both streaming and buffered modes, so your applications only need one parsing approach.

Troubleshooting

If Streaming Seems Slow

Check network conditions - Streaming needs stable connections
Verify client support - Some tools don't handle streaming well
Try buffered mode - Add ?stream=false to compare
Check server load - High activity can affect streaming performance

Common Issues

Response appears incomplete:

Client may not support chunked encoding
Try with curl or modern browsers
Use ?stream=false as fallback

Streaming not working as expected:

Verify endpoint supports streaming
Check client HTTP library compatibility
Use curl to test streaming behavior directly

Authentication

Secure authentication system using RSA key-pair authentication for accessing dxflow engine functionality

Tunneling

Create secure tunnels and proxy connections for exposing local services through WebSocket bridges