Docs
User Interface

Files & Data

The dxflow files and data management features allow you to efficiently handle input and output data within your workflows. This guide will help you understand how to manage files and data in dxflow.

The Files & Data section of the dxflow interface provides a comprehensive file management system for handling data assets, workflow inputs and outputs, and shared resources. This interface serves as a central hub for all your data management needs.

Overview

The Files interface offers enterprise-grade file management capabilities:

Centralized Storage

Unified Storage System

  • Single location for all workflow data
  • Organized file hierarchy
  • Metadata and tagging support
  • Advanced search capabilities

File Operations

Complete File Management

  • Upload, download, organize files
  • Bulk operations support
  • Drag-and-drop interface
  • Resume interrupted transfers

Data Integration

Pipeline Integration

  • Seamless workflow data handling
  • Automatic output collection
  • Input validation and preprocessing
  • Format conversion support

Access Control

Security & Sharing

  • Granular permissions system
  • Secure file sharing
  • Time-limited access links
  • Activity tracking and audit logs

Key Features

File Browser

The main file browser provides an intuitive interface similar to your operating system's file manager:

Efficient File Browsing

  • Folder Tree: Hierarchical folder structure with expandable/collapsible nodes
  • Breadcrumb Navigation: Quick navigation to parent directories
  • Search Functionality: Full-text search across filenames and metadata
  • Filter Options: Filter by file type, size, date modified, or custom tags

File Upload & Import

Multiple methods for uploading and importing files:

Drag & Drop

Intuitive Upload

  • Drag files or folders directly
  • Progress bars for all uploads
  • Automatic file type detection
  • Resume interrupted uploads

Traditional Upload

File Picker Dialog

  • Browse and select files
  • Bulk selection support
  • Preview before uploading
  • Upload queue management

CLI Integration

Command Line Upload

  • Direct CLI uploads
  • Batch upload with wildcards
  • Archive support (zip, tar, gz)
  • Directory synchronization

File Organization

Folder Management

  • Create Directories: Organize files in nested folder structures
  • Bulk Operations: Move multiple files and folders simultaneously
  • Templates: Predefined folder structures for common workflows
  • Smart Folders: Auto-organizing folders based on file type or metadata

Tagging & Metadata

  • Custom Tags: Add descriptive tags for better file organization
  • Metadata Fields: Store additional information like descriptions, authors, versions
  • Auto-tagging: Automatic tagging based on file content or naming patterns
  • Advanced Search: Search using tags and metadata combinations

File Sharing & Collaboration

Team Collaboration

  • Share files with team members
  • Set granular permissions (read, write, admin)
  • Time-limited access with expiration dates
  • Activity tracking for shared files

Data Pipeline Integration

Workflow Data Management

Input Data Handling

Data Preparation

  • Designated input folders
  • Automatic validation
  • Type conversion support
  • Schema validation

Output Collection

Result Management

  • Automatic output collection
  • Organized by workflow/timestamp
  • Result archiving
  • Export capabilities

Data Lineage Tracking

  • Source Tracking: Track data origins and transformations
  • Dependency Mapping: Visualize data dependencies across workflows
  • Impact Analysis: Understand downstream effects of data changes
  • Audit Trails: Complete history of data access and modifications

Advanced Features

Archive Management

  • Create Archives: Package files into compressed archives
  • Extract Archives: Support for zip, tar, and other formats
  • Smart Compression: Automatic compression for storage
  • Selective Extraction: Extract specific files from archives

Getting Started

Step 1: Access File Management

After logging in, navigate to the "Files & Data" section from the main menu.

Step 2: Upload Your First Files

  1. Navigate to desired folder or create a new one
  2. Choose upload method:
    • Drag and drop files into the browser window
    • Click "Upload Files" button to use file picker
    • Use CLI: dxflow object fs upload <local-path> <remote-path>
  3. Monitor upload progress in the status panel
  4. Organize files using folders and tags

Step 3: Share Files

  1. Select the file(s) you want to share
  2. Click the "Share" button
  3. Configure sharing settings:
    • Choose recipients (internal users or external email addresses)
    • Set permissions (view, edit, admin)
    • Add expiration date if needed
    • Enable/disable password protection
  4. Send sharing invitation or copy the share link

File Operations Reference

API Endpoints

The file management system provides comprehensive REST API endpoints:

# File operations
GET    /api/object/fs/                    # List files and directories
POST   /api/object/fs/                    # Create files and directories
PUT    /api/object/fs/upload/             # Upload files
PUT    /api/object/fs/rename/             # Rename/move files
DELETE /api/object/fs/                    # Delete single file
DELETE /api/object/fs/batch/              # Delete multiple files

CLI Commands

# Core file operations
dxflow object fs list                    # List files and directories
dxflow object fs upload <local> <remote> # Upload files
dxflow object fs download <remote> <local> # Download files
dxflow object fs create <path>           # Create directory
dxflow object fs rename <old> <new>      # Rename files/folders
dxflow object fs delete <path>           # Delete files/folders

# Advanced operations
dxflow object fs share <path>            # Create share link
dxflow object fs zip <source> <target>   # Create zip archive
dxflow object fs unzip <archive> <dest>  # Extract archive

Batch Operations

  • Multi-select: Hold Ctrl/Cmd while clicking to select multiple files
  • Select Range: Hold Shift to select a range of files
  • Select All: Ctrl/Cmd+A to select all files in current folder
  • Context Menu: Right-click for context-sensitive operations

Best Practices

File Organization

  • Use descriptive folder names and maintain consistent naming conventions
  • Leverage tags and metadata for better searchability
  • Regular cleanup of temporary and outdated files
  • Implement folder templates for recurring project structures
Pro Tip: The Files & Data interface integrates seamlessly with all dxflow workflows, automatically organizing inputs and outputs for easy access and management.

The Files & Data interface provides everything you need to manage your workflow data efficiently, from simple file storage to complex data pipeline integration.

Apps & Pipelines

Comprehensive platform for managing containerized applications and complex workflows using Docker Compose integration

Terminal Access

The dxflow terminal provides a command-line interface to interact with the dxflow engine, allowing you to execute commands, manage workflows, and access system functionalities directly from your terminal.