Files & Data

The Files & Data section of the dxflow interface provides a comprehensive file management system for handling data assets, workflow inputs and outputs, and shared resources. This interface serves as a central hub for all your data management needs.

Overview

The Files interface offers enterprise-grade file management capabilities:

• Centralized Storage: Unified storage system for all workflow data
• File Operations: Upload, download, organize, and share files
• Data Pipeline Integration: Seamless integration with workflow data inputs and outputs
• Access Control: Granular permissions and secure file sharing
• Version Control: Track file changes and maintain data lineage

Key Features

1. File Browser

The main file browser provides an intuitive interface similar to your operating system's file manager:

Navigation

• Folder Tree: Hierarchical folder structure with expandable/collapsible nodes
• Breadcrumb Navigation: Quick navigation to parent directories
• Search Functionality: Full-text search across filenames and metadata
• Filter Options: Filter by file type, size, date modified, or custom tags

File Preview

• Built-in Viewers: Preview images, text files, JSON, CSV, and code files
• Metadata Display: File size, creation date, modification date, and permissions
• Thumbnail Generation: Automatic thumbnails for images and documents
• Quick Actions: Copy, move, rename, delete files directly from the browser

2. File Upload & Import

Multiple Upload Methods

Drag & Drop Interface

• Drag files or entire folders directly into the browser
• Progress bars for individual and batch uploads
• Automatic file type detection and validation
• Resume capability for interrupted uploads

Traditional Upload Dialog

• Browse and select files using the file picker
• Bulk selection support with Ctrl/Cmd+click
• Preview selected files before uploading
• Upload queue management

CLI Integration

• Upload files directly from command line using dxflow CLI
• Batch upload with wildcard patterns
• Compressed archive support (zip, tar, tar.gz)
• Sync local directories with remote storage

3. File Organization

Folder Management

• Create Directories: Organize files in nested folder structures
• Bulk Operations: Move multiple files and folders simultaneously
• Templates: Predefined folder structures for common workflows
• Smart Folders: Auto-organizing folders based on file type or metadata

Tagging & Metadata

• Custom Tags: Add descriptive tags for better file organization
• Metadata Fields: Store additional information like descriptions, authors, versions
• Auto-tagging: Automatic tagging based on file content or naming patterns
• Advanced Search: Search using tags and metadata combinations

4. File Sharing & Collaboration

Sharing Options

Internal Sharing

• Share files with team members and collaborators
• Set granular permissions (read, write, admin)
• Time-limited access with expiration dates
• Activity tracking for shared files

External Sharing

• Generate secure, time-limited download links
• Password-protected sharing for sensitive data
• Upload-only shares for collecting external data
• Customizable sharing notifications

Collaboration Features

• File Comments: Add comments and annotations to files
• Change Notifications: Get notified when shared files are modified
• Version History: Track all changes and revert to previous versions
• Collaborative Editing: Real-time editing for supported file types

5. Data Pipeline Integration

Workflow Data Management

Input Data Handling

• Designated input folders for workflow data
• Automatic data validation and preprocessing
• Data type conversion and format normalization
• Schema validation for structured data

Output Collection

• Automatic collection of workflow outputs
• Organized storage by workflow and execution timestamp
• Result archiving and compression
• Export capabilities to various formats

Data Lineage Tracking

• Source Tracking: Track data origins and transformations
• Dependency Mapping: Visualize data dependencies across workflows
• Impact Analysis: Understand downstream effects of data changes
• Audit Trails: Complete history of data access and modifications

6. Advanced Features

Compression & Archives

• Create Archives: Package files and folders into compressed archives
• Extract Archives: Extract zip, tar, and other archive formats
• Smart Compression: Automatic compression for optimal storage
• Selective Extraction: Extract specific files from archives

File Synchronization

• Sync Folders: Keep local and remote folders in sync
• Conflict Resolution: Handle file conflicts during synchronization
• Selective Sync: Choose specific files and folders to synchronize
• Offline Access: Cache files for offline access and editing

Data Export & Import

• Bulk Export: Export entire folder structures
• Format Conversion: Convert between different file formats
• API Integration: Programmatic access via REST API
• Third-party Integrations: Connect with cloud storage providers

Getting Started

1. Access File Management

After logging in, navigate to the "Files & Data" section from the main menu.

2. Upload Your First Files

1. Navigate to desired folder or create a new one
2. Choose upload method:
   • Drag and drop files into the browser window
   • Click "Upload Files" button to use file picker
   • Use CLI: dxflow object fs upload <local-path> <remote-path>
3. Monitor upload progress in the status panel
4. Organize files using folders and tags

3. Share Files

1. Select the file(s) you want to share
2. Click the "Share" button
3. Configure sharing settings:
   • Choose recipients (internal users or external email addresses)
   • Set permissions (view, edit, admin)
   • Add expiration date if needed
   • Enable/disable password protection
4. Send sharing invitation or copy the share link

File Operations Reference

API Endpoints Reference

The file management system provides the following REST API endpoints:

# Core file operations
GET    /api/object/fs/                    # List files and directories
POST   /api/object/fs/                    # Create files and directories
PUT    /api/object/fs/upload/             # Upload files
PUT    /api/object/fs/rename/             # Rename/move files
DELETE /api/object/fs/                    # Delete single file
DELETE /api/object/fs/batch/              # Delete multiple files

# Download operations
GET    /api/object/fs/download/           # Download files
GET    /api/object/fs/presigned/download/ # Presigned download URLs

# Advanced operations
PUT    /api/object/fs/zip/                # Create zip archives
PUT    /api/object/fs/unzip/              # Extract zip archives
PUT    /api/object/fs/share/              # Generate secure sharing tokens
PUT    /api/object/fs/presigned/upload/   # Presigned upload URLs

CLI Commands

# CLI commands for file operations
dxflow object fs list                    # List files and directories
dxflow object fs upload <local> <remote> # Upload files
dxflow object fs download <remote> <local> # Download files
dxflow object fs create <path>           # Create directory
dxflow object fs rename <old> <new>      # Rename files/folders
dxflow object fs delete <path>           # Delete files/folders
dxflow object fs share <path>            # Create share link
dxflow object fs zip <source> <target>   # Create zip archive
dxflow object fs unzip <archive> <dest>  # Extract archive

Batch Operations

• Multi-select: Hold Ctrl/Cmd while clicking to select multiple files
• Select Range: Hold Shift to select a range of files
• Select All: Ctrl/Cmd+A to select all files in current folder
• Context Menu: Right-click for context-sensitive operations

Best Practices

Organization

• Use descriptive folder names and maintain consistent naming conventions
• Leverage tags and metadata for better searchability
• Regular cleanup of temporary and outdated files
• Implement folder templates for recurring project structures

Security

• Use appropriate sharing permissions and time limits
• Regular audit of file sharing and access permissions
• Encrypt sensitive data before upload
• Implement backup strategies for critical data

Performance

• Compress large files before upload
• Use batch operations for multiple files
• Regular cleanup of unnecessary file versions
• Monitor storage usage and optimize file organization

The Files & Data interface provides everything you need to manage your workflow data efficiently, from simple file storage to complex data pipeline integration.