File Sources

File sources define where MediaMoth reads input files, writes output files, and loads configuration files. Whether your media lives on the local filesystem or a remote server, file sources provide a consistent way to reference files across different storage locations.

Overview

When creating jobs or configuring pipelines, you'll specify file paths for:

• Input files - Media to process
• Output files - Where to save processed media
• Configuration files - Encoding presets, worker configurations

MediaMoth supports multiple file source types, allowing you to store files wherever makes sense for your infrastructure.

Supported Source Types

MediaMoth currently supports two file source types:

Source Type	Use Case	Authentication
Filesystem (Local)	Files on the same machine as the worker	None required
Rsync (Remote)	Files stored on a remote server	SSH key authentication

Filesystem Source

When to Use

Use filesystem sources when:

• Files are stored locally on the same machine as the worker
• Workers have direct filesystem access to the files
• You want the fastest possible I/O performance

Path Syntax

Files:

/path/to/file.mp4
/home/user/videos/input.mkv
/mnt/storage/media/episode-01.mov

Directories:

/path/to/directory/
/home/user/videos/
/mnt/storage/output/

TIP

Directory paths must end with a trailing slash (/) to indicate they represent directories rather than files.

Examples

Single file input:

/var/media/input/source-video.mp4

Output to specific directory:

/var/media/output/processed/

Configuration file:

/etc/mediamoth/configs/h264-preset.json

Rsync Source (Remote)

When to Use

Use rsync sources when:

• Files are stored on a remote server
• You have SSH access to the remote host
• You want to centralize storage across multiple workers

Path Syntax

Rsync sources use SSH-style remote path syntax:

Files:

user@hostname:/path/to/file.mp4
[email protected]:/data/input/video.mkv
[email protected]:/mnt/storage/source.mov

Directories:

user@hostname:/path/to/directory/
[email protected]:/data/output/
[email protected]:/mnt/storage/processed/

TIP

Like filesystem sources, directory paths must end with a trailing slash (/).

Authentication

Rsync sources require SSH key-based authentication configured between the worker machine and the remote host. Password authentication is not supported.

For details on setting up SSH keys and remote storage infrastructure, see External Storage Setup.

Examples

Remote input file:

[email protected]:/videos/raw/input-4k.mp4

Remote output directory:

deploy@backup-server:/processed/2024-01-15/

Remote configuration:

admin@config-server:/configs/encoding/high-quality.json

Best Practices

Choosing the Right Source Type

Use Filesystem when:

• ✅ Workers have local storage
• ✅ Maximum performance is critical
• ✅ Files are worker-specific
• ✅ You want to avoid network overhead

Use Rsync when:

• ✅ Centralizing storage across multiple workers
• ✅ Workers are stateless/ephemeral
• ✅ You need shared access to media files
• ✅ Backup and recovery are important

Performance Considerations

Filesystem Performance:

• Fastest option - direct I/O with no network overhead
• Consider using fast local storage (SSD/NVMe) for temporary files
• For large files, filesystem sources can use hard links for optimization

Rsync Performance:

• Network latency affects transfer times
• Rsync uses compression for directory operations
• Consider network bandwidth when processing large files
• Multiple workers accessing remote storage may bottleneck

Directory Operations

Some operations work with entire directories rather than individual files:

Syncing directories (filesystem):

/mnt/batch-input/2024-01-15/

Syncing directories (rsync):

backup@server:/archive/processed/2024-01-15/

Directory operations are useful for:

• Batch processing multiple files
• Archiving job outputs
• Backing up processed media

Common Patterns

Local Input, Remote Output

Process local files and save results to centralized storage:

Input:  /var/media/input/raw-footage.mp4  (Filesystem)
Output: backup@storage:/archive/processed.mp4  (Rsync)

Remote Input, Local Processing, Remote Output

Fetch from remote storage, process locally, save back:

Input:  media@source:/videos/episode-05.mkv  (Rsync)
Output: media@dest:/processed/episode-05.mp4  (Rsync)

Next Steps

• Video Encoding Presets - Reference config files using file sources
• Creating Pipelines - Build workflows that use different source types
• External Storage Setup - Configure SSH and remote storage infrastructure
• Running Jobs - Create jobs with input and output file sources