YanokArtifact Transformation
Data conversion processes for seamless workflow execution
Overview
In the context of Molfar workflows, Artifact Transformation refers to the conversion processes needed to bridge data incongruencies between different nodes. This transformation happens when one task (node) in a workflow outputs data in one format, but the next task expects that data in a different format.
Key Benefits
- • Ensures data compatibility between nodes
- • Enables seamless workflow execution
- • Handles format mismatches automatically
- • Supports complex pipeline architectures
Transformation Types
Parameter to Artifact
Converting in-memory variables to persistent file formats
Parameters represent in-memory variables (like strings, numbers, or JSON objects) that exist temporarily in RAM. Artifacts are file-based inputs or outputs that live on disk, such as images, documents, or large datasets.
Use Case
This transformation is particularly relevant when transitioning from an "entry node" to a subsequent task that expects file-based input. The transformation handles scenarios where an in-memory parameter needs to be converted into a persistent file format.
Collection to Singular
Breaking collections into individual items for parallel processing
In cases where a task outputs a collection but the next task expects only a single item, a Collection to Singular transformation occurs. The workflow enters a "parallel processing" mode.
Collection Subtypes:
1. List Type
A simple list of values stored in a file. Each line represents a single item to be processed.
2. File Sorted Type
A set of files within a directory, named in a sorted and zero-padded manner (e.g., 0001.jpg, 0002.jpg).
3. JSONL (JSON List) Type
A list of JSON objects stored in a file where each line is a standalone JSON object.
Singular to Collection
Converting single items into collection formats
The Singular to Collection transformation occurs when a single output item needs to be transformed into a collection format. The transformation might involve "wrapping" the single item within a list structure.
Use Case
This transformation is essential when tasks are designed to handle collections, but only one item is provided. It enables flexibility in workflow design by adapting single items to collection-expecting tasks.
Key Benefits
Format Bridging
Seamlessly converts between in-memory and file-based data
Parallel Processing
Enables efficient collection handling and distribution
Flexible Architecture
Adapts to varying data requirements across tasks
These transformations ensure data flows smoothly across workflows regardless of format differences, enabling Molfar to handle complex pipelines with varying data requirements.