How to orchestrate a full beVault project pipeline

Overview

This guide explains how to design and implement a global orchestration state machine that drives the complete end-to-end data pipeline for a beVault project — from raw extraction to information mart delivery.

The orchestration relies on States (beVault's workflow engine). It ties together:

Custom extraction and export state machines (fully project-specific)
Custom load-dv-[source] state machines (bridge between extraction and beVault-generated loaders)
beVault-generated state machines for staging load, snapshot computation, and information mart generation

Pipeline Architecture

The diagram below shows the high-level structure of a global orchestration pipeline for a project with two source systems.

The pipeline is composed of four sequential phases:

Phase	Type	State Machines
1. Extract & Load Staging	Parallel (per source)	Custom `extract-[source]` + Custom `load-dv-[source]`
2. Load Snapshots	Parallel	beVault-generated `[proj]_[env]_Load_Snapshot_[name]`
3. Generate Information Marts	Parallel	beVault-generated `[proj]_[env]_generate_[name]`
4. Export	Sequential or Parallel	Custom `export`

Building Blocks

1. Custom Extraction State Machines (`extract-[source]`)

These are fully custom state machines responsible for pulling data from a source system (API, database, file, etc.) and writing it to the staging area (level 1) in the data warehouse.

💡 For implementation details, refer to the dedicated how-tos:

Error handling during extraction

Extraction is the step most exposed to external failures: a source system may be temporarily unavailable, its API may be down, or its data structure may have unexpectedly changed. To avoid a single failing source from blocking the rest of the pipeline, always add a Catch block on extraction tasks and redirect to a dummy Pass state.

This is the pattern used in the global-process example:

JSON

{
  "Type": "Task",
  "Resource": "Production-ExecuteStateMachine",
  "Parameters": {
    "stateMachineArn": "extract-xxx"
  },
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "ResultPath": "$.error",
      "Next": "dummy-xxx"
    }
  ],
  "Next": "load-dv-xxx"
}

The error is captured in $.error for observability (visible in the execution history in States), but execution continues gracefully. The downstream load-dv-[source] step is simply skipped for that source.

⚠️ If the extraction is skipped due to an error, the corresponding staging tables will not be refreshed. The subsequent Load_Staging state machines will still run but will load no new data. Depending on your business requirements, you may want to add alerting or reporting on $.error to be notified of silent extraction failures.

2. Custom Load DV State Machine (`load-dv-[source]`)

This custom state machine acts as the bridge between your extraction step and the beVault-generated Load_Staging state machines. It must be created once per source system.

Its role is to:

Define (or retrieve) the list of staging tables belonging to the source system
Iterate over each staging table and call the corresponding beVault-generated Load_Staging state machine

State Machine Naming Convention for `Load_Staging`

When you deploy a version in beVault (metaVault), a state machine is automatically generated for each staging table of each data package. The naming format is:

[project]_[environment]_Load_Staging_[staging table name]

For example, for a project default, environment Production, and a staging table odoo14_companies, the state machine name will be:

default_Production_Load_Staging_odoo14_companies

Structure of `load-dv-[source]`

The state machine starts with a Pass state that defines the list of data packages to process, followed by a Map state that iterates over them.

JSON

{
  "StartAt": "get-data-packages",
  "TimeoutSeconds": 1800,
  "States": {
    "get-data-packages": {
      "Type": "Pass",
      "Result": {
        "name": [
          "companies",
          "contacts",
          "employees"
        ]
      },
      "ResultPath": "$.data_packages",
      "Next": "process-load"
    },
    "process-load": {
      "Type": "Map",
      "MaxConcurrency": 1,
      "ItemsPath": "$.data_packages.name",
      "ItemSelector": {
        "data_package.$": "$$.Map.Item.Value"
      },
      "ResultPath": "$.loaded",
      "ItemProcessor": {
        "StartAt": "load",
        "States": {
          "load": {
            "Type": "Task",
            "Resource": "Production-ExecuteStateMachine",
            "TimeoutSeconds": 600,
            "Parameters": {
              "stateMachineArn.$": "States.Format('default_Production_Load_Staging_odoo14_{}', $.data_package)"
            },
            "ResultPath": "$.taskResult",
            "End": true
          }
        }
      },
      "End": true
    }
  }
}

Key points:

get-data-packages (Pass state): Injects a static list of data package names into the execution context. This can also be replaced by a Task state that dynamically queries the list from the meta schema.
process-load (Map state): Iterates over each data package name. MaxConcurrency: 1 ensures sequential loading — increase this value to parallelize.
States.Format(): Dynamically builds the state machine ARN by combining the fixed prefix (default_Production_Load_Staging_odoo14_) with the current data package name.
Resource: "Production-ExecuteStateMachine": Uses the beVault ExecuteStateMachine worker to invoke sub-state machines.

⚠️ Adapt the stateMachineArn pattern to match your own project name, environment, and source system code or pass parameters to your state machine to make it dynamic.

3. beVault-Generated State Machines

These state machines are automatically created by beVault when you deploy a version in metaVault. You do not write them manually — you only invoke them from your orchestration.

Load Staging

Loads data from the staging level 1 tables into the raw vault (Data Vault layer).

[project]_[environment]_Load_Staging_[staging table name]

Load Snapshot

Computes PIT (Point-in-Time) tables, prepares snapshot dates for information mart generation and runs data quality checks related to the snapshot.

[project]_[environment]_Load_Snapshot_[snapshot name]

Generate Information Mart

Runs the custom SQL scripts of an information mart.

[project]_[environment]_generate_[information mart name]

📘 For more details on the data workflow (Staging → Raw Vault → Business Vault → Information Mart), refer to the Data Workflows documentation.

4. Custom Export State Machine (`export`)

The final step of the pipeline delivers data to downstream consumers (Power BI, APIs, files, etc.). This is entirely custom and depends on your business case.

Examples of export steps:

Triggering a Power BI dataset refresh
Exporting query results to CSV or JSON files
Calling an external API with transformed data

The Global Process State Machine

The global-process state machine ties all building blocks together. It controls the overall execution order and parallelism.

Input Parameters

The global-process state machine accepts an input that allows you to selectively enable or disable specific sources. This is useful for testing or for running partial refreshes:

JSON

{
  "project": "main",
  "environment": "production",
  "SourcePropilot": true,
  "SourceOdoo14": true
}

Complete Example

The example below orchestrates two sources (propilot and odoo14), two snapshots (dq and im), two information marts (Financial_Reporting and Star_Schema), and a final Power BI export.

JSON

{
  "StartAt": "extract-sources",
  "States": {
    "extract-sources": {
      "Type": "Parallel",
      "ResultPath": "$.extracted",
      "Branches": [
        {
          "StartAt": "source-propilot-choice",
          "States": {
            "source-propilot-choice": {
              "Type": "Choice",
              "Choices": [
                {
                  "Variable": "$.SourcePropilot",
                  "BooleanEquals": true,
                  "Next": "extract-propilot"
                }
              ],
              "Default": "dummy-propilot"
            },
            "extract-propilot": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 2400,
              "Parameters": {
                "stateMachineArn": "extract-propilot",
                "project.$": "$.project",
                "environment.$": "$.environment"
              },
              "ResultPath": null,
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "ResultPath": "$.error",
                  "Next": "dummy-propilot"
                }
              ],
              "Next": "load-propilot"
            },
            "load-propilot": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 2400,
              "Parameters": {
                "stateMachineArn": "load-dv-propilot",
                "project.$": "$.project",
                "environment.$": "$.environment"
              },
              "ResultPath": null,
              "End": true
            },
            "dummy-propilot": {
              "Type": "Pass",
              "End": true
            }
          }
        },
        {
          "StartAt": "source-odoo14-choice",
          "States": {
            "source-odoo14-choice": {
              "Type": "Choice",
              "Choices": [
                {
                  "Variable": "$.SourceOdoo14",
                  "BooleanEquals": true,
                  "Next": "extract-odoo14"
                }
              ],
              "Default": "dummy-odoo14"
            },
            "extract-odoo14": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 6000,
              "Parameters": {
                "stateMachineArn": "extract-odoo14",
                "project.$": "$.project",
                "environment.$": "$.environment"
              },
              "ResultPath": null,
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "ResultPath": "$.error",
                  "Next": "dummy-odoo14"
                }
              ],
              "Next": "load-dv-odoo-14"
            },
            "load-dv-odoo-14": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 600,
              "Parameters": {
                "stateMachineArn": "load-dv-odoo14",
                "project.$": "$.project",
                "environment.$": "$.environment"
              },
              "ResultPath": null,
              "End": true
            },
            "dummy-odoo14": {
              "Type": "Pass",
              "End": true
            }
          }
        }
      ],
      "Next": "generate-snapshots"
    },
    "generate-snapshots": {
      "Type": "Parallel",
      "ResultPath": "$.generated",
      "Branches": [
        {
          "StartAt": "main_production_Load_Snapshot_dq",
          "States": {
            "main_production_Load_Snapshot_dq": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 6000,
              "Parameters": {
                "stateMachineArn": "main_production_Load_Snapshot_dq"
              },
              "ResultPath": null,
              "End": true
            }
          }
        },
        {
          "StartAt": "main_production_Load_Snapshot_im",
          "States": {
            "main_production_Load_Snapshot_im": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 6000,
              "Parameters": {
                "stateMachineArn": "main_production_Load_Snapshot_im"
              },
              "ResultPath": null,
              "End": true
            }
          }
        }
      ],
      "Next": "generate-information-marts"
    },
    "generate-information-marts": {
      "Type": "Parallel",
      "ResultPath": "$.generated",
      "Branches": [
        {
          "StartAt": "main_production_generate_Financial_Reporting",
          "States": {
            "main_production_generate_Financial_Reporting": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 6000,
              "Parameters": {
                "stateMachineArn": "main_production_generate_Financial_Reporting"
              },
              "ResultPath": null,
              "End": true
            }
          }
        },
        {
          "StartAt": "main_production_generate_Star_Schema",
          "States": {
            "main_production_generate_Star_Schema": {
              "Type": "Task",
              "Resource": "Production-ExecuteStateMachine",
              "TimeoutSeconds": 6000,
              "Parameters": {
                "stateMachineArn": "main_production_generate_Star_Schema"
              },
              "ResultPath": null,
              "End": true
            }
          }
        }
      ],
      "Next": "refresh_PowerBi"
    },
    "refresh_PowerBi": {
      "Type": "Task",
      "Resource": "Production-ExecuteStateMachine",
      "TimeoutSeconds": 7200,
      "Parameters": {
        "stateMachineArn": "refresh_PowerBi"
      },
      "ResultPath": null,
      "End": true
    }
  }
}

State Machine Design Patterns

Parallel execution

Use a Parallel state to run multiple branches concurrently. All branches must complete before moving to the next state.

Best used for: Running multiple source extractions, loading multiple snapshots, or generating multiple information marts simultaneously.

JSON

{
  "Type": "Parallel",
  "Branches": [ ... ],
  "Next": "next-state"
}

Conditional source activation (Choice + dummy Pass)

Use a Choice state to conditionally enable or skip a source based on a boolean flag in the input. The dummy Pass state acts as a no-op when the source is skipped.

Best used for: Making sources optional so you can run partial pipelines (e.g., for testing or selective refresh).

JSON

{
  "Type": "Choice",
  "Choices": [
    {
      "Variable": "$.SourceXxx",
      "BooleanEquals": true,
      "Next": "extract-xxx"
    }
  ],
  "Default": "dummy-xxx"
}

Error handling with Catch

Use Catch on extraction tasks to gracefully handle failures. Instead of failing the entire pipeline, the execution continues with a dummy Pass state.

Best used for: Non-critical sources where a failure should not block the rest of the pipeline.

JSON

{
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "ResultPath": "$.error",
      "Next": "dummy-xxx"
    }
  ]
}

Dynamic state machine invocation with Map + States.Format

Use a Map state combined with States.Format() to dynamically invoke one beVault-generated state machine per data package.

Best used for: Iterating over all data packages of a given source without hardcoding one task per package.

JSON

{
  "Type": "Map",
  "ItemsPath": "$.data_packages.name",
  "ItemSelector": {
    "data_package.$": "$$.Map.Item.Value",
    "project.$": "$.project",
    "env.$": "$.env",
    "source.$": "$.source"
  },
  "ItemProcessor": {
    "StartAt": "load",
    "States": {
      "load": {
        "Type": "Task",
        "Resource": "Production-ExecuteStateMachine",
        "Parameters": {
          "stateMachineArn.$": "States.Format('{}_{}_Load_Staging_{}_{}', $.project, $.env, $.source, $.data_package)"
        },
        "End": true
      }
    }
  }
}

Step-by-Step: Creating the Global Process

Prerequisites

beVault is installed and a version has been deployed to your target environment
The beVault-generated state machines (Load_Staging_*, Load_Snapshot_*, generate_*) are available in States
Your custom extraction state machines are already created in States

Steps

Identify your sources and note the name of each custom extraction state machine and its corresponding load-dv-[source] state machine.
Identify the beVault-generated state machines in States. Their names follow the format described in the Building Blocks section above. You can list them in the States UI under State Machines.
Create a load-dv-[source] state machine for each source (if not already done). Use the pattern described in Section 2.
Create the global-process state machine in States:
- Go to States → State Machines → Create State Machine
- Enter a name (e.g., global-process)
- Paste your JSON definition (adapt the example above to your project)
- Optionally define a default input to run the pipeline with all sources enabled by default
Schedule the execution (optional): From the state machine configuration tab, click Schedule to define a CRON expression and timezone for automated daily runs.
Test the pipeline by clicking Execute, monitoring the visual workflow, and verifying the output of each step.

Overview

Pipeline Architecture

Building Blocks

1. Custom Extraction State Machines (extract-[source])

Error handling during extraction

2. Custom Load DV State Machine (load-dv-[source])

State Machine Naming Convention for Load_Staging

Structure of load-dv-[source]

3. beVault-Generated State Machines

Load Staging

Load Snapshot

Generate Information Mart

4. Custom Export State Machine (export)

The Global Process State Machine

Input Parameters

Complete Example

State Machine Design Patterns

Parallel execution

Conditional source activation (Choice + dummy Pass)

Error handling with Catch

Dynamic state machine invocation with Map + States.Format

Step-by-Step: Creating the Global Process

Prerequisites

Steps

1. Custom Extraction State Machines (`extract-[source]`)

2. Custom Load DV State Machine (`load-dv-[source]`)

State Machine Naming Convention for `Load_Staging`

Structure of `load-dv-[source]`

4. Custom Export State Machine (`export`)