Workers Config

The component’s name of metaVault is DFAKTO_WORKERS. You will need to prefix the environment variable with this value.

Step Function Config

This section gives the necessary information for the orchestrator to connect to either AWS Step Function or, most likely, dFakto states.

{
  "stepFunctions": {
    "authenticationKey": "",
    "authenticationSecret": "",
    "serviceUrl": "http://localhost:5500",
    "roleArn": "role",
    "awsRegion": "eu-west-1",
    "ignoreSelfSignedCertificates": false,
    "RegisterRetryDelay": 5,
    "TaskTimeoutSeconds": 10000,
    "HeartBeatTimeoutSeconds": 100,
    "DefaultHeartbeatDelay": 5,
    "DefaultMaxConcurrency" : 5,
    "EnvironmentName": ""
  }
}

Environment variables

YAML

stepFunctions__authenticationKey=
stepFunctions__authenticationSecret=
stepFunctions__awsRegion=eu-west-1
stepFunctions__DefaultHeartbeatDelay=5
stepFunctions__DefaultMaxConcurrency=5
stepFunctions__EnvironmentName=
stepFunctions__HeartBeatTimeoutSeconds=100
stepFunctions__ignoreSelfSignedCertificates=False
stepFunctions__RegisterRetryDelay=5
stepFunctions__roleArn=role
stepFunctions__serviceUrl=http://localhost:5500
stepFunctions__TaskTimeoutSeconds=10000

Don’t forget to prefix the variables with the component’s name

• AuthenticationKey and AuthenticationSecret : allow to safely authenticate with the orchestrator. The values need to be retrieved from the orchestrator.

• serviceUrl: url where the orchestrator is running (not required if running using AWS Step Functions)

• roleArn and AWSRegion : if the orchestrator is states, can be left with the default value.

• RegisterRetryDelay: Delay in seconds between two attempts of registering an activity.

• TaskTimeoutSeconds: METAVAULT CONFIG - When the user pushes a new version of the data vault, state machine whose purpose is to load the data vault are generated and sent to the orchestrator. This field sets the maximum duration of each state of a state machine.

• HeartbeatTimeoutSeconds: METAVAULT CONFIG - Same as the previous field, except that it sets how long the orchestrator will wait at most between heartbeat from the workers. Can most likely be left with the default value.

• DefaultMaxConcurrency: WORKERS CONFIG - (1.5.1+) Maximum number of tasks processed at the same time for a given activity. Some Workers can have a hard-coded value and ignore the default configuration.

• DefaultHeartbeatDelay: WORKERS CONFIG - (1.5.1+) Default delay in seconds between two Heartbeat sent to the server while processing a Task. Some Workers can have a hard-coded value and ignore the default configuration.

• EnvironmentName: WORKERS CONFIG - For the other workers, can be left to any values. The value will be used to prefix the activities the worker connect to. For example, if the value chosen is “Prod”, the worker will connect to the Prod-gzip activity.

Stores

The stores are physical endpoints where you can either extract or send your data. They are documented here: Stores with the required fields to configure them.

{
    "Stores": {
        "Stores": [
            {
                "Name": "postgresql_store",
                "Type": "postgresql",
                "Config": {
                "ConnectionString": "Server=localhost;Port=5432;User Id=postgres;Password=password;Database=main"
                }
            }
        ]
    }
}

DFAKTO_WORKERS_STORES__STORES__0__Name: files
DFAKTO_WORKERS_STORES__STORES__0__Type: file
DFAKTO_WORKERS_STORES__STORES__0__HealthCheck: true
DFAKTO_WORKERS_STORES__STORES__0__Config__BasePath: /some/path
DFAKTO_WORKERS_STORES__STORES__1__Name: mydb
DFAKTO_WORKERS_STORES__STORES__1__Type: sqlserver
DFAKTO_WORKERS_STORES__STORES__1__HealthCheck: true
DFAKTO_WORKERS_STORES__STORES__1__Config__ConnectionString: "ConnectionString here :-)"

Logs

By default, all applications are sending reasonable logs to the console, the configuration can be updated using Serilog configuration section.

Here is the configuration of the logs. The most useful field to set is probably the path field, which sets where the logs will be stored on the disk.

For the other options :

• MinimumLevel: Indicate the level of log we want to store. From low to high, these are Verbose, Debug, Information, Warning, Error and Fatal

• rollOnfileSizeLimit: Indicate if we want to create a new log file when the current one reaches its size limit

• fileSizeLimitByte: Indicate the size limit of a log file. Once this size is reached, a new file will be created if the rollOnfileSizeLimit is set to true

• retainedFileCountLimit: Indicate how much file we should have, we start overriding the first log file.

• option : formatter: The formatter decides the format of the logs (text, json, …)

For more option, see https://github.com/serilog/serilog-settings-configuration

{
    "Serilog": {
        "MinimumLevel": {
            "Default": "Information",
            "Override": {
                "Microsoft": "Warning",
                "Microsoft.EntityFrameworkCore": "Warning",
                "System": "Warning"
            }
        },
        "WriteTo": [
            {
                "Name": "Console",
                "Args": {
                    "theme": "Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console",
                    "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}"
                }
            },
            { 
                "Name": "File",
                "Args":{ 
                    "path": "/var/log/testlog_.txt", 
                    "rollingInterval": "Day",
                    "fileSizeLimitBytes": 10000000,
                    "rollOnFileSizeLimit": true,
                    "retainedFileCountLimit": 10
                }
            }
        ]
    }
}

Environment variables

YAML

Serilog__MinimumLevel__Default=Information
Serilog__MinimumLevel__Override__Microsoft=Warning
Serilog__MinimumLevel__Override__System=Warning
Serilog__WriteTo__0__Args__outputTemplate=[{Timestamp:yyyy-MM-dd HH:mm:ss.fff} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}
Serilog__WriteTo__0__Args__theme=Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console
Serilog__WriteTo__0__Name=Console
Serilog__WriteTo__1__Args__fileSizeLimitBytes=10000000
Serilog__WriteTo__1__Args__path=/var/log/testlog_.txt
Serilog__WriteTo__1__Args__retainedFileCountLimit=10
Serilog__WriteTo__1__Args__rollingInterval=Day
Serilog__WriteTo__1__Args__rollOnFileSizeLimit=True
Serilog__WriteTo__1__Name=File

Don’t forget to prefix the variables with the component’s name

Other

Sentry

Sentry (https://sentry.io/) is an application monitoring platform.

• DSN: where to send events, so the events are associated with the correct project.

• IncludeRequestPayload: whether we should send the request body to Sentry. This is done so that the request data can be read at a later point in case an error happens while processing the request.

• SendDefaultPii: Whether we should report the user who made the request

• MinimumBreadcrumbLevel: Configure the lowest level a message has to be to become a breadcrumb. Breadcrumbs are the last (by default 100) log that were sent before the event was fired to Sentry.

• MinimumEventLevel: A LogLevel which indicates the minimum level a log message has to be sent to Sentry as an event. By default, this value is Error.

• AttachStackTrace: Configures whether Sentry should generate and attach stack traces to capture message calls.

• Debug: Turns debug mode on or off. If debug is enabled, Sentry will attempt to print out useful debugging information if something goes wrong with sending the event. The default is always false. It's generally not recommended to turn it on in production, though turning debug mode on will not cause any safety concerns.

• DiagnosticsLevel: Debug by default.

• DefaultTags: Defaults tags to add to all events.
  

{
  "Sentry": {
      "Dsn": "",
      "IncludeRequestPayload": true,
      "SendDefaultPii": true,
      "MinimumBreadcrumbLevel": "Debug",
      "MinimumEventLevel": "Warning",
      "AttachStackTrace": true,
      "Debug": true,
      "DiagnosticsLevel": "Error",
      "DefaultTags": {
          "client": "XXX"
      }
  }
}

Environment variables

YAML

Sentry__AttachStackTrace=True
Sentry__Debug=True
Sentry__DefaultTags__client=XXX
Sentry__DiagnosticsLevel=Error
Sentry__Dsn=
Sentry__IncludeRequestPayload=True
Sentry__MinimumBreadcrumbLevel=Debug
Sentry__MinimumEventLevel=Warning
Sentry__SendDefaultPii=True

Don’t forget to prefix the variables with the component’s name