Workers Config

logs

By default all applications are sending reasonable logs to console, the configuration can be updated using Serilog configuration section.

JSON

{
    "Serilog": {
        "MinimumLevel": {
            "Default": "Information",
            "Override": {
                "Microsoft": "Warning",
                "System": "Warning"
            }
        },
        "WriteTo": [
            {
                "Name": "Console",
                "Args": {
                    "theme": "Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console",
                    "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}"
                }
            },
            { 
                "Name": "File",
                "Args":{ 
                    "path": "/var/log/testlog_.txt", 
                    "rollingInterval": "Day",
                    "fileSizeLimitBytes": 10000000,
                    "rollOnFileSizeLimit": true,
                    "retainedFileCountLimit": 10
                }
            }
        ]
    }
}

By default all applications are sending reasonable logs to console, the configuration can be updated using Serilog configuration section.

Here is the configuration of the logs. The most useful field to set is probably the path field, which set where the logs are going to be stored on the disk.

for the others options :

MinimumLevel: Indicate the level of log we want to store. From low to high, these are Verbose, Debug, Information, Warning, Error and Fatal
rollOnfileSizeLimit: Indicate if we want to create a new log file when the current one reach its size limit
fileSizeLimitByte: Indicate the size limit of a log file. once this size is reached, a new file will be created if the rollOnfileSizeLimit is set to true
retainedFileCountLimit: Indicate how much file we should have we start overriding the first log file.
option : formatter: The formatter decide the format of the logs (text, json, ..)

For more option, see https://github.com/serilog/serilog-settings-configuration

Sentry

Sentry (https://sentry.io/) is an application monitoring platform

CODE

{
  "Sentry": {
      "Dsn": "",
      "IncludeRequestPayload": true,
      "SendDefaultPii": true,
      "MinimumBreadcrumbLevel": "Debug",
      "MinimumEventLevel": "Warning",
      "AttachStackTrace": true,
      "Debug": true,
      "DiagnosticsLevel": "Error",
      "DefaultTags": {
          "client": "XXX"
      }
  }
}

DNS: where to send events so the events are associated with the correct project.
IncludeRequestPayload: whether or not we should send the request body to Sentry. This is done so that the request data can be read at a later point in case an error happens while processing the request.
SendDefaultPii: Whether or not we should we report the user that made the request
MinimumBreadcrumbLevel: Configure the lowest level a message has to be to become a breadcrumb. Breadcrumbs are the last (by default 100) log that were send before the event was fired to Sentry.
MinimumEventLevel: A LogLevel which indicates the minimum level a log message has to be to be sent to Sentry as an event. By default this value is Error.
AttachStackTrace: Configures whether Sentry should generate and attach stacktraces to capture message calls.
Debug: Turns debug mode on or off. If debug is enabled Sentry will attempt to print out useful debugging information if something goes wrong with sending the event. The default is always false. It's generally not recommended to turn it on in production, though turning debug mode on will not cause any safety concerns.
DiagnosticsLevel: Debug by default.
DefaultTags: Defaults tags to add to all events.

Servers

The server field allow the use of multiple DBMS with multiple user. These database configuration will be presented to the user when he wants to create an environment inside of a project, allowing him to decide where the datavault ultimately reside.

CODE

"Servers": {
  "Development": {
    "Name": "",
    "allowCustomEnvironmentDatabaseName": true,
    "DatabaseType": "PostgreSQL",
    "Username": "",
    "Password": ""
  },
  "ProductionSQLServer": {
    "Name": "localhost",
    "allowCustomEnvironmentDatabaseName": true,
    "DatabaseType": "SQLServer",
    "Username": "",
    "Password": ""
  }
},

Having multiple servers can be useful if we want to have a “production” database and a “test” one, for example.

name : the host where the DBMS reside, for example localhost
allowCustomEnvironmentDatabaseName : If the users have the right to choose a database name when creating an environment. If set to false, the database name will be the same as the project name.
DatabaseType : The type of Database, currently one of the two value : “PostgreSQL”, “SQLServer”
Username : the name of the database user. This user will need to have the read and write access on the database, the right to create table, schema, and databases.
Password : the password of the database user

Step Function Config

this section give the necessary information for the orchestrator to connect to either AWS Step Function or , most likely, dFakto states.

CODE

{
  "stepFunctions": {
    "authenticationKey": "",
    "authenticationSecret": "",
    "serviceUrl": "http://localhost:5500",
    "roleArn": "role",
    "awsRegion": "eu-west-1",
    "ignoreSelfSignedCertificates": false,
    "TaskTimeoutSeconds": 10000,
    "HeartBeatTimeoutSeconds": 100,
    "EnvironmentName": ""
  }
}

AuthenticationKey and AuthenticationSecret : allow to safely authenticate with the orchestrator. the values needs to be retrieved from the orchestrator.
serviceUrl: url where the orchestrator is running (not required if running using AWS Step Functions)
roleArn and AWSRegion : if the orchestrator is states, can be left with the default value.
TaskTimeoutSeconds: When the user push a new version of the datavault, state machine whose purpose are to load the datavault are generated and send to the orchestrator. this field set maximum duration of each state of a state machine.
HeartbeatTimeoutSeconds: Same as the previous field, except that it set how long the orchestrator will wait at most between heartbeat from the workers. Can most likely be left with the default value.
EnvironmentName:
- Ignored in the metavault configuration.
- For the other workers, can be left to any values. The value will be used to prefix the activities the worker connect to. For exemple, if the value chosen is “Prod”, the worker will connect to the Prod-gzip activity.