Skip to main content
Skip table of contents

Metavault Configuration

Configuration of the metavault is greatly simplified thanks to the use of the appData. all the configuration does not need to be in the appsettings file, instead it can be spread across multiple file, as long as they reside in the config folder of the appData.

If there is a conflict between the content of a config file and the appsettings, the content of the config file will override the content of the appsettings. If there is a conflict between multiple config file (where two or more of them set the same config field), it will be decided by the file name alphabetical order which config is kept.

How to set the config

Lets take the example of the logs. Here is the configuration of the logs, as it is in the appsettings:

CODE 
"Serilog": {
  "Using": [ "Serilog.Sinks.File" ],
  "MinimumLevel": "Debug",
  "WriteTo": [
    {
      "Name": "File",
      "Args": {
        "rollOnFileSizeLimit": true,
        "fileSizeLimitBytes": "10000000",
        "retainedFileCountLimit": 10,
        "outputTemplate": "{Timestamp:o} [{Level:u3}] ({ThreadId}) {Message}{NewLine}{Exception}"
      }
    }
  ]
},

The path is intentionally missing : if you want to add a path where the logs will be sent (and you do), DO NOT modify the appsettings directly, instead you need to add a file in the config folder of the AppData. The name of the file does not matter.

Here is an example of the content of the file that will define apath for the logs :

CODE 
{
  "Serilog": {
    "WriteTo": [
      {
        "Args": {
          "path": "/var/logs"
        }
      }
    ]
  }
}

The content of the config files will override the content of the appsettings where there is a conflict, so if we are not happy with the default value present on the appsettings, we can easily set new values in the config file. For example, if we wanted to change the logs minimum level as well as settings the logs path, we could do it like so :

CODE 
{
  "Serilog": {
    "MinimumLevel": "Error",

    "WriteTo": [
      {
        "Args": {
          "path": "/var/logs"
        }
      }
    ]
  }
}

Logs

By default all applications are sending reasonable logs to console, the configuration can be updated using Serilog configuration section.

JSON 
{
    "Serilog": {
        "MinimumLevel": {
            "Default": "Information",
            "Override": {
                "Microsoft": "Warning",
                "System": "Warning"
            }
        },
        "WriteTo": [
            {
                "Name": "Console",
                "Args": {
                    "theme": "Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console",
                    "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}"
                }
            },
            { 
                "Name": "File",
                "Args":{ 
                    "path": "/var/log/testlog_.txt", 
                    "rollingInterval": "Day",
                    "fileSizeLimitBytes": 10000000,
                    "rollOnFileSizeLimit": true,
                    "retainedFileCountLimit": 10
                }
            }
        ]
    }
}

By default all applications are sending reasonable logs to console, the configuration can be updated using Serilog configuration section.

Here is the configuration of the logs. The most useful field to set is probably the path field, which set where the logs are going to be stored on the disk.

for the others options :

  • MinimumLevel: Indicate the level of log we want to store. From low to high, these are Verbose, Debug, Information, Warning, Error and Fatal

  • rollOnfileSizeLimit: Indicate if we want to create a new log file when the current one reach its size limit

  • fileSizeLimitByte: Indicate the size limit of a log file. once this size is reached, a new file will be created if the rollOnfileSizeLimit is set to true

  • retainedFileCountLimit: Indicate how much file we should have we start overriding the first log file.

  • option : formatter: The formatter decide the format of the logs (text, json, ..)

 

For more option, see https://github.com/serilog/serilog-settings-configuration

Sentry

Sentry (https://sentry.io/) is an application monitoring platform

CODE 
{
  "Sentry": {
      "Dsn": "",
      "IncludeRequestPayload": true,
      "SendDefaultPii": true,
      "MinimumBreadcrumbLevel": "Debug",
      "MinimumEventLevel": "Warning",
      "AttachStackTrace": true,
      "Debug": true,
      "DiagnosticsLevel": "Error",
      "DefaultTags": {
          "client": "XXX"
      }
  }
}

  • DNS: where to send events so the events are associated with the correct project.

  • IncludeRequestPayload: whether or not we should send the request body to Sentry. This is done so that the request data can be read at a later point in case an error happens while processing the request.

  • SendDefaultPii: Whether or not we should we report the user that made the request

  • MinimumBreadcrumbLevel: Configure the lowest level a message has to be to become a breadcrumb. Breadcrumbs are the last (by default 100) log that were send before the event was fired to Sentry.

  • MinimumEventLevel: A LogLevel which indicates the minimum level a log message has to be to be sent to Sentry as an event. By default this value is Error.

  • AttachStackTrace: Configures whether Sentry should generate and attach stacktraces to capture message calls.

  • Debug: Turns debug mode on or off. If debug is enabled Sentry will attempt to print out useful debugging information if something goes wrong with sending the event. The default is always false. It's generally not recommended to turn it on in production, though turning debug mode on will not cause any safety concerns.

  • DiagnosticsLevel: Debug by default.

  • DefaultTags: Defaults tags to add to all events.

Servers

The server field in the config allow the use of multiple DBMS with multiple user. These database configuration will be presented to the user when he wants to create an environment inside of a project, allowing him to decide where the datavault ultimately reside.

CODE 
{
  "Servers": {
    "Development": {
      "Name": "",
      "allowCustomEnvironmentDatabaseName": true,
      "DatabaseType": "PostgreSQL",
      "Username": "",
      "Password": "",
      "ReadOnlyUsername":"",
      "ReadOnlyPassword":"",
    },
    "ProductionSQLServer": {
      "Name": "localhost",
      "allowCustomEnvironmentDatabaseName": true,
      "DatabaseType": "SQLServer",
      "Username": "",
      "Password": "",
      "ReadOnlyUsername":"",
      "ReadOnlyPassword":"",
    }
  }
}

Having multiple servers can be useful if we want to have a “production” database and a “test” one, for example.

  • name : the host where the DBMS reside, for example localhost

  • allowCustomEnvironmentDatabaseName : If the users have the right to choose a database name when creating an environment. If set to false, the database name will be the same as the project name.

  • DatabaseType : The type of Database, currently one of the two value : “PostgreSQL”, “SQLServer”

  • Username : the name of the database user. This user will need to have the read and write access on the database, the right to create table, schema, and databases (if the user has no right to create database you will have to create them manually).

  • Password : the password of the database user

  • ReadOnlyUsername : User name of a user with stg_reader, ref_reader, dv_reader Role and im_reader

  • ReadOnlyPassword : password of the readonly user

Git

CODE 
{
  "git": {
    "EndOfLine": "\n"
  }
}

The git section of the config contains only one field, the EndOfLine field : This field can most likely be left untouched. It can be changed if a .git folder is copied from an unix system to a windows system or vice-versa, where the end of line character is encoded differently.

Step Functions

this section give the necessary information for the orchestrator to connect to either AWS Step Function or , most likely, dFakto states.

CODE 
{
  "stepFunctions": {
    "authenticationKey": "",
    "authenticationSecret": "",
    "serviceUrl": "http://localhost:5500",
    "roleArn": "role",
    "awsRegion": "eu-west-1",
    "ignoreSelfSignedCertificates": false,
    "TaskTimeoutSeconds": 10000,
    "HeartBeatTimeoutSeconds": 100
  }
}

  • AuthenticationKey and AuthenticationSecret : allow to safely authenticate with the orchestrator. the values needs to be retrieved from the orchestrator

  • serviceUrl: url where the orchestrator is running

  • roleArn and AWSRegion : if the orchestrator is states, can be left with the default value.

  • TaskTImeoutSeconds: When the user push a new version of the datavault, state machine whose purpose are to load the datavault are generated and send to the orchestrator. this field set maximum duration of each state of a state machine.

  • HeartbeatTimeoutSeconds: Same as the previous field, except that it set how long the orchestrator will wait at most between heartbeat from the workers. Can most likely be left with the default value.

Authentication

CODE 
{
  "Authentication": {
      "Authority": "",
      "ClientId": "",
      "Audience": ""
      "Scope":""
  }
}

 

Prometheus integration

Prometheus (https://prometheus.io/)  is an open-source systems monitoring and alerting

CODE 
 {
    "MetricsOptions": {
        "DefaultContextLabel": "dFakto States",
        "Enabled": true
    },
    "MetricsWebTrackingOptions": {
        "ApdexTrackingEnabled": true,
        "ApdexTSeconds": 0.1,
        "IgnoredHttpStatusCodes": [ 404 ],
        "IgnoredRoutesRegexPatterns": [],
        "OAuth2TrackingEnabled": true
    },
    "MetricEndpointsOptions": {
        "MetricsEndpointEnabled": true,
        "MetricsTextEndpointEnabled": false,
        "EnvironmentInfoEndpointEnabled": true
    }
}

 

  • DefaultContextLabel: Metrics recorded are grouped into “Contexts”, for example a database context or application context. Metrics names should be unique per context. default is “Application”.

  • Enabled: Allows recording of all metrics to be enabled/disabled, default is true.

  • ApdexTrackingEnabled: Allows enabling/disabling of calculating the apdex score on the overall responses times. Defaults to true. the Apdex (Application Performance Index) is used to monitor end-user satisfaction. It is an open industry standard that estimates the end user’s satisfaction level on an application’s response time through a score between 0 and 1.

  • apdexTSeconds: The Apdex T seconds value used in calculating the score on the samples collected.

  • IgnoredHttpStatusCode: Allows specific http status codes to be ignored when reporting on response related information, e.g. You might not want to monitor 404 status codes.

  • IngoredRoutesRegexPatterns: An list of regex patterns used to ignore matching routes from metrics tracking.

  • Oauth2TrackingEnabled: Allows recording of all OAuth2 Client tracking to be enabled/disabled. Defaults to true.

  • MetricsEndPointEnabled: Allows enabling/disabling of the /metrics endpoint, when disabled will result in a 404 status code, the default is true.

  • MetrucsTextEndpointEnabled: Allows enabling/disabling of the /metrics-text endpoint, when disabled will result in a 404 status code, the default is true.

  • EnvironmentInfoEndpointEnabled: Allows enabling/disabling of the /env endpoint, when disabled will result in a 404 status code, the default is true.

Other

CODE 
{
  "Hsts": {
        "MaxAge": "00.00:05:00",
        "Preload": true
    }
}

 HTTP Strict Transport Security (HSTS) is a simple and widely supported standard to protect visitors by ensuring that their browsers always connect to a website over HTTPS.

  • MaxAge: The time, that the browser should remember that a site is only to be accessed using HTTPS.

  • Preload: it is possible to enforce secure connections on a higher level, even before visiting a website for the first time: the HSTS preload list. This is a list, managed by google, with domain names that by default support HSTS: 

CODE 
{
  "Urls": "http://localhost:5000",
  "ForwardedHeadersOptions": {
      "ForwardedHeaders": "All"
  }
}

The ForwardedHeadersOptions set the behavior of proxied headers onto the requests. You can most likely leave it to the default All value.

The accepted values are :

  • All : Process X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Proto.

  • None : Do not process any forwarders

  • XForwardedFor : Process X-Forwarded-For, which identifies the originating IP address of the client.

  • XForwardedHost : Process X-Forwarded-Host, which identifies the original host requested by the client.

  • XForwardedProto : Process X-Forwarded-Proto, which identifies the protocol (HTTP or HTTPS) the client used to connect.

Fore more information, see https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-5.0

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close