sources.md

Sources

A source is a place which provides the data use to generate the events which are then sent to a destination. We can transform the data from any number of supported sources however we see fit.

The details of the data source are in the source parameter of the YML configuration file

See the examples below for more information.

DynamoDb

Data can be read from a DynamoDb.

Example for DynamoDb:

source:
  type: dynamo
  table: table-name

JSON

Data can be sourced from a JSON file

Example for JSON:

source:
  type: json
  file: ./examples/events/pets.seed.json

CSV

Data can be sourced from a CSV file.

Example for CSV:

source:
  type: csv
  file: ./examples/events/pets.seed.csv

To use a different delimiter, specify this in the delimiter option

Example for Tab delimted text:

source:
  type: csv
  file: ./examples/events/data.txt
  delimiter: "\t"

OpenSearch

Data can be read via an OpenSearch Query.

Example for OpenSearch:

source:
    type: opensearch
    table: attestation
    path: _source
    get:
        query:
        match_all: {}
    size: 1000

MSSQL

MSSQL can be used as a source of data by providing a connection string and an SQL SELECT statement describing the event.

Example for MSSQL:

source:
    type: mssql
    connectionString: >
        Data Source=${SQL_SERV};
        Initial Catalog=TEST_ATTESTATIONS;
        User ID=${SQL_USER};
        Password=${SQL_PASS};
        Connection Timeout=1000;
        TrustServerCertificate=true;
    sql: >
        SELECT
            'organisation' AS noun,
            'create' AS verb,
            [id],
            [name]
        FROM [TEST_ATTESTATIONS].[dbo].[tbl_organisation]
    recordSetIndex: 0

The parser will attempt to resolve environment variable in the connection string.

The recordSetIndex parameter determines which record-set generated by the SQL to read events from. If the value is not supplied the default is 0.