A source is a place which provides the data use to generate the events which are then sent to a destination. We can transform the data from any number of supported sources however we see fit.
The details of the data source are in the source
parameter of the YML configuration file
See the examples below for more information.
Data can be read from a DynamoDb.
Example for DynamoDb:
source:
type: dynamo
table: table-name
Data can be sourced from a JSON file
Example for JSON:
source:
type: json
file: ./examples/events/pets.seed.json
Data can be sourced from a CSV file.
Example for CSV:
source:
type: csv
file: ./examples/events/pets.seed.csv
To use a different delimiter, specify this in the delimiter
option
Example for Tab delimted text:
source:
type: csv
file: ./examples/events/data.txt
delimiter: "\t"
Data can be read via an OpenSearch Query.
Example for OpenSearch:
source:
type: opensearch
table: attestation
path: _source
get:
query:
match_all: {}
size: 1000
MSSQL can be used as a source of data by providing a connection string and an SQL SELECT statement describing the event.
Example for MSSQL:
source:
type: mssql
connectionString: >
Data Source=${SQL_SERV};
Initial Catalog=TEST_ATTESTATIONS;
User ID=${SQL_USER};
Password=${SQL_PASS};
Connection Timeout=1000;
TrustServerCertificate=true;
sql: >
SELECT
'organisation' AS noun,
'create' AS verb,
[id],
[name]
FROM [TEST_ATTESTATIONS].[dbo].[tbl_organisation]
recordSetIndex: 0
The parser will attempt to resolve environment variable in the connection string.
The recordSetIndex
parameter determines which record-set generated by the SQL to read events from. If the value is not supplied the default is 0.