chore: update filebeat templates to support drive from file and tenderdash only on logs nodes

[strophy]

Issue being fixed or feature implemented

Filebeat needs to support the following new configs:

• Reading drive logs from drive-json.log on dashmate-managed evonodes
• Reading tenderdash logs from both evonodes and seed nodes

What was done?

• Split platform logs rules into separate files and update template merge logic
• Read tenderdash logs from Docker because tenderdash.log is not available on seed nodes

How Has This Been Tested?

On testnet hp-masternode-10

Breaking Changes

None

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

[strophy]

While testing this I noticed that drive writes slightly different timestamps to different log files:

dashmate@hp-masternode-10:~/logs$ cat drive-json.log | grep 020000008a05000000
{"timestamp":"2023-10-13T02:28:22.775719Z","level":"TRACE","fields":{"message":"vote/commit sign bytes calculated","sign_bytes":"020000008a0500000000000000000000000000009d73924fb0527e1485f07eae4cdaaf409078af0c7c59dab80accb11edd426fa35ae78b2b9d542fc4669d82878893c3a3656a40a77cf55a14a113e4e24b9cbf53646173682d746573746e65742d3236","height":1418,"round":0,"vote":"CanonicalVote { r#type: Precommit, height: 1418, round: 0, block_id: [157, 115, 146, 79, 176, 82, 126, 20, 133, 240, 126, 174, 76, 218, 175, 64, 144, 120, 175, 12, 124, 89, 218, 184, 10, 204, 177, 30, 221, 66, 111, 163], state_id: [90, 231, 139, 43, 157, 84, 47, 196, 102, 157, 130, 135, 136, 147, 195, 163, 101, 106, 64, 167, 124, 245, 90, 20, 161, 19, 228, 226, 75, 156, 191, 83], chain_id: \"dash-testnet-26\" }"},"target":"tenderdash_abci::signatures","span":{"endpoint":"FinalizeBlock","height":1418,"request_id":"f5620f3d-e053-4840-8964-f233d74e8b60","round":0,"name":"abci"},"spans":[{"endpoint":"FinalizeBlock","height":1418,"request_id":"f5620f3d-e053-4840-8964-f233d74e8b60","round":0,"name":"abci"}]}
dashmate@hp-masternode-10:~/logs$ cat drive-pretty.log | grep 020000008a05000000
  2023-10-13T02:28:22.775753Z TRACE tenderdash_abci::signatures: vote/commit sign bytes calculated, sign_bytes: "020000008a0500000000000000000000000000009d73924fb0527e1485f07eae4cdaaf409078af0c7c59dab80accb11edd426fa35ae78b2b9d542fc4669d82878893c3a3656a40a77cf55a14a113e4e24b9cbf53646173682d746573746e65742d3236", height: 1418, round: 0, vote: CanonicalVote { r#type: Precommit, height: 1418, round: 0, block_id: [157, 115, 146, 79, 176, 82, 126, 20, 133, 240, 126, 174, 76, 218, 175, 64, 144, 120, 175, 12, 124, 89, 218, 184, 10, 204, 177, 30, 221, 66, 111, 163], state_id: [90, 231, 139, 43, 157, 84, 47, 196, 102, 157, 130, 135, 136, 147, 195, 163, 101, 106, 64, 167, 124, 245, 90, 20, 161, 19, 228, 226, 75, 156, 191, 83], chain_id: "dash-testnet-26" }

[shumkov]

tenderdash

[strophy]

If we name the top level of the filebeat templates like core_filebeat_inputs, tenderdash_filebeat_inputs, etc., it will result in more conditional logic to handle seed nodes because you cannot merge with an undefined variable, and we cannot initialize variables due to precedence rules. So we would either need an extra step to rename the top level of tenderdash_filebeat_inputs, or three final merge steps, depending on if we are running on a masternode, seed node or evonode.

For this reason I think it is less confusing to give the top level of the partial filebeat template we are merging into a predictable name that we can then reference in following steps, eventually merging everything into filebeat_inputs, which is directly ready for consumption by the geerlingguy.filebeat role.

I remember struggling with this a lot when I first wrote the templates.

[shumkov]

You can define empty definitions at the top. Then conditionally overwrite each one with actual data and bellow merge all three without any conditions

[strophy]

This won't work because set_fact has higher precedence than include_vars, so the variable definition will still be empty when we try to merge.

https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#understanding-variable-precedence

[ktechmidas]

LGTM (after explained on call)

[shumkov]

👍