Allow attempt to load security config in case of plugin restart even …

[dhiAmzn]

…if security index already exists

Signed-off-by: Dhiresh Jain dhireshj@amazon.com

opendistro-for-elasticsearch/security pull request intake form

Please provide as much details as possible to get feedback/acceptance on your PR quickly

NOTE: This commit has been moved from opendistro repo to opensearch. The manual test was done using docker in opendistro. Docker testing is currently unavailable in opensearch so did not perform a manual test here yet.

1. Category: (_Enhancement)

2. Github Issue # or road-map entry, if available:

3. Description of changes: Put security configs if absent in the security index. Security config could be absent in index due to missing files or errored out during loading of files

4. Why these changes are required? In case of issues with loading static configuration from files during plugin bootstrap, we currently need to fix the issue, delete security index and restart a plugin. This change will allow us to just fix the issue and restart the plugin without needing to delete the security index.

5. What is the old behaviour before changes and new behaviour after changes? (Please add any example/logs/screen-shot if available)
   [Docker Setup - 2 nodes cluster]
   Old Behaviour:
   Corrupt one of the static securityconfig files and start the nodes in the cluster. Verify cluster requests fail. Fix the corrupt file and restart a node. Verify cluster requests fail.
   New Behaviour:
   Corrupt one of the static securityconfig files and start the nodes in the cluster. Verify cluster requests fail. Fix the corrupt file and restart a node. Verify cluster requests succeed.
   Note: Each time security plugin bootstraps, there will be an attempt to load all securityconfig files into security index. Attempting to index a document from static files when it already exists will throw a VersionConflictEngineException which is anyway swallowed and is a no/op, so this will not be a cause of concern.

6. Testing done: (Please provide details of testing done: Unit testing, integration testing and manual testing)
   Old Behaviour:

elasticsearch1    | [2021-04-03T16:31:56,632][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch1] Doc with id 'config' and version 2 is updated in .opendistro_security index.
elasticsearch1    | [2021-04-03T16:31:56,632][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch1] Will update 'roles' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
elasticsearch1    | [2021-04-03T16:31:56,641][ERROR][c.a.o.s.c.ConfigurationRepository] [elasticsearch1] Cannot apply default config (this is maybe not an error!) due to while parsing a block mapping
elasticsearch1    |  in 'reader', line 1, column 1:
elasticsearch1    |     _meta:
elasticsearch1    |     ^
elasticsearch1    | expected <block end>, but found '<scalar>'
elasticsearch1    |  in 'reader', line 13, column 2:
elasticsearch1    |      INVALID_DATA_HERE
elasticsearch1    |      ^
elasticsearch1    |
elasticsearch1    |  at [Source: (FileReader); line: 13, column: 2]
elasticsearch2    | [2021-04-03T16:31:56,650][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch2] Doc with id 'audit' and version 2 is updated in .opendistro_security index.
elasticsearch1    | [2021-04-03T16:31:56,947][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:31:56,948][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:31:57,003][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:31:57,005][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:31:57,198][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:31:57,199][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:31:57,202][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:31:57,548][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:31:57,550][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:31:57,553][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)

dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> curl "https://localhost:9200/_cat/nodes?v" -k -u admin:admin
Open Distro Security not initialized.⏎
dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> docker exec -it elasticsearch1 /bin/bash
[root@008f7b698c7f elasticsearch]# vi plugins/opendistro_security/securityconfig/roles.yml # FIXED roles.yml HERE
[root@008f7b698c7f elasticsearch]# exit
exit
dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> docker restart elasticsearch1
elasticsearch1
dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> curl "https://localhost:9200/_cat/nodes?v" -k -u admin:admin
Open Distro Security not initialized.⏎

New Behaviour:

elasticsearch2    | [2021-04-03T16:53:04,382][INFO ][c.a.o.s.c.ConfigurationRepository] [elasticsearch2] Index .opendistro_security already exists
elasticsearch2    | [2021-04-03T16:53:04,389][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch2] Will update 'config' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
elasticsearch2    | [2021-04-03T16:53:04,959][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch2] Doc with id 'config' and version 2 is updated in .opendistro_security index.
elasticsearch2    | [2021-04-03T16:53:04,959][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch2] Will update 'roles' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
elasticsearch2    | [2021-04-03T16:53:04,965][ERROR][c.a.o.s.c.ConfigurationRepository] [elasticsearch2] Cannot apply default config (this is maybe not an error!) due to while parsing a block mapping
elasticsearch2    |  in 'reader', line 1, column 1:
elasticsearch2    |     _meta:
elasticsearch2    |     ^
elasticsearch2    | expected <block end>, but found '<scalar>'
elasticsearch2    |  in 'reader', line 13, column 2:
elasticsearch2    |      INVALID_DATA_HERE
elasticsearch2    |      ^
elasticsearch2    |
elasticsearch2    |  at [Source: (FileReader); line: 13, column: 2]
elasticsearch2    | [2021-04-03T16:53:05,179][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:53:05,181][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:53:05,314][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:53:05,320][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch2    | [2021-04-03T16:53:05,328][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:53:05,733][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch1] Index .opendistro_security already contains doc with id config, skipping update.
elasticsearch1    | [2021-04-03T16:53:05,734][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch1] Will update 'roles' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
elasticsearch1    | [2021-04-03T16:53:05,740][ERROR][c.a.o.s.c.ConfigurationRepository] [elasticsearch1] Cannot apply default config (this is maybe not an error!) due to while parsing a block mapping
elasticsearch1    |  in 'reader', line 1, column 1:
elasticsearch1    |     _meta:
elasticsearch1    |     ^
elasticsearch1    | expected <block end>, but found '<scalar>'
elasticsearch1    |  in 'reader', line 13, column 2:
elasticsearch1    |      INVALID_DATA_HERE
elasticsearch1    |      ^
elasticsearch1    |
elasticsearch1    |  at [Source: (FileReader); line: 13, column: 2]
elasticsearch1    | [2021-04-03T16:53:05,845][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:53:05,846][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:53:06,025][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:53:06,027][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:53:06,028][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch1] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)
elasticsearch1    | [2021-04-03T16:53:06,970][INFO ][o.e.c.r.a.AllocationService] [elasticsearch1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
elasticsearch2    | [2021-04-03T16:53:13,113][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [elasticsearch2] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=_doc)

dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> docker exec -it elasticsearch1 /bin/bash
[root@70c45c060e53 elasticsearch]# vi plugins/opendistro_security/securityconfig/roles.yml # FIXED roles.yml HERE
[root@70c45c060e53 elasticsearch]# exit
exit
dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> docker restart elasticsearch1
elasticsearch1

elasticsearch1    | [2021-04-03T16:54:34,669][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch1] added {{elasticsearch2}{gEdbQA5ISwKPf0VHBkNRyA}{DWciasNGT9Sq_ftXgl1SRA}{172.31.0.3}{172.31.0.3:9300}{dir}}, term: 2, version: 12, reason: Publication{term=2, version=12}
elasticsearch1    | [2021-04-03T16:54:35,188][INFO ][o.e.c.r.a.AllocationService] [elasticsearch1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opendistro_security][0]]]).
elasticsearch1    | [2021-04-03T16:54:35,931][WARN ][o.e.c.r.a.AllocationService] [elasticsearch1] [.opendistro_security][0] marking unavailable shards as stale: [qv7nUU8yS2K7_cUNnxS5jQ]
elasticsearch1    | [2021-04-03T16:54:36,691][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch1] Index .opendistro_security already contains doc with id config, skipping update.
elasticsearch1    | [2021-04-03T16:54:36,692][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch1] Will update 'roles' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
elasticsearch1    | [2021-04-03T16:54:37,450][INFO ][o.e.c.m.MetadataMappingService] [elasticsearch1] [.opendistro_security/9zrUWomFQpO7x1bppVVJ8w] update_mapping [_doc]
elasticsearch1    | [2021-04-03T16:54:38,206][INFO ][o.e.c.r.a.AllocationService] [elasticsearch1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
elasticsearch1    | [2021-04-03T16:54:38,941][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch1] Doc with id 'roles' and version 2 is updated in .opendistro_security index.


dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> curl "https://localhost:9200/_cat/nodes?v" -k -u admin:admin
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.31.0.2           17          21  13    0.70    1.68     1.40 dimr      *      elasticsearch1
172.31.0.3           40          21  23    0.70    1.68     1.40 dir       -      elasticsearch2
dhireshj@f45c899db61b ~/w/d/dhi_security (improve_security_index_bootstrap)> curl "https://localhost:9200/_opendistro/_security/authinfo?pretty" -k -u admin:admin
{
  "user" : "User [name=admin, backend_roles=[admin], requestedTenant=null]",
  "user_name" : "admin",
  "user_requested_tenant" : null,
  "remote_address" : "172.31.0.1:58882",
  "backend_roles" : [
    "admin"
  ],
  "custom_attribute_names" : [ ],
  "roles" : [
    "own_index",
    "all_access"
  ],
  "tenants" : {
    "global_tenant" : true,
    "admin_tenant" : true,
    "admin" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}

7. Setup Instructions

-> Checkout https://github.com/opendistro-for-elasticsearch/opendistro-build
-> cd into {$PATH}/opendistro-build/elasticsearch/docker
-> Create directory build/elasticsearch/plugins/
-> Create file build/elasticsearch/plugins/plugins.list
-> Pull changes of this PR locally
-> cd into plugin directory
-> Add invalid config to any configuration file in securityconfig directory
-> Build the plugin: mvn clean package -Padvanced -DskipTests
-> Copy artifact target/releases/*.zip to ../opendistro-build/elasticsearch/docker/build/elasticsearch/plugins/
-> cd into {$PATH}/opendistro-build/elasticsearch/docker
-> Add the artifact name to the plugins.list file
-> run make-build [TIP: Remove knn dependencies from Dockerfile to speed up build. We do no require knn anyway)
-> run make-cluster
-> Once the cluster is up with two nodes, verify requests fail with Open Distro Security not initialised
-> Log into a docker container docker exec -it elasticsearch1 /bin/bash
-> Fix the corrupted configuration file in plugins/opendistro_security/securityconfig/ directory
-> Exit docker container and restart it docker restart elasticsearch1
-> Cluster should be up and requests to it should succeed now
-> Without the plugin changes, the previous step would still fail

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #1154 (e954faa) into main (f698fe6) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1154      +/-   ##
============================================
- Coverage     64.61%   64.58%   -0.03%     
- Complexity     3169     3172       +3     
============================================
  Files           244      244              
  Lines         17124    17122       -2     
  Branches       3036     3035       -1     
============================================
- Hits          11064    11058       -6     
- Misses         4517     4520       +3     
- Partials       1543     1544       +1     

Impacted Files	Coverage Δ	Complexity Δ
...ecurity/configuration/ConfigurationRepository.java	74.03% <100.00%> (+0.26%)	22.00 <0.00> (+1.00)
...relasticsearch/security/rest/TenantInfoAction.java	77.61% <0.00%> (-10.45%)	15.00% <0.00%> (ø%)
...relasticsearch/security/rest/KibanaInfoAction.java
...sticsearch/security/rest/DashboardsInfoAction.java	25.80% <0.00%> (ø)	4.00% <0.00%> (?%)
.../dlic/auth/ldap2/LDAPConnectionFactoryFactory.java	58.01% <0.00%> (+0.76%)	24.00% <0.00%> (+1.00%)
...ty/configuration/ConfigurationLoaderSecurity7.java	68.85% <0.00%> (+0.81%)	10.00% <0.00%> (ø%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f698fe6...e954faa. Read the comment docs.

[sujithvm]

I believe alternate interpretation to the issue is "Put security configs if absent in the security index". It may be missing if manually deleted or error-ed out while loading.

[vrozov]

Thank you for rebasing