Refactor coordinator to support create and delete index events when different index types exist on cluster

[thalurur]

Issue #, if available:
N/A

Description of changes:
Adding support for coordinator to correctly determine the indices that should be managed/delete currently managed indices by checking all of the metadata services to get the existing indices uuids and there existence in the cluster - rather than relying on cluster state metadata alone

CheckList:

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #272 (5441548) into development-extension (10d724b) will increase coverage by 0.04%.
The diff coverage is 79.16%.

@@                     Coverage Diff                     @@
##             development-extension     #272      +/-   ##
===========================================================
+ Coverage                    67.95%   67.99%   +0.04%     
- Complexity                    1718     1727       +9     
===========================================================
  Files                          250      251       +1     
  Lines                        10847    10917      +70     
  Branches                      1681     1686       +5     
===========================================================
+ Hits                          7371     7423      +52     
- Misses                        2561     2582      +21     
+ Partials                       915      912       -3     

Impacted Files	Coverage Δ
...management/indexstatemanagement/MetadataService.kt	30.35% <0.00%> (-1.18%)	⬇️
...ment/indexstatemanagement/util/RestHandlerUtils.kt	63.33% <ø> (ø)
...ment/indexstatemanagement/IndexMetadataProvider.kt	65.62% <65.62%> (ø)
...pensearch/indexmanagement/IndexManagementPlugin.kt	91.77% <80.95%> (-1.22%)	⬇️
...nt/indexstatemanagement/ManagedIndexCoordinator.kt	65.31% <85.71%> (+0.24%)	⬆️
...ndexstatemanagement/DefaultIndexMetadataService.kt	100.00% <100.00%> (ø)
...sport/action/addpolicy/TransportAddPolicyAction.kt	51.82% <100.00%> (ø)
...nt/indexstatemanagement/model/destination/Chime.kt	40.90% <0.00%> (-13.64%)	⬇️
...ent/rollup/action/explain/ExplainRollupResponse.kt	75.00% <0.00%> (-10.00%)	⬇️
...earch/indexmanagement/transform/model/Transform.kt	86.00% <0.00%> (-1.00%)	⬇️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10d724b...5441548. Read the comment docs.

[downsrob]

We can do this elsewhere if you want, but it seems like it is safe to remove document count now, right?

[thalurur]

Lets check after all code is merged if we want to remove it or keep it - one concern is with evaluating conditions in transition which has the document count, I know we are skipping this evaluation at the moment in transition step

[downsrob]

I noticed the failure, DeleteActionIT.test basic

java.lang.NullPointerException: null cannot be cast to non-null type kotlin.collections.Map<*, *>
...
 at org.opensearch.indexmanagement.indexstatemanagement.action.DeleteActionIT$test basic$4.invoke(DeleteActionIT.kt:64)

I had this flaky failure on another one of my development-extension PRs, we may have introduced some new flakiness in this test with our changes. I ran this IT(only this single test) 100 times yesterday and couldn't replicate the failure.

[thalurur]

I noticed the failure, DeleteActionIT.test basic

java.lang.NullPointerException: null cannot be cast to non-null type kotlin.collections.Map<*, *>
...
 at org.opensearch.indexmanagement.indexstatemanagement.action.DeleteActionIT$test basic$4.invoke(DeleteActionIT.kt:64)

I had this flaky failure on another one of my development-extension PRs, we may have introduced some new flakiness in this test with our changes. I ran this IT(only this single test) 100 times yesterday and couldn't replicate the failure.

I see, let me check if there is something we are doing incorrect