Adds an alias action (#35) #575

goyamegh · 2022-10-20T17:43:29Z

Signed-off-by: Megha Goyal goyamegh@amazon.com

Issue #, if available:
#35 (#304)

Description of changes:
Adds support for an alias action in ISM.
Gives complete support for all update alias API options by reusing the alias actions parser internally and explicitly not defining the mappings in our config index.

Coming from the GitHub feature request, the use cases people asked for:

My use case is, If we have an alias for an index with rollover ILM policy, can we remove it from alias after some predefined days, let's say 30 days? I know there is a way to delete an index once its old, but in our case we do not want to delete but want the index to be removed from alias and have the alias switch and point to indices whose age is less than 30 days.

This should be easily doable with the Alias action, just have a state that gets transitioned into after the index is 30 days old (or 30 days rollover age) and then remove the alias.

My use case is: we provide search_alias to search our dataset. We publish new version of dataset every week as a new index (e.g. dataset-yyyy-MM-dd. We want the search_alias always pointing to the latest published dataset. The process is: ingest new data to a new index, once ingestion finish, re-pointing search_alias to the new index, remove old index from the search_alias

Should be possible, the newly created index can remove the alias from all search-index-* indices and then add the alias it itself in a single call. The main thing would be knowing when ingestion finished for the new index.. which you could just use some absolute time using the transition condition if you know it only takes x amount of time.

CheckList:

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Testing steps:

Rollover index with deletion of indices when the doc count exceeds a certain value.
1.1. Create an ISM policy

curl -X PUT "localhost:9200/_plugins/_ism/policies/rollover_policy?pretty" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "description": "Example rollover policy.",
    "default_state": "rollover",
    "states": [
      {
        "name": "rollover",
        "actions": [
          {
            "rollover": {
              "min_doc_count": 1
            }
          }
        ],
        "transitions": [{
            "state_name": "alias",
            "conditions": {
              "min_doc_count": "2"
            }
          }]
      },
      {
        "name": "alias",
        "actions": [
          {
            "alias": {
              "actions": [
                {
                  "remove": {
                      "alias": "log"
                  }
                }
              ]
            }
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": ["log*"],
      "priority": 100
    }
  }
}
'

1.2 Create an index template to enable the policy on:

curl -X PUT "localhost:9200/_index_template/ism_rollover1?" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["1log*"],
  "template": {
   "settings": {
    "plugins.index_state_management.rollover_alias": "1log"
   }
 }
}
'

1.3 Change the cluster settings to trigger jobs every minute:

curl -X PUT "localhost:9200/_cluster/settings?pretty=true" -H 'Content-Type: application/json' -d'
{
  "persistent" : {
    "plugins.index_state_management.job_interval" : 1
  }
}
'

1.4 Create a new index:

curl -X PUT "localhost:9200/log-000001" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "log": {
      "is_write_index": true
    }
  }
}
'

1.5 Add a document to the index to trigger the job:


curl -X POST "localhost:9200/log-000001/_doc" -H 'Content-Type: application/json' -d'
{
  "message": "dummy"
}
'

1.6 The first job will trigger the rollover action and a new index will be created.
1.7 Add another document to the two indices like step 1.5.
1.8 The new job will cause the second index to point to the log alias and the older one will be removed due to alias action.
1.9. Verify these steps using alias and index API:

curl -X GET "localhost:9200/_cat/indices?pretty"

curl -X GET "localhost:9200/_cat/aliases?pretty"

Fail the case if a user tries to remove index using alias action.
2.1 Create the following policy:

curl -X PUT "localhost:9200/_plugins/_ism/policies/remove_index_policy12345?pretty" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "description": "Example rollover policy.",
    "default_state": "rollover",
    "states": [
      {
        "name": "rollover",
        "actions": [
          {
            "rollover": {
              "min_doc_count": 1
            }
          }
        ],
        "transitions": [{
            "state_name": "alias",
            "conditions": {
              "min_doc_count": "2"
            }
          }]
      },
      {
        "name": "alias",
        "actions": [
          {
            "alias": {
              "actions": [
                {
                  "add": {
                      "alias": "log"
                  }
                },
                {
                  "remove": {
                      "index": "log-00001"
                  }
                }
              ]
            }
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": ["MMMMMMMMttestlog*"],
      "priority": 100
    }
  }
}
'

2.2 This will create an Illegal argument exception with the error: "Remove_index is not allowed here."

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/action/AliasAction.kt

bowenlan-amzn · 2022-10-24T22:13:52Z

I think the alias action should only operator on the managed index. e.g. we have a policy1 with alias action, apply policy1 to index1, this alias action cannot change the alias to other indices than index1.

We can see there're use cases some actions (shrink, rollover, rollup) create new index and we would like some alias to be add to the new index. To do that, we can do it at least in 2 ways:

Prepare a index template to add alias at new index creation time
Use ISM template to apply an ISM policy to the new index, and this ISM policy has alias action
So I don't think the alias action should work on more than the applied index itself. To achieve this, we can block the index or indices options in alias action, and auto populate index to be the applied index itself.

{
  "actions" : [
    { "remove" : { "index" : "<managed index itself>", "alias" : "<any alias>" } },
    { "add" : { "index" : "<managed index itself>", "alias" : "<any alias>" } }
  ]
}

And we should block remove_index action in alias action, we already have delete action.

Alias API

bowenlan-amzn · 2022-10-24T22:15:28Z

src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/action/AliasActionParser.kt

+            val fieldName = xcp.currentName()
+            xcp.nextToken()
+            when (fieldName) {
+                ACTIONS -> {


Block remove_index action
Block index indices fields if user provided

Added checks in initiate function to raise Illegal argument exception.

bowenlan-amzn · 2022-10-24T22:15:50Z

...in/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/alias/AttemptAliasStep.kt

+                // Take the indices that were set to be updated on the alias action and join them into a single string
+                // So we don't have to pass each one individually into the script service to be compiled
+                val indices = it.indices().joinToString(",")
+                // Compile them so the user can use the current dynamic index name in the name such as "{{ctx.index}}" in the static policy
+                val indicesScript = Script(ScriptType.INLINE, Script.DEFAULT_TEMPLATE_LANG, indices, mapOf())
+                val compiledIndices = compileTemplate(indicesScript, context.metadata, indices, context.scriptService)
+                // Split them back into individual index names and set back on the AliasActions request
+                val splitIndices = Strings.splitStringByCommaToArray(compiledIndices)
+                it.indices(*splitIndices)
+                request.addAliasAction(it)


Populate the index to be the managed index itself

Removed the logic for user input and using the managed index itself.

codecov-commenter · 2022-10-31T19:49:16Z

Codecov Report

Merging #575 (7989760) into main (1856295) will increase coverage by 0.21%.
The diff coverage is 64.28%.

@@             Coverage Diff              @@
##               main     #575      +/-   ##
============================================
+ Coverage     75.55%   75.77%   +0.21%     
- Complexity     2478     2516      +38     
============================================
  Files           319      322       +3     
  Lines         14633    14703      +70     
  Branches       2255     2263       +8     
============================================
+ Hits          11056    11141      +85     
+ Misses         2312     2287      -25     
- Partials       1265     1275      +10

Impacted Files	Coverage Δ
...ndexstatemanagement/step/alias/AttemptAliasStep.kt	`58.33% <58.33%> (ø)`
...agement/indexstatemanagement/action/AliasAction.kt	`68.75% <68.75%> (ø)`
...t/indexstatemanagement/action/AliasActionParser.kt	`70.58% <70.58%> (ø)`
...anagement/indexstatemanagement/ISMActionsParser.kt	`91.80% <100.00%> (+0.13%)`	⬆️
...nt/indexstatemanagement/model/destination/Slack.kt	`55.55% <0.00%> (-22.23%)`	⬇️
...nt/indexstatemanagement/ManagedIndexCoordinator.kt	`68.71% <0.00%> (-0.59%)`	⬇️
.../opensearch/indexmanagement/rollup/model/Rollup.kt	`86.04% <0.00%> (+0.46%)`	⬆️
...nt/rollup/action/stop/TransportStopRollupAction.kt	`77.64% <0.00%> (+1.17%)`	⬆️
...ent/indexstatemanagement/util/ManagedIndexUtils.kt	`77.87% <0.00%> (+1.76%)`	⬆️
.../rollup/action/start/TransportStartRollupAction.kt	`75.29% <0.00%> (+3.52%)`	⬆️
... and 5 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

bowenlan-amzn · 2022-11-01T20:58:48Z

src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/action/AliasAction.kt

@@ -21,6 +21,8 @@ class AliasAction(

    init {
        require(actions.isNotEmpty()) { "At least one alias action needs to be specified." }
+        require(!actions.any { it.actionType() == IndicesAliasesRequest.AliasActions.Type.REMOVE_INDEX }) { "Remove_index is not allowed here." }
+        require(actions.all { it.indices().isNullOrEmpty() }) { "Alias actions are only allowed on managed indices." }


need to verify both indices and index to be empty.
And change the message to be "Alias action can only work on its applied index so don't accept index/indices parameter."

This is not needed as we store them as a string list in the indices field only. Refer this line: https://github.com/opensearch-project/OpenSearch/blob/4d045a164e12a382881140e32f9285a3224fecc7/server/src/main/java/org/opensearch/action/admin/indices/alias/IndicesAliasesRequest.java#L360

Changed the message.

src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/action/AliasAction.kt

src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/action/AliasActionParser.kt

khushbr · 2022-11-02T00:41:47Z

...in/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/alias/AttemptAliasStep.kt

+    private var info: Map<String, Any>? = null
+
+    override suspend fun execute(): Step {
+        val context = this.context ?: return this


Any specific reason, we are using this.context instead of using a (final) val param like the rest of code ?

Looks like it is being used in the same manner in all parsers. Kindly refer:

index-management/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/allocation/AttemptAllocationStep.kt

Line 25 in 2d3fd97

val context = this.context ?: return this

,

index-management/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/close/AttemptCloseStep.kt

Line 26 in efe2d33

val context = this.context ?: return this

khushbr · 2022-11-02T00:54:47Z

...in/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/alias/AttemptAliasStep.kt

+    private fun handleException(e: Exception, indexName: String) {
+        val message = getFailedMessage(indexName)
+        logger.error(message, e)
+        stepStatus = StepStatus.FAILED
+        val mutableInfo = mutableMapOf("message" to message)
+        val errorMessage = e.message
+        if (errorMessage != null) mutableInfo["cause"] = errorMessage
+        info = mutableInfo.toMap()
+    }


This code is repeated in 20+ files, can extract this and put it in the BaseClass / Utility function ?

Found one more file, but if we extract it, then we need to return info and step status as new objects, as they are getting modified in the function. Keeping it the same for now, but will figure out a better way to handle this and raise another PR.

...in/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/alias/AttemptAliasStep.kt

khushbr · 2022-11-02T01:26:30Z

The Rollup alias work is separately under review at https://github.com/opensearch-project/index-management/pull/445/files, so let us exclude Rollup action from this Alias code-path.

The question from @bowenlan-amzn has still not been addressed We can see there're use cases some actions (shrink, rollover, rollup) create new index and we would like some alias to be add to the new index..

Most often, the alias action will work in conjunction with another action (As multiple use-cases have also pointed), and in cases the new index is created, the ISM workflow must provide in-built support to add alias-ing to the new index (some options were laid out in Bowen's comment).

bowenlan-amzn · 2022-11-02T22:48:37Z

The Rollup alias work is separately under review at https://github.com/opensearch-project/index-management/pull/445/files, so let us exclude Rollup action from this Alias code-path.

If an index is under a rollup target alias, ISM policy should not remove this alias from it. Is this what you are thinking about? We can add an issue to enhance this later. @khushbr

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

khushbr · 2022-11-03T01:46:25Z

The Rollup alias work is separately under review at https://github.com/opensearch-project/index-management/pull/445/files, so let us exclude Rollup action from this Alias code-path.

If an index is under a rollup target alias, ISM policy should not remove this alias from it. Is this what you are thinking about? We can add an issue to enhance this later. @khushbr

That is fine by me. @goyamegh Let us open an issue to track the conflicting alias action not allowed.

bowenlan-amzn · 2022-11-03T16:57:53Z

src/test/kotlin/org/opensearch/indexmanagement/indexstatemanagement/action/AliasActionIT.kt

+        }
+    }
+
+    fun `test adding alias to index using ctx`() {


What's different between this one and the first test test adding alias to index?

This was added earlier when the changes were intended for any index, not just managed index. As discussed, feel free to remove this as it is redundant now.

* Adds an alias action (#35) Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Enabling alias action on managed indices only * Add checks to aid debugging and testing steps for using alias action Signed-off-by: Megha Goyal <goyamegh@amazon.com> Signed-off-by: Megha Goyal <goyamegh@amazon.com> (cherry picked from commit 50072af)

* Adds an alias action (#35) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com>

madhukarmmallia-plivo · 2022-11-21T14:09:23Z

@goyamegh @bowenlan-amzn : I have a query here with respect to dynamic aliases. If the policy is used for multiple index name patterns, can this feature be used for dynamically adding an alias using the mustache template (for example by passing {{ ctx.index }} to the add step in the alias?

bowenlan-amzn · 2022-11-21T22:25:00Z

@goyamegh @bowenlan-amzn : I have a query here with respect to dynamic aliases. If the policy is used for multiple index name patterns, can this feature be used for dynamically adding an alias using the mustache template (for example by passing {{ ctx.index }} to the add step in the alias?

@madhukarmmallia-plivo
I am not sure ctx.index as the alias name is useful, because this alias is gonna be different for every indices, but normally one alias contains a group of indices. Maybe what you want is to parse sth from the index name to be used as the alias? I suggest to create different policy for different index patterns, and in each one you can define a different alias name to add.
Feel free to open a feature request issue to discuss more on this.

…opensearch-project#589) * Adds an alias action (opensearch-project#35) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com>

…opensearch-project#589) * Adds an alias action (opensearch-project#35) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> Signed-off-by: Ronnak Saxena <ronsax@amazon.com>

Adds an alias action (opensearch-project#35)

e0bb3ef

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

goyamegh requested a review from a team October 20, 2022 17:43

bowenlan-amzn added the v2.4.0 'Issues and PRs related to version v2.4.0' label Oct 20, 2022

khushbr reviewed Oct 24, 2022

View reviewed changes

src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/action/AliasAction.kt Show resolved Hide resolved

bowenlan-amzn reviewed Oct 24, 2022

View reviewed changes

bowenlan-amzn added the backport 2.x label Oct 27, 2022

Enabling alias action on managed indices only

dc380a5

goyamegh self-assigned this Oct 28, 2022

goyamegh marked this pull request as draft October 28, 2022 23:49

Merge branch 'opensearch-project:main' into main-goyamegh

7989760

Angie-Zhang removed the v2.4.0 'Issues and PRs related to version v2.4.0' label Oct 31, 2022

goyamegh marked this pull request as ready for review October 31, 2022 21:06

bowenlan-amzn reviewed Nov 1, 2022

View reviewed changes

khushbr reviewed Nov 2, 2022

View reviewed changes

Add checks to aid debugging and testing steps for using alias action

ae7021d

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

khushbr approved these changes Nov 3, 2022

View reviewed changes

khushbr added the v2.4.0 'Issues and PRs related to version v2.4.0' label Nov 3, 2022

bowenlan-amzn approved these changes Nov 3, 2022

View reviewed changes

bowenlan-amzn added the backport 2.4 label Nov 3, 2022

bowenlan-amzn reviewed Nov 3, 2022

View reviewed changes

bowenlan-amzn merged commit 50072af into opensearch-project:main Nov 3, 2022

opensearch-trigger-bot bot mentioned this pull request Nov 3, 2022

[Backport 2.x] Adds an alias action (#35) #589

Merged

opensearch-trigger-bot bot mentioned this pull request Nov 3, 2022

[Backport 2.4] Adds an alias action (#35) #590

Merged

khushbr pushed a commit that referenced this pull request Nov 3, 2022

Adds an alias action (#35) (#575) (#590)

8f78467

* Adds an alias action (#35) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com>

khushbr pushed a commit that referenced this pull request Nov 3, 2022

Adds an alias action (#35) (#575) (#589)

580370a

* Adds an alias action (#35) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com>

bowenlan-amzn added the needs-documentation label Nov 7, 2022

This was referenced Nov 8, 2022

[DOC] ISM Alias action opensearch-project/documentation-website#1835

Closed

[FEATURE] Add an alias action to ISM policy opensearch-project/index-management-dashboards-plugin#374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds an alias action (#35) #575

Adds an alias action (#35) #575

goyamegh commented Oct 20, 2022 •

edited by khushbr

Loading

bowenlan-amzn commented Oct 24, 2022

bowenlan-amzn Oct 24, 2022

goyamegh Oct 28, 2022

bowenlan-amzn Oct 24, 2022

goyamegh Oct 28, 2022

codecov-commenter commented Oct 31, 2022

bowenlan-amzn Nov 1, 2022

goyamegh Nov 3, 2022

khushbr Nov 2, 2022

goyamegh Nov 3, 2022

khushbr Nov 2, 2022

goyamegh Nov 3, 2022

khushbr commented Nov 2, 2022

bowenlan-amzn commented Nov 2, 2022

khushbr commented Nov 3, 2022

bowenlan-amzn Nov 3, 2022

goyamegh Nov 10, 2022

madhukarmmallia-plivo commented Nov 21, 2022

bowenlan-amzn commented Nov 21, 2022

Adds an alias action (#35) #575

Adds an alias action (#35) #575

Conversation

goyamegh commented Oct 20, 2022 • edited by khushbr Loading

bowenlan-amzn commented Oct 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Oct 31, 2022

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khushbr commented Nov 2, 2022

bowenlan-amzn commented Nov 2, 2022

khushbr commented Nov 3, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madhukarmmallia-plivo commented Nov 21, 2022

bowenlan-amzn commented Nov 21, 2022

goyamegh commented Oct 20, 2022 •

edited by khushbr

Loading