Optimize the manifest workflow to reduce the run-time

[prudhvigodithi]

Is your feature request related to a problem? Please describe

Current manifest workflow which is being called by the GH action versions.yml takes lot of time to run and randomly fails with Error: The operation was canceled. The cause of the issue is executing the methods gradle_cmd which calls ./gradlew properties and then outputs the version, based on this version the flow executes to raise a PR as follows #3008

With ./gradlew properties running on the OpenSearch repo taking lot of time the GH runner breaks with Error: The operation was canceled where as this works on local without any error.

Describe the solution you'd like

To get the version instead of running ./gradlew properties which starts Gradle Daemon runs the task Task :properties , its easy to query the file buildSrc/version.properties from the OpenSearch repo to get the version, something like OPENSEARCH_VERSION=$(cat buildSrc/version.properties | grep opensearch | cut -d= -f2 | grep -oE '[0-9.]+') which is already being used in version increment workflow, this should significantly reduce the time in the manifest workflow execution.

Describe alternatives you've considered

No response

Additional context

No response

[prudhvigodithi]

Adding @dblock and @opensearch-project/engineering-effectiveness.

[peterzhuamazon]

Using version.properties directly might cause issue later if there is more than one opensearch key value pairs.

[prudhvigodithi]

Should there be only one OpenSearch version in version.properties ? @peterzhuamazon

[dblock]

Maybe we can turn the workflow into a matrix where each runs in parallel? Here's how to generate a matrix at runtime: https://code.dblock.org/2021/09/03/generating-task-matrix-by-looping-over-repo-files-with-github-actions.html

[prudhvigodithi]

One more data point, adding PR manually #3164

[bbarani]

This issue seems to be recurring for every upcoming release version and I would like to close it out as soon as possible. @prudhvigodithi @zelinh Can we move ahead with @dblock suggestion?

[zelinh]

Another dp here #3192.
I would like we prioritize this issue so we could get unblocked.

[rishabh6788]

As per my understanding we are running ./gradlew properties -Dbuild.snapshot=false and writing the property output. We are then reading the properties to get the value of version key to get the version here.
I do not see any other usage of values returned by the gradle command and believe it is safe to be replaced by @prudhvigodithi proposed solution, unless, I may have missed any other usage of the properties value retrieved. Please correct me if my understanding is wrong @peterzhuamazon @dblock.

Also, @peterzhuamazon since opensearch is supposed to be a key, it cannot be duplicated, right?

@dblock Can you also please add some more details regarding the solution you proposed with respect to running in parallel?
@bbarani @gaiksaya @zelinh

[prudhvigodithi]

Hey @rishabh6788 @dblock just to add running in parallel with the solution proposed (a matrix strategy for GitHub runners) might not be a right solution here, it would work for instances when a loop of isolated process has to be called. For manifest workflow its all single process ./manifests.sh update and does not take any inputs example like ./ci.sh ${{ matrix.manifest }} --snapshot where ./ci.sh can be split into multiple process each process having an input of manifests under the manifests folder.

Here ./manifests.sh update is a single process that add manifests based on OpenSearch Core branches, if we have to run in a matrix may be we have modify the code manifest workflow like ./manifests.sh update ${{ matrix.core_branch }}, having this we can run in a loop for each core branch, rather than this querying version.properties would be easy and less complex approach go with.

Thank you

[gaiksaya]

Until the optimization takes place, we moved the workflow to jenkins here: https://build.ci.opensearch.org/job/manifest-update/
It was able to create a PR with 2.8.0 #3433

[gaiksaya]

Closing this issue as we have moved the workflow to run on higher specs machine that can handle gradle properties check which seems more appropriate than parsing the version.properties file.

Please re-open if you think proposed solution above is more appropriate.

Thanks!