Add performance and longevity testing validation to the release template

[travisbenedict]

Signed-off-by: Travis Benedict benedtra@amazon.com

Description

Add steps to the release template for validating performance and longevity test results. These steps are not necessarily exhaustive but can serve as a starting point for our release process. Ultimately, these validations should be automated as our performance testing automation is built out more.

More details on the testing setup can be found in this issue outlining the results of performance experiments that I conducted - opensearch-project/OpenSearch#2461

Issues Resolved

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dblock]

I think the long page of details of how to identify regression tests doesn't belong here. We have a TODO in https://github.com/opensearch-project/opensearch-build/tree/main/src/test_workflow#performance-tests to fill out performance testing, care to move it there as part of this PR?

[travisbenedict]

These instructions can be updated with #1671 or after it's merged

[dblock]

Add to TOC, cleanup capitalization to match other topics.

I would rename to "Identifying Regressions in Performance Tests"

[dblock]

This disclaimer is scary. It says that you cannot trust results. Try to be more prescriptive, remove that this is a disclaimer. What is one supposed to actually do? Run tests, then compare results. That's what this doc should say.

[travisbenedict]

Good points. I'll rewrite this section

[dblock]

This should be a section of its own. What happens if the error rates are higher? What does one do?

[dblock]

There's zero information on what longevity tests are anywhere in these docs. To an uneducated reader it's impossible to understand what one does with it. Please provide context to all these things.

[codecov-commenter]

Codecov Report

Merging #1752 (1841fc2) into main (faf26d1) will increase coverage by 5.20%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #1752      +/-   ##
=============================================
+ Coverage     94.79%   100.00%   +5.20%     
=============================================
  Files           169         6     -163     
  Lines          3536       105    -3431     
  Branches         26        19       -7     
=============================================
- Hits           3352       105    -3247     
+ Misses          181         0     -181     
+ Partials          3         0       -3     

Impacted Files	Coverage Δ
src/test_workflow/integ_test/integ_test_runner.py
...ests/jenkins/jobs/CopyContainer_docker_Jenkinsfile
...est_workflow/bwc_test/bwc_test_suite_opensearch.py
.../assemble_workflow/bundle_opensearch_dashboards.py
tests/jenkins/jobs/CreateReleaseTag_Jenkinsfile
...workflow/opensearch/build_artifact_check_plugin.py
src/run_manifests.py
src/ci_workflow/ci_check_list_dist.py
src/paths/tree_walker.py
...bs/PrintArtifactDownloadUrlsForStaging_Jenkinsfile
... and 153 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update faf26d1...1841fc2. Read the comment docs.

[dblock]

This is a lot better, thank you! I went super nitpicky on the comments, feel free to make some/none/all the changes.

[dblock]

Did you mean all these to the at the same level?

[travisbenedict]

I meant for the Nightly test section to be a subsection of performance tests in general, since it's worth reading that section first to get some background.

[travisbenedict]

Thanks for the help on this @dblock. I really appreciate the feedback