Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README for smoke tests with usages and examples #5244

Merged
merged 4 commits into from
Jan 28, 2025
Merged

Update README for smoke tests with usages and examples #5244

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
66a1e27
Add readme for smoke tests
zelinh Jan 7, 2025
76e3491
Merge remote-tracking branch 'upstream/main' into smoke-test-guide
zelinh Jan 15, 2025
f212d81
Add smoke tests in README file from test workflow
zelinh Jan 15, 2025
4922498
Update wording
zelinh Jan 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 44 additions & 11 deletions src/test_workflow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
- [Test.sh Options](#testsh-options)
- [Integration Tests](#integration-tests)
- [Backwards Compatibility Tests](#backwards-compatibility-tests)
- [Smoke Tests](#smoke-tests)
- [Performance Tests](#performance-tests)
- [Identifying Regressions in Performance Tests](#identifying-regressions-in-performance-tests)
- [Identifying Regressions in Nightly Performance Tests](#identifying-regressions-in-nightly-performance-tests)
Expand All @@ -15,6 +16,7 @@
- [integTest job](#integtest-job)
- [bwcTest job](#bwctest-job)
- [perfTest job](#perftest-job)
- [smokeTest job](#smoketest-job)
- [Manifest Files](#manifest-files)
- [Dependency Management](#dependency-management)
- [S3 Permission Model](#s3-permission-model)
Expand All @@ -28,15 +30,15 @@ Testing is run via `./test.sh`.

The following options are available.

| name | description |
|------------------------|-----------------------------------------------------------------------------------------------------------------------|
| test-type | Run tests of a test suite. [integ-test, bwc-test, perf-test, benchmark-test execute-test, benchmark-test compare] |
| test-manifest-path | Specify a test manifest path. |
| --paths | Location of manifest(s). |
| --test-run-id | Unique identifier for a test run. |
| --component [name ...] | Test a subset of specific components. |
| --keep | Do not delete the temporary working directory on both success or error. |
| -v, --verbose | Show more verbose output. |
| name | description |
|------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| test-type | Run tests of a test suite. [integ-test, bwc-test, smoke-test, perf-test, benchmark-test execute-test, benchmark-test compare] |
| test-manifest-path | Specify a test manifest path. |
| --paths | Location of manifest(s). |
| --test-run-id | Unique identifier for a test run. |
| --component [name ...] | Test a subset of specific components. |
| --keep | Do not delete the temporary working directory on both success or error. |
| -v, --verbose | Show more verbose output. |

### Integration Tests

Expand Down Expand Up @@ -122,6 +124,29 @@ On CI level for plugins, security certificates need to be manually imported when

See [anomaly-detection#766](https://github.com/opensearch-project/anomaly-detection/pull/766) or [observability#1366](https://github.com/opensearch-project/observability/pull/1366) for more information.

### Smoke Tests
Run Smoke tests invoking `run_smoke_test.py` in each component from a distribution manifest.

The smoke tests are designed to verify basic functionality of a distribution bundle with all plugins installed.
Each plugin being tested has its own test specification YAML file, located in the [OpenSearch Build repository](https://github.com/opensearch-project/opensearch-build/tree/main/src/test_workflow/smoke_test/smoke_tests_spec), organized by major version. These test specification files list all testing API cases. The smoke test workflow executes each API case and compares the results with the API specifications from the [OpenSearch API Specification repository](https://github.com/opensearch-project/opensearch-api-specification) to perform basic validation and testing.

Usage:
To execute the smoke tests, use the following command:
```bash
./test.sh smoke-test <test-manifest-path> <target>
```

Examples
Running smoke tests locally with artifacts built and assembled:
```bash
./test.sh smoke-test manifests/2.19.0/opensearch-2.19.0-test.yml --paths opensearch=/workspace/opensearch-build/tar
```

Running smoke tests against an existing build from CI:
```bash
./test.sh smoke-test manifests/2.19.0/opensearch-2.19.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.19.0/10545/linux/x64/tar/
```

### Performance Tests

TODO: Add instructions for running performance tests with `test.sh`
Expand Down Expand Up @@ -249,7 +274,7 @@ The development of `test-orchestration-pipeline` is tracked by meta issue [#123]

It is a Jenkins job that runs integration tests on a build artifact. It reads the build artifact composition from the associated manifest files and spins up parallel, independent integrationTest runs for each component built inside the artifact. For instance, if the artifact is a full distribution, which has all OpenSearch plugins, the job will kick off integration test suite for each individual plugin. Each plugin integration tests would run against a dedicated single node cluster, which is created from the built artifact. Once all integration tests complete, this job publishes the test results to an S3 bucket.

See the integration test [configuration file](jenkins/opensearch/integ-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/integ-test/)
See the integration test [configuration file](../../jenkins/opensearch/integ-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/integ-test/)

The development of `integTest` job is tracked by meta issue [#818](https://github.com/opensearch-project/opensearch-build/issues/818)

Expand All @@ -259,7 +284,7 @@ It is a Jenkins job that runs bwc tests on the current version and compatible bw

When the bwc test is triggered for a particular component, the tests set up their own cluster and test the required functionalities in the upgrade paths, for the example above, a multi-node cluster starts with bwc versions of OpenSearch and AD installed on it, one or more nodes are upgraded to the current version of OpenSearch and AD installed on it and backwards compatibility is tested. The plugins would add tests for all bwc versions (similar to OpenSearch core) and they can be triggered from the bwcTest job.

See the bwc test [configuration file](jenkins/opensearch/bwc-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/bwc-test/)
See the bwc test [configuration file](../../jenkins/opensearch/bwc-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/bwc-test/)

The development of the bwc test automation is tracked by meta issue [#90](https://github.com/opensearch-project/opensearch-build/issues/90).

Expand Down Expand Up @@ -290,6 +315,14 @@ After the performance test completes, it will report back the test results as we

The development is tracked by meta issue [#126](https://github.com/opensearch-project/opensearch-build/issues/126)

#### smokeTest job

It is a Jenkins job designed to run smoke tests on bundle artifacts. It retrieves the bundle artifacts using the provided manifest URL parameter and spins up a cluster with all plugins installed. The cluster is deployed using the default configuration without any customizations. Smoke tests for all components are executed sequentially, and the results are returned at the end.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
It is a Jenkins job designed to run smoke tests on bundle artifacts. It retrieves the bundle artifacts using the provided manifest URL parameter and spins up a cluster with all plugins installed. The cluster is deployed using the default configuration without any customizations. Smoke tests for all components are executed sequentially, and the results are returned at the end.
It is a Jenkins job designed to run smoke tests on distribution artifacts. It retrieves the distribution artifacts using the provided manifest URL parameter and spins up a cluster with all plugins installed. The cluster is deployed using the default configuration without any customization. Smoke tests for all components are executed sequentially, and the results are returned at the end.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the cluster cleaned up after every component?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated based on suggestions. Thanks.

Is the cluster cleaned up after every component?

No. We will only clean up after all components test completed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might cause conflicts right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have test requirements that if it causes conflicts, we shouldn't add it into the framework.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think that is feasible in the long term. Adding @prudhvigodithi @rishabh6788 @peterzhuamazon to get some more inputs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our current design involves spinning up a single cluster and accommodating non-conflicting smoke test cases for multiple plugins. Perhaps we could create a separate issue to discuss this further?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please create an issue to discuss this. We can move forward with merge.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cut an enhancement issue here. We may revisit it to analyze its benefits further.
#5274


See the smoke test [configuration file](../../jenkins/opensearch/smoke-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/smoke-test/)

The development is tracked by meta issue [#5223](https://github.com/opensearch-project/opensearch-build/issues/5223)

## Manifest Files

Manifest files are configurations for a particular bundle. `test-workflow` uses three types of manifest files to run test suites.
Expand Down
Loading