Run conformance tests regularly 🏃♀️ #3865
Conversation
In tektoncd#3400 we added an initial conformance test, and added a build tag `conformance` to make it so that folks could run just conformance tests if they wanted to. From what I can see it doesn't look like we use this tag at all in our automation, so I want to add it to the tags we use to run our end to end test to make sure this continues working over time. In the long run I think we might want to explore a couple other alternatives for implementing conformance tests, e.g. including making use of our example tests (though those dont express the desired end state), maybe a tool like [kuttl](https://github.com/kudobuilder/kuttl#kuttl) which @chhsia0 has mentioned in preivous working groups, or like the approach that knative seems to be taking via https://github.com/knative-sandbox/reconciler-test But in the short run let's at least make sure this test keeps working :D Also renamed the test to make it a bit more clear what this test is for in case it fails.
tekton-robot
added
the
release-note-none
tekton-robot
added
the
size/XS
tekton-robot
added
the
approved
|
/test pull-tekton-pipeline-integration-tests |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sbwsg, vdemeester The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| header "Running Go e2e tests" | ||
| go_test_e2e -timeout=20m ./test/... || failed=1 | ||
| go_test_e2e tags=e2e,conformance -timeout=20m ./test/... || failed=1 |
There was a problem hiding this comment.
Should we run them seperate though ? (maybe even different jobs in the end)
There was a problem hiding this comment.
eventually i agree, but in the meantime, probably better to run them than not? (i think there's only 1 anyway)
|
@bobcatfish: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
1 similar comment
|
@bobcatfish: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@bobcatfish: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
tekton-robot
added
the
needs-rebase
|
@bobcatfish needs a rebase 🐼 |
|
I seem to be unable to get back to this so just gonna close it for now XD |
Changes
In #3400 we added an initial
conformance test, and added a build tag
conformanceto make it so thatfolks could run just conformance tests if they wanted to. From what I
can see it doesn't look like we use this tag at all in our automation,
so I want to add it to the tags we use to run our end to end test to
make sure this continues working over time.
In the long run I think we might want to explore a couple other
alternatives for implementing conformance tests, e.g. including making
use of our example tests (though those dont express the desired end
state), maybe a tool like kuttl
which @chhsia0 has mentioned in preivous working groups, or like the
approach that knative seems to be taking via
https://github.com/knative-sandbox/reconciler-test
But in the short run let's at least make sure this test keeps working :D
Also renamed the test to make it a bit more clear what this test is for
in case it fails.
/kind clenaup
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:
See the contribution guide for more details.
Double check this list of stuff that's easy to miss:
cmddir, please updatethe release Task to build and release this image.
Reviewer Notes
If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.
Release Notes