From b460e9df77e41ef3658507a39393c7f57ff2bcf0 Mon Sep 17 00:00:00 2001 From: James Mead Date: Mon, 12 Oct 2015 15:45:13 +0100 Subject: [PATCH 1/4] Extract instructions for adding new regression tests These instructions don't feel core to Smart Answers. --- doc/adding-new-regression-tests.md | 71 +++++++++++++++++++++++++++++ doc/smart-answer-flows.md | 72 +----------------------------- 2 files changed, 72 insertions(+), 71 deletions(-) create mode 100644 doc/adding-new-regression-tests.md diff --git a/doc/adding-new-regression-tests.md b/doc/adding-new-regression-tests.md new file mode 100644 index 00000000000..6f75979be9a --- /dev/null +++ b/doc/adding-new-regression-tests.md @@ -0,0 +1,71 @@ +# Adding new regression tests + +1. Update the flow to replace any single line conditionals around `Phraselist`s with multiple line conditionals. This is so that we get useful information from the running the coverage utility. Single line conditionals will show up as having been exercised irrespective of whether they caused something to be added to the `Phraselist`. + + # Replace single line conditional + phrases << :new_phrase if condition + + # With multiple line alternative + if condition + phrases << :new_phrase + end + +2. Generate a set of responses for the flow that you want to add regression tests to. + + $ rails r script/generate-questions-and-responses-for-smart-answer.rb \ + + +3. Commit the generated questions-and-responses.yml file (in test/data) to git. + +4. Change the file by adding/removing and changing the responses: + + * Add responses for any of the TODO items in the file. + + * Remove responses that you don't think cause the code to follow different branches, e.g. it might be useful to remove all but one (or a small number) of countries to avoid a combinatorial explosion of input responses. + + * Combine responses for checkbox questions where the effect of combining them doesn't affect the number of branches of the code that are exercised. + +5. Commit the updated questions-and-responses.yml file to git. + +6. Generate a set of input responses and expected results for the Smart Answer. + + $ rm -rf coverage && \ + TEST_COVERAGE=true \ + rails r script/generate-responses-and-expected-results-for-smart-answer.rb \ + + +7. Inspect the code coverage report for the Smart Answer under test (`open coverage/rcov/index.html` and find the smart answer under test). + + * If all the branches in the flow have been exercised then you don't need to do anything else at this time. + + * Code in node-level blocks (e.g. in `value_question`, `date_question`, `multiple_choice` & `outcome` blocks) will always be executed at *flow-definition-time*, and so coverage of these lines is of **no** significance when assessing test coverage of the flow logic. + + * Code in blocks inside node-level blocks (e.g. in `precalculate`, `next_node_calculation`, `validate` & `define_predicate` blocks) will be executed at *flow-execution-time*, and so coverage of these lines is of significance when assessing test coverage of the flow logic. + + * Coverage of code in ancillary classes (e.g. calculators) should also be considered at this point. + + * If there are branches in the flow that haven't been exercised then: + + * Determine the responses required to exercise those branches. + + * Go to Step 4, add the new responses and continue through the steps up to Step 7. + +8. Commit the generated responses-and-expected-results.yml file (in test/data) to git. + +9. Generate a yaml file containing the set of source files that this Smart Answer depends upon. The script will automatically take the ruby flow file, locale file and erb templates into account. You just need to supply it with the location of any additional files required by the Smart Answer (e.g. calculators and data files). This data is used to determine whether to run the regression tests based on whether the source files have changed. + + $ rails r script/generate-checksums-for-smart-answer.rb \ + \ + + +10. Commit the generated yaml file to git. + +11. Run the regression test to generate the Govspeak of each landing page and outcome reached by the set of input responses. + + $ RUN_REGRESSION_TESTS= \ + ruby test/regression/smart_answers_regression_test.rb + +If you want individual tests to fail early when differences are detected, set `ASSERT_EACH_ARTEFACT=true`. +Note that this more than doubles the time it takes to run regression tests. + +12. Commit the generated Govspeak files (in test/artefacts) to git. diff --git a/doc/smart-answer-flows.md b/doc/smart-answer-flows.md index 8e4bb9014ee..d3a7091a70a 100644 --- a/doc/smart-answer-flows.md +++ b/doc/smart-answer-flows.md @@ -260,74 +260,4 @@ This is how we were previously writing integration tests for Smart Answer flows. ### Adding regression tests to Smart Answers -We're not imagining introducing new regression tests but I think these instructions are still useful while we still have them in the project. - -1. Update the flow to replace any single line conditionals around `Phraselist`s with multiple line conditionals. This is so that we get useful information from the running the coverage utility. Single line conditionals will show up as having been exercised irrespective of whether they caused something to be added to the `Phraselist`. - - # Replace single line conditional - phrases << :new_phrase if condition - - # With multiple line alternative - if condition - phrases << :new_phrase - end - -2. Generate a set of responses for the flow that you want to add regression tests to. - - $ rails r script/generate-questions-and-responses-for-smart-answer.rb \ - - -3. Commit the generated questions-and-responses.yml file (in test/data) to git. - -4. Change the file by adding/removing and changing the responses: - - * Add responses for any of the TODO items in the file. - - * Remove responses that you don't think cause the code to follow different branches, e.g. it might be useful to remove all but one (or a small number) of countries to avoid a combinatorial explosion of input responses. - - * Combine responses for checkbox questions where the effect of combining them doesn't affect the number of branches of the code that are exercised. - -5. Commit the updated questions-and-responses.yml file to git. - -6. Generate a set of input responses and expected results for the Smart Answer. - - $ rm -rf coverage && \ - TEST_COVERAGE=true \ - rails r script/generate-responses-and-expected-results-for-smart-answer.rb \ - - -7. Inspect the code coverage report for the Smart Answer under test (`open coverage/rcov/index.html` and find the smart answer under test). - - * If all the branches in the flow have been exercised then you don't need to do anything else at this time. - - * Code in node-level blocks (e.g. in `value_question`, `date_question`, `multiple_choice` & `outcome` blocks) will always be executed at *flow-definition-time*, and so coverage of these lines is of **no** significance when assessing test coverage of the flow logic. - - * Code in blocks inside node-level blocks (e.g. in `precalculate`, `next_node_calculation`, `validate` & `define_predicate` blocks) will be executed at *flow-execution-time*, and so coverage of these lines is of significance when assessing test coverage of the flow logic. - - * Coverage of code in ancillary classes (e.g. calculators) should also be considered at this point. - - * If there are branches in the flow that haven't been exercised then: - - * Determine the responses required to exercise those branches. - - * Go to Step 4, add the new responses and continue through the steps up to Step 7. - -8. Commit the generated responses-and-expected-results.yml file (in test/data) to git. - -9. Generate a yaml file containing the set of source files that this Smart Answer depends upon. The script will automatically take the ruby flow file, locale file and erb templates into account. You just need to supply it with the location of any additional files required by the Smart Answer (e.g. calculators and data files). This data is used to determine whether to run the regression tests based on whether the source files have changed. - - $ rails r script/generate-checksums-for-smart-answer.rb \ - \ - - -10. Commit the generated yaml file to git. - -11. Run the regression test to generate the Govspeak of each landing page and outcome reached by the set of input responses. - - $ RUN_REGRESSION_TESTS= \ - ruby test/regression/smart_answers_regression_test.rb - -If you want individual tests to fail early when differences are detected, set `ASSERT_EACH_ARTEFACT=true`. -Note that this more than doubles the time it takes to run regression tests. - -12. Commit the generated Govspeak files (in test/artefacts) to git. +We're not imagining introducing new regression tests but I think [these instructions](doc/adding-new-regression-tests.md) are still useful while we still have them in the project. From 782cf39c84ab5490832ef68794099f602bf1a09a Mon Sep 17 00:00:00 2001 From: James Mead Date: Mon, 12 Oct 2015 15:46:26 +0100 Subject: [PATCH 2/4] First attempt at documentation about regression tests --- doc/regression-tests.md | 58 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 doc/regression-tests.md diff --git a/doc/regression-tests.md b/doc/regression-tests.md new file mode 100644 index 00000000000..d6d6f7803c8 --- /dev/null +++ b/doc/regression-tests.md @@ -0,0 +1,58 @@ +# Regression tests + +The project includes a set of regression tests. These tests are not *normally* run as part of the standard build, because they take a long time to run. You can run just the regression tests with the following command: + + $ RUN_REGRESSION_TESTS=true ruby test/regression/smart_answers_regression_test.rb + +You can run just the regression tests for a single flow using this command: + + $ RUN_REGRESSION_TESTS= ruby test/regression/smart_answers_regression_test.rb + +Note that the `RUN_REGRESSION_TESTS` environment variable can also be used in conjunction with the rake `test` task if you want to force regression tests to run as part of the standard build. + +By default most of the assertions in the regression tests are combined into a single assertion at the end. If you want the regression tests to fail fast then set `ASSERT_EACH_ARTEFACT=true`. However, you should note that this more than doubles the time it takes them to run. + +Running the test re-generates a set of HTML/Govspeak files in `test/artefacts` based on the files in `test/data`. + +## Test data + +* `-questions-and-responses.yml` - defines a set of responses to the flow's questions +* `-responses-and-expected-results.yml` - a record of the question & outcome nodes visited when the above responses are applied combinatorially + +## Artefacts + +The following artefacts are saved in `test/artefacts`: + +* `/.txt` - rendered Govspeak for landing page +* `/.html` - rendered HTML for question page +* `/.txt` - rendered Govspeak for outcome pages + +The `` is a forward-slash separated list of responses which closely relates to the URL paths to question & outcome pages in the app. + +The regression test fails if any of the following is true: + +* the newly generated artefact files differ at all from the committed files +* not all nodes are exercised by the test data +* the checksum data is out-of-date (see below) + +If you've added extra questions, responses or outcomes, then you should change the `test/data` files to exercise the new paths through the flow. See the instructions for [adding new regression tests](doc/adding-new-regression-tests.md) + +If there's a difference in the artefacts, you need to carefully review the changes to the artefacts to make sure they all relate to the changes you have made before committing them. + +Once you're happy that the changes to the files correspond to the changes to the test artefacts, you can update the checksums using the following command: + + $ rails r script/generate-checksums-for-smart-answer.rb + +When you've resolved all these issues, you should be able to run the regression tests for the flow as before and all the tests should pass. + +## Automatic trigger + +The regression test for a given flow are triggered automatically when you run the rake `test` task if you've made changed to any of the files whose checksums are listed in `test/data/-files.yml`. Although this won't always trigger the regression test when it should be run, it covers most common scenarios. + +If you've added new classes, modules or data which is used by a flow, you should add the relevant files to the checksums file. + +## Continuous integration + +The [main CI instance](https://ci-new.alphagov.co.uk/job/govuk_smart_answers/) and the [corresponding branches one](https://ci-new.alphagov.co.uk/job/govuk_smartanswers_branches/) run the rake `test` task and so work in the same way as above. + +We also have a [separate CI instance](https://ci-new.alphagov.co.uk/job/govuk_smart_answers_regressions/) which runs **all** the regression tests every so often. This should catch any scenarios missed by the automatic trigger. From 2b2b45df1d54152ceaff3922d03fae2c14ff8e66 Mon Sep 17 00:00:00 2001 From: James Mead Date: Mon, 12 Oct 2015 15:46:55 +0100 Subject: [PATCH 3/4] Link to regression tests doc from CONTRIBUTING.md --- CONTRIBUTING.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8e93e190ded..46d83150caa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -20,3 +20,5 @@ ## Testing ## Write tests. + +Make sure the [regression tests](doc/regression-tests.md) are passing. From c072ded3f8b641dc66836fe14d3f683a9a07c34c Mon Sep 17 00:00:00 2001 From: James Mead Date: Mon, 12 Oct 2015 15:50:02 +0100 Subject: [PATCH 4/4] Link to regression test doc from merging-content-prs doc The former gives more detail than the latter. --- doc/merging-content-prs.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/merging-content-prs.md b/doc/merging-content-prs.md index bd7c45cec14..cac6ce8b0ee 100644 --- a/doc/merging-content-prs.md +++ b/doc/merging-content-prs.md @@ -40,3 +40,5 @@ files relating to the regression tests e.g. file checksums, Govspeak artefacts, 10. Push the branch to GitHub and submit a new pull request so that people have a chance to review the changes and a Continuous Integration build is triggered. Close the original pull request. $ git push origin + +See documentation on [regression tests](doc/regression-tests.md) for further details.