Improve test reporting to avoid inconsequential buildbot failures

[wd15]

We need to improve the test suite in the following ways:

• Individual test execution should be environment dependent in certain cases where dependencies are non-critical.
• The test suite should warn when non-critical packages are missing loudly at the end of the tests.
• These warnings should not indicate a failure or error to Buildbot.
• Maybe doctest could report how many tests were not executed as well.

Test execution should be environment dependent whan any of the following dependencies are not met.

• gmsh
• scipy
• vtk or mayavi in some way

Any others?

Presumably tests are not run using trilinos if the --trilinos flag is not used. There may be some tests in fipy/matrices where this is not the case.

In a similar vein it might be a good idea to update the buildbot script so it is easier to specify unique buildbot configurations. For example, if we can't install trilinos on Windows, we don't want to run the --trilinos option or the parallel versions of the tests. Currently that is all hard wires into the script. Also add sdist and build_docs options that auto upload???

Imported from trac ticket #390, created by wd15 on 11-18-2011 at 12:45, last modified: 01-31-2012 at 14:36

[guyer]

doctest directives to skip inappropriate tests are implemented in r4987 (see documentation:GUIDELINES.txt).

slave-specific buildbot configurations remains to be done.

Trac comment by guyer on 11-22-2011 at 17:50

[guyer]

The [http://build.cmi.kent.edu:8010/console buildbot] configuration is now easier to customize to the capabilities of different slaves. Still need to actually adjust the script (or install the prerequisites) for, e.g., PyAmg on both Ubuntu and OS X runs, PySvn on Ubuntu, and the proper MPI command on Ubuntu.

Trac comment by guyer on 12-02-2011 at 18:19

[guyer]

slaves are configured and doctests are flagged

Trac comment by guyer on 01-31-2012 at 14:36