TESTS: replace old test suite with the new one

get rid of Bats definitively

Author: thomasleveil

.travis.yml

@@ -4,8 +4,11 @@ services:
   - docker
 
 env:
-    global:
-        - DOCKER_VERSION=1.13.1-0~ubuntu-trusty
+  global:
+  - DOCKER_VERSION=1.13.1-0~ubuntu-trusty
+  matrix:
+  - TEST_TARGET: test-debian
+  - TEST_TARGET: test-alpine
 
 before_install:
   # list docker-engine versions
@@ -14,22 +17,8 @@ before_install:
   - sudo apt-get -o Dpkg::Options::="--force-confnew" install -y --force-yes docker-engine=${DOCKER_VERSION}
   - docker version
   - docker info
-  # install bats
-  - sudo add-apt-repository ppa:duggan/bats --yes
-  - sudo apt-get update -qq
-  - sudo apt-get install -qq bats
-  # prepare docker images
+  # prepare docker test requirements
   - make update-dependencies
 
-matrix:
-  include:
-    - env: TEST_ID=test-debian
-    - env: TEST_ID=test-alpine
-    - env: TEST_ID=test2-debian
-    - env: TEST_ID=test2-alpine
-  allow_failures:
-    - env: TEST_ID=test2-debian
-    - env: TEST_ID=test2-alpine
-
 script:
-  - make $TEST_ID
+  - make $TEST_TARGET

Makefile

@@ -1,27 +1,16 @@
 .SILENT :
-.PHONY : test test2
+.PHONY : test-debian test-alpine test
+
 
 update-dependencies:
-	docker pull jwilder/docker-gen:0.7.3
-	docker pull nginx:1.11.9
-	docker pull nginx:1.11.9-alpine
-	docker pull python:3
-	docker pull rancher/socat-docker:latest
-	docker pull appropriate/curl:latest
-	docker pull docker:1.10
+	test/requirements/build.sh
 
-test-debian:
-	docker build -t jwilder/nginx-proxy:bats .
-	bats test
+test-debian: update-dependencies
+	docker build -t jwilder/nginx-proxy:test .
+	test/pytest.sh
 
-test-alpine:
-	docker build -f Dockerfile.alpine -t jwilder/nginx-proxy:bats .
-	bats test
+test-alpine: update-dependencies
+	docker build -f Dockerfile.alpine -t jwilder/nginx-proxy:test .
+	test/pytest.sh
 
 test: test-debian test-alpine
-
-test2-debian:
-	$(MAKE) -C test2 test-debian
-
-test2-alpine:
-	$(MAKE) -C test2 test-alpine

README.md

@@ -325,6 +325,22 @@ Before submitting pull requests or issues, please check github to make sure an e
 
 #### Running Tests Locally
 
-To run tests, you'll need to install [bats 0.4.0](https://github.com/sstephenson/bats).
+To run tests, you need to prepare the docker image to test which must be tagged `jwilder/nginx-proxy:test`:
+
+    docker build -t jwilder/nginx-proxy:test .  # build the Debian variant image
+    
+and call the [test/pytest.sh](test/pytest.sh) script.
+
+Then build the Alpine variant of the image:
+
+    docker build -f Dockerfile.alpine -t jwilder/nginx-proxy:test .  # build the Alpline variant image
+
+and call the [test/pytest.sh](test/pytest.sh) script again.
+
+
+If your system has the `make` command, you can automate those tasks by calling:
 
     make test
+    
+
+You can learn more about how the test suite works and how to write new tests in the [test/README.md](test/README.md) file.

test/README.md

@@ -1,14 +1,107 @@
-Test suite
-==========
-
-This test suite is implemented on top of the [Bats](https://github.com/sstephenson/bats/blob/master/README.md) test framework.
-
-It is intended to verify the correct behavior of the Docker image `jwilder/nginx-proxy:bats`.
-
-Running the test suite
-----------------------
-
-Make sure you have Bats installed, then run:
-
-    docker build -t jwilder/nginx-proxy:bats .
-    bats test/
+Nginx proxy test suite
+======================
+
+Install requirements
+--------------------
+
+You need [python 2.7](https://www.python.org/) and [pip](https://pip.pypa.io/en/stable/installing/) installed. Then run the commands:
+
+    requirements/build.sh
+    pip install -r requirements/python-requirements.txt
+
+If you can't install those requirements on your computer, you can alternatively use the _pytest.sh_ script which will run the tests from a Docker container which has those requirements.
+
+
+Prepare the nginx-proxy test image
+----------------------------------
+
+    docker build -t jwilder/nginx-proxy:test ..
+
+or if you want to test the alpine flavor:
+
+    docker build -t jwilder/nginx-proxy:test -f Dockerfile.alpine ..
+
+make sure to tag that test image exactly `jwilder/nginx-proxy:test` or the test suite won't work.
+
+
+Run the test suite
+------------------
+
+    pytest
+
+need more verbosity ?
+
+    pytest -s
+
+
+Run one single test module
+--------------------------
+
+    pytest test_nominal.py
+
+
+Write a test module
+-------------------
+
+This test suite uses [pytest](http://doc.pytest.org/en/latest/). The [conftest.py](conftest.py) file will be automatically loaded by pytest and will provide you with two useful pytest [fixtures](http://doc.pytest.org/en/latest/fixture.html#fixture): 
+
+- docker_compose
+- nginxproxy
+
+
+### docker_compose fixture
+
+When using the `docker_compose` fixture in a test, pytest will try to find a yml file named after your test module filename. For instance, if your test module is `test_example.py`, then the `docker_compose` fixture will try to load a `test_example.yml` [docker compose file](https://docs.docker.com/compose/compose-file/).
+
+Once the docker compose file found, the fixture will remove all containers, run `docker-compose up`, and finally your test will be executed.
+
+The fixture will run the _docker-compose_ command with the `-f` option to load the given compose file. So you can test your docker compose file syntax by running it yourself with:
+
+    docker-compose -f test_example.yml up -d
+
+In the case you are running pytest from within a docker container, the `docker_compose` fixture will make sure the container running pytest is attached to all docker networks. That way, your test will be able to reach any of them.
+
+In your tests, you can use the `docker_compose` variable to query and command the docker daemon as it provides you with a [client from the docker python module](https://docker-py.readthedocs.io/en/2.0.2/client.html#client-reference).
+
+Also this fixture alters the way the python interpreter resolves domain names to IP addresses in the following ways:
+
+Any domain name containing the substring `nginx-proxy` will resolve to the IP address of the container that was created from the `jwilder/nginx-proxy:test` image. So all the following domain names will resolve to the nginx-proxy container in tests:
+- `nginx-proxy`
+- `nginx-proxy.com`
+- `www.nginx-proxy.com`
+- `www.nginx-proxy.test`
+- `www.nginx-proxy`
+- `whatever.nginx-proxyooooooo`
+- ...
+
+Any domain name ending with `XXX.container.docker` will resolve to the IP address of the XXX container.
+- `web1.container.docker` will resolve to the IP address of the `web1` container
+- `f00.web1.container.docker` will resolve to the IP address of the `web1` container
+- `anything.whatever.web2.container.docker` will resolve to the IP address of the `web2` container
+
+Otherwise, domain names are resoved as usual using your system DNS resolver.
+
+
+### nginxproxy fixture
+
+The `nginxproxy` fixture will provide you with a replacement for the python [requests](https://pypi.python.org/pypi/requests/) module. This replacement will just repeat up to 30 times a requests if it receives the HTTP error 404 or 502. This error occurs when you try to send queries to nginx-proxy too early after the container creation.
+
+Also this requests replacement is preconfigured to use the Certificate Authority root certificate [certs/ca-root.crt](certs/) to validate https connections.
+
+Furthermore, the nginxproxy methods accept an additional keyword parameter: `ipv6` which forces requests made against containers to use the containers IPv6 address when set to `True`. If IPv6 is not supported by the system or docker, that particular test will be skipped.
+
+    def test_forwards_to_web1_ipv6(docker_compose, nginxproxy):
+        r = nginxproxy.get("http://web1.nginx-proxy.tld/port", ipv6=True)
+        assert r.status_code == 200   
+        assert r.text == "answer from port 81\n"
+
+
+
+### The web docker image
+
+When you ran the `requirements/build.sh` script earlier, you built a [`web`](requirements/README.md) docker image which is convenient for running a small web server in a container. This image can produce containers that listens on multiple ports at the same time.
+
+
+### Testing TLS
+
+If you need to create server certificates, use the [`certs/create_server_certificate.sh`](certs/) script. Pytest will be able to validate any certificate issued from this script.