Async API testing

[ghost]

Looking to test with Gabbi a REST API that for some endpoints spawns a job (with job ID returned in JSON). This API expects the client to poll on a job status endpoint for that job ID to show complete (job status reported in JSON). Once job is complete, it would be great to do a GET on the item created to be sure it's actually there.

Is this manner of testing possible with Gabbi today? If so, is Gabbi also capable of separating concerns of setup, testing and teardown such that (for example) object "GET" is "tested" separately from "POST"?

Thanks!

[cdent]

Gabbi provides a poll attribute in a test that makes it possible to do something like POST a job and then GET on the value of the returned location header until you get the expected response or time or number of allowed loops has run out. There's some information on that in the poll section of this table. Also there are some examples in gabbi's tests of itself, see, for example: https://github.com/cdent/gabbi/blob/master/gabbi/tests/gabbits_intercept/poll.yaml

In your specific example jsonpath could be used to inspect the returned json to get the id of the job and create the URL (using a $RESPONSE substitution) to poll the job.

Isolation, setup and teardown in gabbi is per YAML file, not per test, because each YAML file is expected to be a sequence of connected calls that describe an interaction with an API:

• GET a /collection, see empty results
• POST to /collection, create a new item
• GET /collection/{id}, prove it is there
• PUT /collection/{id}, update it
• DELETE /collection/{id}, delete it
• GET /collection/{id}, prove it is not there

That sort of thing.

There are fixtures that operate per YAML file which can be used to establish things like a persistence layer and any resources that need to exist.

If you need to test GET of something, in isolation, then you would want a YAML file with one test (that does the get) associated with a fixture that creates (and destroys) the expected object. When I'm doing my own testing, I don't usually do it this way, I'd instead do a sequence like I've described above.

If this isn't enough help to get you rolling, let me know, either here or in #gabbi on freenode IRC. If you've got some existing code we can look at together we can probably come up with a reasonable way to test things or I can at least point you at some more examples.

[ghost]

This is great info. I think I have enough to get started, but I will definitely reach out if I run into any more questions. Thanks!

[ghost]

Awesome. So polling is working.

• Collecting job ID from the POST JSON
• Doing a GET on the job status endpoint (no helpful location headers on this API :( ) until JSON says job complete
• Still working on the confirmation GET on the newly created object, but that should be no problem.

in the meantime, I can see how the following block will get repeated in my code a lot because this API has so many async calls.

  - name: GET job V1
    GET: /services/reqId/$RESPONSE['$.RequisitionSubmit.id']
    status: 200
    poll:
      count: 15
      delay: 2
    response_json_paths:
      $.requisition.percentageCompleted: 100
      $.requisition.status: "Closed"

Is there any manner of short-handing such a call? Maybe I can use a fixture somehow to make this into an easily callable "function"?

Another thing I want to look at is string generation. Creating a user with a random base string would be good. Eventually, I want to wade into doing that kinda stuff with Hypothesis + Gabbi, but I'm not there just yet. Plus, with a async API, it's suite run-to-completion time will suffer greatly given 200 rounds for each call.

[cdent]

YAML itself might be able to help you with that. I've known for a while that something like the following ought to be possible, but not tried it. Basically YAML has anchors that you can use to repeat a node. In some simple testing (with gabbi-run https://burningchrome.com < /tmp/t.yaml) the following appears to work:

defaults:
    verbose: headers

sample_test: &repeated
    name: sample test
    GET: /
    response_headers:
        server: /Apache/2.4.10/

tests:

- <<: *repeated
  name: first test

- <<: *repeated
  name: other test
  GET: /~cdent/index.cgi

- name: own thing
  GET: /fdafdasfsafassa
  status: 404

The YAML processor will resolve the & and * referencing before gabbi ever sees the data, so things like defaults and substitutions should work as expected. When referring back to an anchor you can override any of the keys. I suspect things get complicated with those keys that are nested dicts. You'll still need to set name for every test as that has to be unique (at least in a unittest-based runner).

Thanks for prompting this. Let us know if that works for you.

Another thing to keep in mind is that you can programmatically create tests, or preprocess the yaml before handing to test_suite_from_dict. If you look at the recent changes on the master branch, the data flow may be a bit more clear (modules have been split out). So if you have some way to create the dict, you can cook and slice how you like, and then get back tests. There are probably ways that gabbi could change to make that easier.

On string generation, there's #12 but no resolution yet for how to do it directly in the YAML. If you've got some ideas, most welcome.

I'm just on my way out the door for the day, but will look into this a bit more closely later.

[ghost]

Great work! The YAML anchor method works well for me! Much thanks!

I propose an new (and already working) design pattern for leveraging anchors: A top-level section called "anchors". Looks cleaner without random blocks hanging about.

Also worth noting that I could poll on the get where thing should show up, which means I wouldn't have to look at the job endpoint. For this present API though I'll stick with this for now.

defaults:
  request_headers:
    Accept: application/json

anchors:

    track_job: &track_job
        GET: /job/$RESPONSE['$.JobSubmit.id']
        status: 200
        poll:
          count: 4
          delay: 15
        response_json_paths:
          $.job.percentageCompleted: 100
          $.job.status: "Closed"

tests:

  - name: create thing blue
    POST: /thing
    status: 202
    data:
      color: blue
      description: This is a description.

  - name: track create thing blue
    <<: *track_job

  - name: create thing red
    POST: /thing
    status: 202
    data:
      color: red
      description: This is a description.

  - name: track create thing
    <<: *track_job

[cdent]

Just for reference, I use this anchor handling a lot lately, using the "put it at the top" pattern mentioned.

It's also useful for shortcuts to $HISTORY substitutions. See: https://github.com/cdent/gabbi-tempest/blob/master/samples/multi.yaml