Evolutive Integration Testing of HTTP APIs with OpenAPI


Automated tests are vital to prevent regressions and find problems early on, but the value they provide is directly proportional to the tester’s skills. Usually, automated test suites end up covering only a subset of the happy path.

Automating the process of writing automated tests is the premise behind property-based and generative testing. The idea is that testers write logical statements about the system, and the test framework, such as JSVerify, will generate test cases with semi-random inputs to find contradictions. I found that this technique works well if what you are testing can be elegantly represented as a logical invariant of an pure-ish algorithm, but not for stateful higher-level integration test cases on business user-facing applications.

Leaving some outliers aside, users try to use a piece of software for its intended purpose. Therefore, on my own experience, most impactful bugs on business applications are due to untested subtle variations from the happy path. For this reason, a core part of my testing process is to write test cases for the happy path and them come up with many variations.

Can this process be automated?

We can’t attempt to perform non-trivial automated mutation of test cases without making a fair amount of assumptions over the software under test and the structure of the test cases themselves.

In this article, I’ll consider HTTP APIs, who share well understood semantics thanks to the HTTP protocol, along with OpenAPI, a specifiation for a YAML/JSON language to describe APIs that allows us to understand in more depth how the software uses the HTTP protocol to do its job. You check some of their examples here.

Finally, integration test cases are usually written in a full-featured programming language, which makes them hard to analyze and change without breaking their semantics, specially if flow control operations are involved. For this reason, I’ll consider a simple DSL tuned to write HTTP integration test cases.

Mutating Test Cases

Here is a non-exhaustive list of transformations that can be applied automatically:

Consider the following DSL test case that creates a device, fetches it back, modifies the name, and fetches it back again, and deletes it. We also have a spec that defines that there are two instances of the server running at localhost:8000 and localhost:8001:

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}

Here is one possible set of mutations that could result from a one-pass of the above algorithms:

400 POST localhost:8000/api/v1/device color="red"
201 POST localhost:8000/api/v1/device name="Device Foo"
405 PUT localhost:8000/api/v1/device name="Device Foo" color="red"
405 DELETE localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
405 POST localhost:8000/api/v1/device/{Id}
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
405 POST localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
405 POST localhost:8000/api/v1/device/{Id}
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8001/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8001/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 DELETE localhost:8000/api/v1/device/{Id}
404 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}
{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8001/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8001/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8001/api/v1/device/{Id}

Even with some rules omitted, we expanded a single medium-sized integration test case into 12 integration test cases. There is also the question of whether we can re-mutate the mutated results, to get even more test cases.

Overall, this approach is not about finding a way for testers to write fewer tests, but a way for those same tests to have much more impact, and unveil the most common bugs as early as possible.