# Evolutive Integration Testing of HTTP APIs with OpenAPI

Automated tests are vital to prevent regressions and find problems early on, but the value they provide is directly proportional to the tester’s skills. Usually, automated test suites end up covering only a subset of the happy path.

Automating the process of writing automated tests is the premise behind property-based and generative testing. The idea is that testers write logical statements about the system, and the test framework, such as JSVerify, will generate test cases with semi-random inputs to find contradictions. I found that this technique works well if what you are testing can be elegantly represented as a logical invariant of an pure-ish algorithm, but not for stateful higher-level integration test cases on business user-facing applications.

Leaving some outliers aside, users try to use a piece of software for its intended purpose. Therefore, on my own experience, most impactful bugs on business applications are due to untested subtle variations from the happy path. For this reason, a core part of my testing process is to write test cases for the happy path and them come up with many variations.

## Can this process be automated?

We can’t attempt to perform non-trivial automated mutation of test cases without making a fair amount of assumptions over the software under test and the structure of the test cases themselves.

In this article, I’ll consider HTTP APIs, who share well understood semantics thanks to the HTTP protocol, along with OpenAPI, a specifiation for a YAML/JSON language to describe APIs that allows us to understand in more depth how the software uses the HTTP protocol to do its job. You check some of their examples here.

Finally, integration test cases are usually written in a full-featured programming language, which makes them hard to analyze and change without breaking their semantics, specially if flow control operations are involved. For this reason, I’ll consider a simple DSL tuned to write HTTP integration test cases.

## Mutating Test Cases

Here is a non-exhaustive list of transformations that can be applied automatically:

• From a test case, pick any HTTP request that includes a required property in the body (check with the OpenAPI spec), omit it, assert that the status code is 400 Bad Request, and discard the remaining of the test case

• From a test case, pick any HTTP request that includes a required header (check with the OpenAPI spec), omit it, assert that the status code is 4xx, and discard the remaining of the test case

• From a test case, pick any HTTP request that includes an optional property in the body (check with the OpenAPI spec), omit it, assert that the status code is the same as when the optional property was there, ignore the assertions on the response body, and discard the remaining of the test case

• From a test case, pick any HTTP request and an alternated undefined HTTP method for that same path (except OPTIONS and TRACE) (check the OpenAPI spec), use the undefined HTTP method, assert that the status code is 405 Method Not Allowed, and discard the remaining of the test case

• If the OpenAPI spec defines more than one server, pick a test case that performs more than one HTTP request, randomly assign servers to each HTTP request in the test case, and leave the assertions intact

• From a test case, pick a PATCH or PUT HTTP request, duplicate those requests X times in a row, and leave the assertions intact

• From a test case, pick a PATCH request, remove all parameters, assert that the status code is 400 Bad Request, and discard the remaining of the test case

• From a test case, pick a DELETE HTTP request, duplicate those requests X times in a row, assert that all DELETE requests other than the first ones result in 404 Not Found, and leave the other assertions intact

Consider the following DSL test case that creates a device, fetches it back, modifies the name, and fetches it back again, and deletes it. We also have a spec that defines that there are two instances of the server running at localhost:8000 and localhost:8001:

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}


Here is one possible set of mutations that could result from a one-pass of the above algorithms:

400 POST localhost:8000/api/v1/device color="red"

201 POST localhost:8000/api/v1/device name="Device Foo"

405 PUT localhost:8000/api/v1/device name="Device Foo" color="red"

405 DELETE localhost:8000/api/v1/device name="Device Foo" color="red"

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
405 POST localhost:8000/api/v1/device/{Id}

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
405 POST localhost:8000/api/v1/device/{Id} name="Device Bar"

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
405 POST localhost:8000/api/v1/device/{Id}

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8001/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8001/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8000/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 DELETE localhost:8000/api/v1/device/{Id}
404 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8000/api/v1/device/{Id}

{ Id, _, _ } := 201 POST localhost:8000/api/v1/device name="Device Foo" color="red"
{ Id1, Name1, Color1 } := 200 GET localhost:8001/api/v1/device/{Id}
ASSERT Id1 = Id
ASSERT Name1 = "Device Foo"
ASSERT Color1 = "red"
200 PATCH localhost:8000/api/v1/device/{Id} name="Device Bar"
{ Id2, Name2, Color2 } := 200 GET localhost:8001/api/v1/device/{Id}
ASSERT Id2 = Id
ASSERT Name2 = "Device Bar"
ASSERT Color2 = "red"
200 DELETE localhost:8000/api/v1/device/{Id}
404 GET localhost:8001/api/v1/device/{Id}


Even with some rules omitted, we expanded a single medium-sized integration test case into 12 integration test cases. There is also the question of whether we can re-mutate the mutated results, to get even more test cases.

Overall, this approach is not about finding a way for testers to write fewer tests, but a way for those same tests to have much more impact, and unveil the most common bugs as early as possible.