RESTest is a framework for automated black-box testing of RESTful web APIs. It follows a model-based approach, where test cases are automatically derived from the OpenAPI Specification (OAS) of the API under test. No access to the source code is required, which makes it possible to test APIs written in any programming language, running in local or remote servers.

In this page you can find a brief description of how RESTest works and an illustrating example. If you want to read the full documentation, please visit the Wiki.

The figure below shows how RESTest works:

1. Test model generation: RESTest takes as input the OAS specification of the API under test, considered the system model. A test model is automatically generated from the system model including test-specific configuration data. The default test model can be manually enriched with fine-grained configuration details such as test data generation settings.

2. Analysis of dependencies: The OAS specification of the API can optionally describe inter-parameter dependencies using the IDL4OAS extension (see examples here and here). If so, inter-parameter dependencies will be automatically analyzed and leveraged for the generation of test cases.

3. Abstract test case generation: The system and the test models drive the generation of abstract test cases following user-defined test case generation strategies such as random testing. In parallel, inter-parameter dependencies, if any, are fed into the tool IDLReasoner, providing support for their automated analysis during test case generation, for instance, to check whether an API call satisfies all the inter-parameter dependencies defined in the specification.

4. Test case generation: The abstract test cases are instantiated into a specific programming language or testing framework using a test writer. RESTest currently supports the generation of REST Assured test cases.

5. Test case execution: The test cases are executed and a set of reports and statistics are generated. These data are machine-readable, therefore test generation algorithms can react to them and generate more sophisticated test cases (e.g., with search-based approaches).

To get started with RESTest, you can try it yourself with some API, for example, Bikewise. First, download the code:

git clone https://.com/isa-group/RESTest.git

1. Get the OAS specification of the API under test. For Bikewise, it is available at the following path: src/test/resources/Bikewise/swagger.yaml.

2. Generate the test configuration file. From the OAS spec, we can automatically generate the test configuration file. To do so, run the CreateTestConf class, located under the es.us.isa.restest.main package. The test configuration file will be generated in the location src/test/Bikewise/testConf.yaml.

3. (Optional) Modify the test configuration file to tailor your needs. For example, you can remove some operations you are not interested to test. For more info, visit the Wiki.

4. Configure RESTest execution. To set things like number of test cases to generate, testing technique, etc., you need to create a RESTest configuration file. You can find the RESTest configuration file for the Bikewise API at src/main/resources/ExperimentsSetup/bikewise.properties. With this configuration, a total of 40 nominal test cases will be randomly generated, and the test outputs and reports will be stored under the folders target/<type_of_data>/bikewise_example:

numtestcases=10
oaispecpath=src/test/resources/Bikewise/swagger.yaml
confpath=src/test/resources/Bikewise/testConf.yaml
targetdirjava=src/generation/java/bikewise_example
packagename=bikewise_example
experimentname=bikewise_example
testclassname=BikewiseTest
enableinputcoverage=true
enableoutputcoverage=true
enablecsvstats=true
ignoredependencies=true
numtotaltestcases=40
delay=-1
faultyratio=0

# CBT only:
faultydependencyratio=0
reloadinputdataevery=10
inputdatamaxvalues=10

5. Run RESTest. Edit the following line of IterativeExample to set the path to the properties file. Then, run the IterativeExample class, located under the es.us.isa.restest.main package.

setEvaluationParameters("src/main/resources/ExperimentsSetup/bikewise.properties");

RESTest generates REST Assured test cases like the following one:

@Test
public void test_1jylmdhlbau76_GETversionincidentsformat() {
	String testResultId = "test_1jylmdhlbau76_GETversionincidentsformat";

	NominalOrFaultyTestCaseFilter nominalOrFaultyTestCaseFilter = new NominalOrFaultyTestCaseFilter(false, false, "none");

	try {
  	Response response = RestAssured
		.given()
			.log().all()
			.queryParam("incident_type", "unconfirmed")
			.queryParam("proximity", "whine")
			.queryParam("occurred_after", "36")
			.queryParam("query", "camper")
			.queryParam("page", "18")
			.filter(new CSVFilter(testResultId, APIName))
			.filter(allureFilter)
			.filter(statusCode5XXFilter)
			.filter(nominalOrFaultyTestCaseFilter)
			.filter(validationFilter)
		.when()
			.get("/v2/incidents");

		response.then().log().all();
		System.out.println("Test passed.");
	} catch (RuntimeException ex) {
		System.err.println(ex.getMessage());
		fail(ex.getMessage());
	}	
}

This test case makes a GET request to the endpoint /v2/incidents with several query parameters. Then it asserts that:

• The status code is not 500 or higher (server error).
• The status code is in the range 2XX if the request is valid or 4XX if the request is faulty.
• The response conforms to the OAS specification of the API.

Finally, test failures are collected and they can be easily spotted and analyzed in a user-friendly GUI, built with Allure. To do so, open the file target/allure-reports/bikewise_example/index.html in your browser: