Page 25 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
Automated Generation of Test Oracles for RESTful APIs
Juan C. Alonso SCORE Lab, I3US Institute, University of Seville, Seville, Spain
Email: javalenzuela@us.es
Sergio Segura SCORE Lab, I3US Institute, University of Seville, Seville, Spain
Email: sergiosegura@us.es
Antonio Ruiz-Corte´s SCORE Lab, I3US Institute, University of Seville, Seville, Spain
Email: aruiz@us.es
Abstract—Web APIs following the REST architectural style (frequently known as RESTful APIs) have become the de-facto standard for web integration. In recent years, a large number of tools for automatically testing this type of API have emerged. However, all these tools fall short when it comes to their fault- detection capabilities, which are limited to unexpected failures (i.e., 5XX code responses) and disconformities with the API speci- fication. This article describes our ongoing work for automatically generating test oracles for RESTful APIs. Specifically, we propose to automatically infer likely invariants from sets of inputs and outputs that can later be used as test oracles by leveraging an extended version of Daikon, a tool that detects likely invariants by processing a program execution. A preliminary evaluation with 8 operations from 6 industrial APIs shows the effectiveness of our approach for automatically generating test oracles, detecting reproducible faults in two of them (GitHub and OMDb).
I. INTRODUCTION
RESTful APIs are the cornerstone of software integration, allowing systems to interact with each other over the network by exchanging messages in JSON or XML format through the HTTP protocol. Web services usually provide RESTful APIs for different clients to access their functionality. This is one of the main reasons why testing these systems is vital, since a fault in an API could compromise hundreds or thousands of other systems consuming it.
RESTful APIs are commonly described using languages such as the OpenAPI Specification (OAS) [1], which provides a machine-readable description of the API functionality that is used by different tools to automatically generate test cases [2]. All these tools are limited by the types of errors they can detect, such as disconformities with the OAS specification and server errors.
An apparently successful response returned by an API (i.e., a 2XX status code that conforms to the API specification) does not guarantee that the system is fulfilling its intended functionality of behavior. This is a classical problem in Soft- ware testing known as the oracle problem [3], that can be expressed as the challenge of, given an input for a system, distinguish the expected behavior from a potentially incorrect one. For example, when performing a search for songs in Spotify establishing a maximum number of results to return (limit parameter), the size of property of the response body whose value is an array of songs (items) should be less or equal than the value set for this parameter (input.limit
>= size(return.items)). Although it is possible to generate these oracles manually, this is a time-consuming task that requires domain knowledge.
An invariant is a property that holds at a certain point or points of the execution of a program, such as its input parameters and responses in the context of black box testing of RESTful APIs. Currently, there are several systems available for the automated detection of likely invariants, with Daikon [4] being one of the most popular. Daikon detects likely invariants by processing an instrumented version of a program, this instrumentation process is performed by an instrumenter, a software that converts a program structure into a format that can be analyzed by Daikon. There are several Daikon instrumenters available, most of them are designed for specific programming languages (i.e., they detect invariants on white box contexts) such as Java or Perl.
In this article, we propose an approach for automatically generating test oracles for RESTful APIs from a set of valid API requests by modelling the generation of oracles as a prob- lem of extracting likely invariants. For this purpose, we created an instrumenter that takes as input an OAS specification and a set of test cases and returns a set of files that can be used as inputs for Daikon (i.e., it works on a black box context). A preliminary evaluation with a set of 8 operations from 6 commercial APIs shows the potential of this approach for automatically generating hundreds of test oracles, detecting real errors in two of the systems under test, namely GitHub and OMDb.
II. AUTOMATED GENERATION OF TEST ORACLES FOR RESTFUL APIS
This section describes our approach for the automated generation of test oracles for RESTful APIs from an OAS specification and a set of test cases (i.e., values of the input parameters and the response body). Our instrumenter receives these two files as input, returning a decls file that describes the structure of the program (in our case, the input and the possible outputs), and a dtrace file specifying, for each test composing the test suite, the values assigned to each part of the structure defined by the decls file.
Specifically, we have modified Daikon by adding a total of 22 new invariants and suppressing 25 invariants that do would
13