Page 26 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
TABLE I
ORACLES GENERATED. TP=TRUE POSITIVES, FP=FALSE POSITIVES
API
Amadeus Hotel GitHub
OMDb
OMDb
Spotify Spotify Yelp YouTube
Operation
# oracles
Precision (%)
60.2 70.1 70 100 100 76.5 32 59.1
65.1
TP FP
Inconsistency/Bug
Total 501
not reveal any relevant information in our current context, resulting in the generation of redundant information or false positives. These new invariants are based on a previous publi- cation of the authors in which an evaluation was performed on a dataset of 48 real world APIs [5]. This version of Daikon supports a total of 140 invariants that can be classified into one of the following categories:
• Arithmetic relationships. They specify comparisons be- tween the values of numeric properties. For example, when searching for albums on Spotify, the track num- ber of a song must be greater than or equal to 1 (return.track_number >= 1).
• Array properties: These invariants indicate that an array has certain characteristics. For example, when searching for hotels by id in Amadeus, the id of each hotel returned must be contained in the array of ids used as the input parameter (return.hotelId in input.hotelIds[]). Also, the size of the ar- ray property containing the results (data) must be less than or equal to the size of the list of ids used as a parameter (size(input.hotelIds[]) >= size(return.data[])).
• Specific values: They specify that a property al- ways has a fixed value or set of values. For ex- ample, in the GitHub API, a repository can be public or private (return.visibility one of "private", "public").
• Specific formatting: These invariants indicate that a string field always follows a specific format, such as URLs, dates or emails. For example, the OMDb’s text field “Poster” must always be of type URL (return.Poster is Url).
III. PRELIMINARY EVALUATION
For our evaluation, we selected a total of 8 operations from 6 industrial RESTful APIs. For each one of these operations, we automatically generated 50 valid API requests (2XX codes) using the black box framework for automated testing of RESTful APIs RESTest [6]. These requests were used as inputs for our approach, resulting in a set of likely invariants
(oracles) for the operation.
These invariants are manually classified as true positives,
false positives or as invariants that reveal the existence of a bug or inconsistency. The results of our evaluation (Table I) show
the potential of our approach for automatically generating oracles for complex real-world systems, achieving a total precision of 65.1% and detecting errors and inconsistencies in the documentation and implementation of systems with millions of users such as GitHub or OMDb.
In the GitHub API, our proposal automatically detected that one of the fields in the response, template_repository, was not present in any of the repositories returned, even in cases in which they had a template repository. This bug has been confirmed by the API developers, who have created an internal issue to update the documentation.
According to their official documentation, “By Search” op- eration of the OMDb API allows to search for titles filtered by type (“movie”, “series” or “episode”). However, our proposal detected not only that the API never returns results of type “episode”, but that it returns results of a fourth type not specified in the documentation (“game”), which can also be used as the value of the parameter used to filter the search.
IV. CONCLUSION
Our future work includes using the generated oracles for the automatic creation of assertions to evaluate the validity of API responses, as well as developing a method to prioritise among the generated oracles and detect false positives.
REFERENCES
[1] “OpenAPI Specification,” https://www.openapis.org, accessed May 2022. [2] S. S. Myeongsoo Kim, Qi Xin and A. Orso, “Automated Test Generation for REST APIs: No Time to Rest Yet,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis,
2022.
[3] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle
problem in software testing: A survey,” IEEE transactions on software
engineering, vol. 41, no. 5, pp. 507–525, 2014.
[4] M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant,
C. Pacheco, M. S. Tschantz, and C. Xiao, “The daikon system for dynamic detection of likely invariants,” Science of Computer Programming, vol. 69, no. 1, pp. 35–45, 2007, special issue on Experimental Software and Toolkits. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S016764230700161X
[5] J. C. Alonso, A. Martin-Lopez, S. Segura, J. M. Garcia, and A. Ruiz- Cortes, “ARTE: Automated Generation of Realistic Test Inputs for Web APIs,” IEEE Transactions on Software Engineering, 2022.
[6] A. Martin-Lopez, S. Segura, and A. Ruiz-Corte´s, “RESTest: Black- Box Constraint-Based Testing of RESTful Web APIs,” in International Conference on Service-Oriented Computing, 2020, pp. 459–475.
Find hotel offers 93 List organization repositories 106 By ID or Title 20 By Search 7 Create Playlist 28 Get Album tracks 51 Search businesses 25 List videos 171
56 37 0 68 29 9 14 6 0 5 0 2 28 0 0 39 12 0 8 17 0 101 70 0
319 171 11
14