Page 28 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
O2 Characterize the resources employed during the E2E testing by considering different attributes and how the resources are used by the test cases.
O3 Improve the E2E test suite execution by considering: (1) relationships between resources, (2) how test cases access the resources, and (3) the resource attributes.
O4 Select a cost-effective infrastructure to execute the test suite in the Cloud, considering both the cost of the resources employed in E2E and oversubscription.
O5 Experimentation and validation of the approach, execut- ing test suites of real-world scenarios.
IV. PROPOSED METHODOLOGY
The methodologies used to achieve the prior objectives are the following:
1) Review the state of the art and literature: we intend to review the state-of-the-art works such as test suite minimization, reduction, prioritization, cloud economics, or cost models in the cloud, among others.
2) Research-action:wearecollaboratingwiththeInstitute of Information Science and Technologies ”Alessandro Faedo” which provides us with real-world problems faced in European projects such as Elastest [2]. Specif- ically, we intend to validate the research results with several test suites in the industrial field.
3) Incremental development: we develop the support tool based on agile methodologies [3] . Each new feature added is a research result according to its relevance
V. FIRST RESULTS
To accomplish the second objective (O2), we carried out a characterization of the resources employed in the E2E testing, consisting of several attributes that represent the different resource features and their relationship with the test cases (e.g., available resources, access mode performed by test cases, or if the resource is shared). We propose a four- phase orchestration process called RETORCH (acronym of Resource aware End-to-end Test ORCHestation) to accom- plish the first three objectives (O1, O2, and O3). RETORCH was published at the QUATIC19 conference [4] and extended to the Software Quality Journal [5]. The work was also presented in the ACM Research competition in the ICSE20 [6] and won an award in the 5th edition of the SISTEDES- Everis Awards [7]. RETORCH uses the information from the characterization (resource identification) to generate sets of test cases with compatible usage of resources (grouping). The sets are split into subsets and scheduled sequentially or in parallel to reduce the execution time and avoid unnec- essary redeployments (scheduling) and are finally deployed into a continuous integration environment (orchestration). The approach was validated (O5) with a real-world example of an educational application called FullTeaching [8], achieving reductions in the execution time (61% less than the non- orchestrated test suite) and fewer resources employed (due to resource sharing between test cases). To accomplish the first and fourth objectives (O1 and O4), we develop a cost
model focused on the cost of the resources employed in E2E test suites. The model estimates the cost invested in executing the test suite and the oversubscription cost. These two costs with the infrastructure cost (contracted) support the tester’s decision-making to choose the most cost-effective infrastructure among those available in the Cloud. This model has also been submitted to the JISBD22 [9] conference.
VI. CONCLUSIONS AND FUTURE WORK
The current thesis addresses the upcoming issue of opti- mizing the execution of E2E test suites in the Cloud. Its first results have proven that using a smart characterization of the resources employed on end-to-end test suites, grouping the test cases according to the resource usage, scheduling, and orchestrating them, savings in terms of resources and time can be achieved. The cost model that considers both the cost of executing the test suite and the cost incurred in oversubscription is a work in progress and we expect that it will help in selecting the most cost-effective Cloud infrastructure and obtain a better execution cost. As future work, we want to validate RETORCH in more real-world end- to-end test suites. We are exploring how a smarter cost model could improve the efficient E2E test execution. Specifically, we aim to integrate the cost model in an infrastructure advisor engine that analyses all three costs. The purpose of the advisor is to make suggestions about Cloud infrastructure changes that lead to a more cost-effective test suite execution (e.g., new infrastructures available, or changes in those selected that reduce one of the costs).
ACKOWLEDGEMENTS
The authors would like to thank Dra. Antonia Bertolino (ISTI-CNR, Pisa Italia) for her contributions to the line of research. This work was supported in part by the Spanish Min- istry of Economy and digital transformation under TestBUS (PID2019-105455GB-C32) and SEBASENet 2.0 (RED2018- 102472-T)”.
REFERENCES
[1] K.Inc¸ki,I.Ari,andH.So¨zer,“Asurveyofsoftwaretestinginthecloud,”
Proceedings of the 2012 IEEE 6th International Conference on Software
Security and Reliability Companion, SERE-C 2012, pp. 18–23, 2012. [2] URJC, FOKUS, TUB, INSTI-CNR, IMDEA, and ATOS, “Elastest.”
[Online]. Available: https://elastest.eu/
[3] P. Deemer, G. Benefield, C. Larman, and B. Vodde, The Scrum Primer:
A Lightweight Guide to the Theory and Practice of Scrum. InfoQ,
2012. [Online]. Available: www.odd-e.com
[4] C. Augusto, J. Mora´n, A. Bertolino, C. de la Riva, and J. Tuya, “Re-
torch: Resource-aware end-to-end test orchestration,” Communications in
Computer and Information Science, vol. 1010, pp. 297–310, 9 2019.
[5] ——, “Retorch: an approach for resource-aware orchestration of end-to-
end test cases,” Software Quality Journal, 2020.
[6] C. Augusto, “Efficient test execution in end to end testing,” in Proceedings
- 2020 ACM/IEEE 42nd International Conference on Software Engineer-
ing: Companion, ICSE-Companion 2020, 2020, pp. 152–154.
[7] C. Augusto and C. de la Riva, “Optimizacio´n de recursos en pruebas de
sistema,” 5th edition of the SISTEDES-Everis Award, 2021.
[8] URJC and P. F. Pe´rez, “Fullteaching - elastest repository,” 2019.
[Online]. Available: https://github.com/elastest/full-teaching
[9] C. Augusto, J. Mora´n, C. de la Riva, and J. Tuya, “Modelo de costes para el despliegue de pruebas e2e en entornos cloud,” 2022, unpublished
Manuscript.
16