Page 27 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
Resource Optimization in End-to-End Testing
Cristian Augusto Department of Computing University of Oviedo Gijo´n, Spain augustocristian@uniovi.es
Jesus Mora´n Department of Computing University of Oviedo Gijo´n, Spain moranjesus@uniovi.es
Claudio de la Riva Department of Computing University of Oviedo Gijo´n, Spain claudio@uniovi.es
Javier Tuya Department of Computing University of Oviedo Gijo´n, Spain tuya@uniovi.es
Abstract—End-to-end (E2E) test suite execution is expensive due to the number of complex resources required. When E2E test suites are executed frequently into a continuous integration system, the total amount of resources required may be pro- hibitive, moreover when the tests are run in the Cloud with different billing strategies. Traditional techniques to optimize the test suites, such as test suite reduction, minimization, or prioritization, are limited in E2E due to the fact that reordering or selecting a subset of test cases also requires deploying the same expensive system. The current Ph.D. thesis aims to achieve an efficient E2E test execution for large systems in the Cloud. This is done through a smart characterization of the resources required by the test cases, grouping and scheduling them according to their resource usage to avoid unnecessary redeployments and reduce execution time, and finally, executing them into a combination of Cloud infrastructure (i.e., containers, virtual machines, and services) to optimize the costs employed in executing the test suite. Based on the scheduled test cases, we elaborate a cost model for selecting the most cost-effective infrastructure of those available in the Cloud, considering both the cost of the resources required by the test cases and the oversubscription cost (cost incurred in infrastructure contracted and not used during the test suite execution).
I. INTRODUCTION
The execution of End-to-end (E2E) test suites is costly in resource terms due to the expensive and complex systems required. When the test suites are put into a continuous in- tegration environment executed with every repository change, the huge number of resources required can be prohibitive for some companies, particularly when they are run in the Cloud with pay-per-usage billing. The traditional approaches used for optimizing test suites, such as test prioritization, minimization, or reduction, are limited in E2E because selecting a subset or executing the test cases in a fixed order to preserve the effectiveness, may require fewer but also expensive resources. Moving E2E testing to the Cloud brings the opportunity for a better cost but is challenging because of the number of different infrastructures with different costs available to deploy the same resource. Often, the selected infrastructure combination does not match the test suite requirements, which leads to oversubscription. Oversubscription is the difference between the infrastructure contracted and used and causes that part of the cost is not invested in testing tasks, incurring in an extra cost budget [1]. In this thesis, we aim to achieve efficient E2E test suite execution of large systems in the Cloud. For this purpose, we focus on the resources: the physical, logical, or computational entities that are required
by the test cases during their execution. We elaborate an orchestration process that, through the information provided by the characterization, schedules the test cases to reduce time and unnecessary resource redeployments and execute the test suite into a combination of Cloud infrastructure. To choose the most cost-effective combination we elaborate a model which considers the cost of the (contracted) infrastructure, as well as the cost of the resources required by the test cases and the cost of oversubscription during the execution.
II. RESEARCH HYPOTHESIS
Working in the Cloud has become complex and heteroge- neous and with it, selecting the most suitable alternative to carrying out the E2E testing, has become increasingly more challenging. The organization and management of the E2E test cases could derive from different execution times and, depending on how they are deployed in the Cloud, the costs might change. To determine the specific objectives of the research we consider the following four hypotheses:
H1: The execution of End-to-End (E2E) test suites can re- quire large amounts of physical-logical resources, limiting the effectiveness of state-of-the-art techniques such as test case minimization, reduction, or parallelization.
H2: The scheduling of E2E test suites can be optimized by analyzing the dependencies between test cases and looking for concurrency in the use of resources (e.g., web servers, databases, hardware devices, and files, among others).
H3: A smart resource characterization, complemented by grouping and scheduling according to the resources used by the test cases, could save time and avoid unnecessary redeployments in the execution of the test suite.
H4: Considering the cost of the resources and oversubscription during the deployment of the test suite in the Cloud could help to select the most cost-effective combination among the available infrastructure.
III. OBJECTIVES OF THE WORK
The general objective of this thesis is to optimize end-to- end test suite execution. This objective is particularized in the following sub-objectives:
O1 Analyze the challenges and issues arising from the execution of the E2E Test Suites on large software developments in the Cloud.
15