Page 47 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
Incremental Just-In-Time Test Generation in Lock-Step with Code Development
Carolin Brandt
Delft University of Technology
c.e.brandt@tudelft.nl
Abstract—State-of-the-art test generation strategies employ ad- vanced analyses of the code under test and powerful optimization algorithms to generate automatic test cases for software systems. As these techniques require a large amount of computational power, they are often limited to generating tests after the code under test is already written. However, today’s broad education about the importance of software testing lets developers strive to create test cases directly with new code they are contributing.
To support these developers, we want to develop an incre- mental just-in-time test generation tool that works in close proximity to the development of the code under test. Whenever the developer creates a new class or functionality, the tool automatically proposes a matching test case. When the developer finishes implementing a new condition, the tool automatically recommends an additional test case that tests the code which was just added. The generated test cases are closely based on the existing test cases in the project with small, incremental changes to test the new lines of code.
To realize such a just-in-time test generation tool we have to tackle many challenges: Detecting the completion of a test-worthy condition, generating a fitting test case in a short time on the developer’s machine, or effectively communicating the value of the new test case to the developer. With the participants of the SMILESENG Summer School we want discuss our new idea, brainstorm on the challenges that this research opens up and identify possible approaches to tackle them.
I. INTRODUCTION
To illustrate our idea of just-in-time test generation, let us introduce this anecdotal use case: Think of Jada, a software engineer in a large software development company. Her team is working on an app for public transport trips and tickets. The transport provider decided to introduce a new promotion: In the summer months of 2022, each monthly pass will cost only nine euros. Today, Jada’s task is to adapt the ticket selection algorithm to propose the new ticket whenever the normal cost of a trip would be more than nine euros. Jada opens the code for the ticket selection component and adds a new condition comparing the trip price to the promotional ticket price. After finishing this new edge case, a notification pops up in the corner of her editor:
“Do you want to add a test that a summer ticket is proposed when it is cheaper than the normal fare?”
As their project policy requests all new code to be fully tested, she is relieved to not have to write a test from scratch. She selects “Inspect Test” and the editor opens on the test class of
This research was funded by the Dutch science foundation NWO through the Vici “TestShift” grant (No. VI.C.182.032)
the ticket selection component. The new test is already added to the source code and a green indicator shows her that the test is passing. Jada reviews the new test case and is pleased that here addition seems to work as she intended: An expensive route—initialized just as in the other tests—is passed to the selection, which then returns the summer ticket instead of the normal ticket. She accepts the new test case and commits it together with her changes. When creating the pull request, she can be confident that all changes are already covered by the test suite. And that mostly automated, thanks to the just-in- time test generator!
This is one use case we envision for our new technology. The just-in-time test generator closely follows the software developer’s actions, identifies testable, test-worthy and fin- ished scenarios, quickly generates matching test cases in the background, and immediately presents these test cases to the developer. The developer inspects the new test cases, modifies them where they see fit and takes them over into their maintained test suite.
Placing the test generation so close to the code development, provides advantages in several known challenges of automatic test generation:
• It narrows the search space by focusing the generation efforts on the just modified code.
• It makes it easier for developers to understand the behavior and coverage impact of the new test case, as they are still in the mental context of the code under test [1].
• By generating the oracle through executing the just modified code, the developer receives immediate feedback on the actual behavior of their code and whether it matches with the behavior they intended.
• This approach widens the application area of automatic test generation to the initial development of code and directly supports developers that aim to write test cases in conjunc- tion with their production code (test-guided [2] or iterative test-last development [3])
II. WHAT WE CAN BUILD UPON: RELATED WORK
The idea of just-in-time test generation is closely related test suite augmentation [4]: adding new test cases to an existing test suite in order to improve its code coverage. Code-to-test traceability approaches [5], [6] can identify test cases that execute code close to the just modified code and with the help of symbolic execution and similar techniques we can modify these base test cases to exercise the new scenario [7].
35