Page 48 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
Powerful search-based test generation tools like EvoSuite [8] are already able to generate whole test suites from scratch. To apply them to just-in-time test generation, one would need to investigate how existing test cases could be used effectively as an initial population and how well the limited search space can improve the runtime/generation quality towards interactive performance. Test amplification [9] generates new test cases by mutating the input stage of the test case and generating assertions matching the new test behavior. In a previous study, we investigated the interaction of software developers with automated test amplification [1]. We saw that it is important to give the developer control over the interaction, provide them with the information necessary to judge the test cases and effectively communicate the impact that the generated test case will have on the quality of their test suite. Taking into account the results of further user studies of test generation tools [10]– [12], we conjecture that a strong focus on the design of the user interaction is crucial for our just-in-time test generation approach to be successful.
Machine learning approaches are gaining popularity also in the area of automatic test generation, e.g. to generate assertion statements [13], [14]. Recently, neural code completion tools such as GitHub Copilot1 are able to propose fully fledged im- plementations when triggered with a natural language method name. Nonetheless, it remains to be investigated how effective they are in generating useful test cases and which information needs to be encoded in the trigger to steer the generation towards the intended code under test.
III. DIVIDE AND CONQUER: STEPS TO TACKLE
On the way to fully-fledged just-in-time test generation we see several challenges to be addressed. These could be the basis for our discussion at the SMILESENG summer school, together with the following questions:
• Cutting out a test-worthy condition / scenario as a target for the test generation. Approaches could be detecting coher- ent edits made by the developer, and learning from single- concern commits or the coverage of existing test cases. Together with the following test generation, this should be a deterministic and transparent process to let the developer build trust and understanding in the capabilities of our tool.
• Detecting the right moment to contact the developer as well as giving them control to start and feed the tool themselves.
• Rapid on-device generation to enable interactive coop- eration between developer and tool. To speed up the test generation, we propose to leverage incrementality: Building upon existing test cases by modifying them only slightly. In
1 https://copilot.github.com/
addition, we look at incremental compilation and building to speed up a run of a test generation tool, as well as explore ways to avoid the expensive executions of test cases to measure adequacy metrics such as mutation score or structural coverage.
IV. AN OUTLOOK INTO THE FUTURE
The possible applications of a just-in-time test generation tool and its components will be much wider than just adding test cases for new conditions in the code. We could propose immediate updates to no-longer-passing test cases before the developer reruns their tests after a change. We could determine no longer needed test cases after large code cleanups and propose suiting re-locations of test code after refactorings. In- tegrating test generation right within the developer’s workflow lets them become familiar with the advantages test generation can offer and lets them gain trust in the capabilities of (partially) automated software engineering.
While just-in-time test generation is still many steps away, the development of each its subparts helps us strengthen and better understand the area of automatic test generation and its interaction with software developers. We are excited to present our idea to the SMILESENG Summer School participants and together discuss approaches to tackle it.
REFERENCES
[1] C. Brandt and A. Zaidman, “Developer-centric test amplification,” Empir. Softw. Eng., vol. 27, no. 4, 2022.
[2] M. Beller et al., “Developer testing in the IDE: patterns, beliefs, and behavior,” IEEE Trans. Softw. Eng., vol. 45, no. 3, 2019.
[3] A. Santos et al., “A family of experiments on test-driven development,” Empir. Softw. Eng., vol. 26, no. 3, 2021.
[4] R. Bloem et al., “Automating test-suite augmentation,” in 2014 14th Int. Conf. on Quality Softw. IEEE, 2014.
[5] N.Aljawabrahetal.,“Understandingtest-to-codetraceabilitylinks:The need for a better visualizing model,” in Computational Science and Its Applications - 19th Int. Conf. Springer, 2019.
[6] B.V.RompaeyandS.Demeyer,“Establishingtraceabilitylinksbetween unit test cases and units under test,” in 13th European Conf. on Softw. Maintenance and Reengineering. IEEE Computer Society, 2009.
[7] Z. Xu, Y. Kim, M. Kim, G. Rothermel, and M. B. Cohen, “Directed test suite augmentation: techniques and tradeoffs,” in 18th ACM SIGSOFT Int. Symp. on Foundations of Softw. Eng. ACM, 2010.
[8] G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” in m19th ACM SIGSOFT Symp. on the Foundations of Softw. Eng. ACM, 2011.
[9] B.Danglot,O.L.Vera-Pe´rez,B.Baudry,andM.Monperrus,“Automatic test improvement with DSpot: A study with ten mature open-source projects,” Empir. Softw. Eng., vol. 24, no. 4, 2019.
[10] M. M. Almasi et al., “An industrial evaluation of unit test generation: Finding real faults in a financial application,” in 39th Int. Conf. on Softw. Eng.: Softw. Eng. in Practice. IEEE Computer Society, 2017.
[11] S. Panichella et al., “The impact of test case summaries on bug fixing performance: An empirical investigation,” in 38th Int. Conf. on Softw. Eng. ACM, 2016.
[12] J. M. Rojas et al., “Automated unit test generation during software development: A controlled experiment and think-aloud observations,” in 2015 Int. Symp. on Softw. Testing and Analysis. ACM, 2015.
[13] C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert statements for unit test cases,” in 42nd Int. Conf. on Softw. Eng. ACM, 2020.
[14] H. Yu et al., “Automated assertion generation via information retrieval and its integration with deep learning,” in 42th Int. Conf. on Softw. Eng., 2022.
Which further challenges do you see as part of realizing just-in-time test generation?
What approaches should we explore to tackle them?
What chances or caveats do you see with these approaches?
36