Page 46 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
the testing process. SDCs were not often considered use cases in previous work, although SDCs are already a reality on our streets.
III. METHODOLOGY
We run test scenarios in simulation to have a labeled dataset. A single test case consists of road points defining the whole road the SDC has to follow. The execution of simulation tests allows to classify the tests as “safe” and “unsafe”. Several test data are already publicly available [11].
The primary pipeline is provided by SDC-Scissor. The com- ponents and APIs of SDC-Scissor allow modifying, adjusting, or creating a pipeline for conducting experiments with SDCs in virtual environments.
IV. PRELIMINARY RESULTS
SDC-Scissor is a cost-effective test selector for SDC soft- ware that can predict failing tests with an F1 score of up to 96% and speed up the test execution in simulation by 170%. This selection approach reduces the time spent running tests that likely pass but increases those that likely fail.
The evaluation of the SDC-Prioritizer shows that our ap- proach outperforms a random and greedy baseline prioritizer statistically significantly. I.e., SDC-Prioritizer detects more defects as the baseline with the same execution time. With this tool, the testing process is getting more efficient by revealing more defects in a shorter time.
V. CONCLUSIONS
Our ongoing work shows that we can improve test selection and prioritization significantly. The testing process for SDCs in virtual environments can be more time-efficient. The next step towards our vision is to enable test case minimization for SDCs in virtual environments sot that only relevant parts of a virtual scenario are executed.
REFERENCES
[1] S. Abbaspour Asadollah, R. Inam, and H. Hansson. A survey on testing for cyber physical system. In International Conference on Testing Software and Systems, pages 194–207. IFIP, Springer, 2015.
[2] A. Arrieta, J. A. Agirre, and G. Sagardui. Seeding strategies for multi- objective test case selection: an application on simulation-based testing. In Genetic and Evolutionary Computation Conference, pages 1222– 1231. ACM, 2020.
[3] A. Arrieta, S. Wang, A. Arruabarrena, U. Markiegi, G. Sagardui, and L. Etxeberria. Multi-objective black-box test case selection for cost- effectively testing simulation models. In Genetic and Evolutionary Computation Conference, pages 1411–1418. ACM, 2018.
[4] A. Arrieta, S. Wang, U. Markiegi, A. Arruabarrena, L. Etxeberria, and G. Sagardui. Pareto efficient multi-objective black-box test case selection for simulation-based testing. Information and Software Technology, 114:137–154, 2019.
[5] A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria. Search-based test case selection of cyber-physical system product lines for simulation- based validation. In International Systems and Software Product Line Conference, pages 297–306, 2016.
[6] A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria. Search-based test case prioritization for simulation-based testing of cyber-physical system product lines. Journal of Systems and Software, 149:1–34, 2019.
[7] C. Birchler, N. Ganz, S. Khatiri, A. Gambi, and S. Panichella. Cost- effective simulation-based test selection in self-driving cars software with sdc-scissor. In IEEE International Conference on Software Analy- sis, Evolution, and Reengineering. IEEE, 2022.
[8] C. Birchler, S. Khatiri, P. Derakhshanfar, S. Panichella, and A. Panichella. Single and multi-objective test cases prioritiza- tion for self-driving cars in virtual environments. arXiv preprint arXiv:2107.09614v2, 2022.
[9] J. H. Castellanos and J. Zhou. A modular hybrid learning approach for black-box security testing of cps. In International Conference on Applied Cryptography and Network Security, pages 196–216. Springer, 2019.
[10] P. M. Chu, M. Wen, J. Park, H. Kaisi, and K. Cho. Three-dimensional simulation for training autonomous vehicles in smart city environments. In International Conference on Internet of Things and IEEE Green Computing and Communications and IEEE Cyber, Physical and Social Computing and IEEE Smart Data, pages 848–853. IEEE, 2019.
[11] P. Derakhshanfar, A. Panichella, A. Gambi, V. Riccio, C. Birchler, and S. Panichella. TRAVEL: A Dataset with Toolchains for Test Generation and Regression Testing of Self-driving Cars Software, Jan. 2022.
[12] J. Deshmukh, M. Horvat, X. Jin, R. Majumdar, and V. S. Prabhu. Testing cyber-physical systems through bayesian optimization. ACM Transactions on Embedded Computing Systems (TECS), 16(5s):1–18, 2017.
[13] H.-M. Huang, T. Tidwell, C. Gill, C. Lu, X. Gao, and S. Dyke. Cyber- physical systems for real-time hybrid structural testing: a case study. In International Conference on Cyber-physical Systems, pages 69–78. ACM/IEEE, 2010.
[14] D. Humeniuk, F. Khomh, and G. Antoniol. A search-based framework for automatic generation of testing environments for cyber-physical systems. Information and Software Technology, page 106936, 2022.
[15] P. S. Kumar, W. Emfinger, and G. Karsai. A testbed to simulate and analyze resilient cyber-physical systems. In International Symposium on Rapid System Prototyping (RSP), pages 97–103. IEEE, 2015.
[16] T.Kuroiwa,Y.Aoyama,andN.Kushiro.Testingenvironmentforcpsby cooperating model checking with execution testing. Procedia Computer Science, 96:1341–1350, 2016.
[17] U. Markiegi, A. Arrieta, L. Etxeberria, and G. Sagardui. Dynamic test prioritization of product lines: An application on configurable simulation models. Software Quality Journal, 29(4):943–988, 2021.
[18] 2 killed in driverless tesla car crash, officials say. https://www.nytimes. com/2021/04/18/business/tesla-fatal-crash-texas.html. Accessed: 2022- 01-28.
[19] Drohne krachte in zu¨rich auf den boden - post bezeichnet vorfall als inakzeptabel. https://www.nzz.ch/zuerich/ absturz- von- post- drohne- bericht- stellt- gravierende- maengel- fest- ld. 1492295?reduced=true. Accessed: 2022-02-08.
[20] J. O’Callaghan. Few aeroplanes land automatically but new systems could make this the norm. https: //ec.europa.eu/research- and- innovation/en/horizon- magazine/
few- aeroplanes- land- automatically- new- systems- could- make- norm, October 2019. Accessed: 2022-06-25.
[21] After deadly 737 max crashes, damning whistleblower report reveals sidelined engineers, scarcity of expertise, more. https://www.theregister. com/2021/12/15/boeing 737 max senate report/. Accessed: 2022-01- 28.
[22] Post-drohne u¨ber zu¨richsee abgestu¨rzt. https: //www.srf.ch/news/regional/zuerich- schaffhausen/
blutprobe- verloren- post- drohne- ueber- zuerichsee- abgestuerzt. Accessed: 2022-02-08.
[23] J. Sun and Z. Yang. Objsim: efficient testing of cyber-physical sys- tems. In International Workshop on Testing, Analysis, and Verification of Cyber-Physical Systems and Internet of Things, pages 1–2. ACM SIGSOFT, 2020.
[24] Future of driving. https://www.tesla.com/autopilot. Accessed: 2022-06- 25.
[25] Transitioning to tesla vision. https://www.tesla.com/support/ transitioning-tesla-vision. Accessed: 2022-06-25.
[26] Waymo. https://waymo.com/. Accessed: 2022-06-25.
[27] X. Zhou, X. Gou, T. Huang, and S. Yang. Review on testing of cyber physical systems: Methods and testbeds. IEEE Access, 6:52179–52194,
2018.
34