Fault-Detection Efficiency Under Fixed Compute Budgets: A Comparative Evaluation of Test Case Selection and Prioritization Strategies for Autonomous Driving Simulation Regression Testing

Jianuo Wang; Yi Guo

Fault-Detection Efficiency Under Fixed Compute Budgets: A Comparative Evaluation of Test Case Selection and Prioritization Strategies for Autonomous Driving Simulation Regression Testing

Authors

Jianuo Wang Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA
Yi Guo Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA

Keywords:

regression test prioritization, autonomous driving simulation, compute budget allocation, empirical comparison

Abstract

Regression testing of autonomous driving software relies on simulation scenario libraries that grow continuously, while the computational budget available for each verification cycle remains bounded. Executing the full scenario suite is expensive, and uninformed sampling frequently fails to surface defect-revealing cases. This study reports a comparative evaluation of five test case selection and prioritization strategies for simulation-based regression testing under fixed execution budgets: random sampling, additional-coverage greedy selection over the scenario parameter space, similarity-based redundancy reduction, learning to rank over historical execution records, and change-aware selection driven by a module dependency mapping. Scenarios are encoded by actor class, manoeuvre, ego behaviour, and proximity, giving a common input representation across strategies. The evaluation uses 33,530 recorded executable driving scenarios from a public autonomous driving system scenario dataset, drawn from 2,880 test executions and arranged as a regression matrix of 87 reusable scenario configurations across 48 operational contexts, of which 40 are used for evaluation; the cost sensitivity of the learned ranker, against a random baseline, is then examined on two industrial continuous-integration histories that record real per-test execution cost. Learning to rank attains the highest average percentage of faults detected (0.837 against 0.754 for random selection) and reaches the first failing case at position 1.19 on average. In this corpus that advantage cannot be separated from replay density, because a case's verdict is the union of violations over an uneven number of recorded instances. Change-aware selection, driven by an operational-context proxy for code change, falls 0.043 APFD below random over the full ordering, a gap that pairwise contrasts detect but that does not survive the Nemenyi post-hoc procedure. Cost-cognizant scoring reverses the ranking of history-based variants on both industrial histories.

References

Dosovitskiy, A., Ros, G., Codevilla, F., López, A., & Koltun, V. (2017). CARLA: An open urban driving simulator. Proceedings of the 1st Annual Conference on Robot Learning (CoRL), 1–16.

Haq, F. U., Shin, D., & Briand, L. (2022). Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization. Proceedings of the 44th International Conference on Software Engineering (ICSE), 811–822.

Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification and Reliability, 22(2), 67–120.

Lu, C., Zhang, H., Yue, T., & Ali, S. (2021). Search-based selection and prioritization of test scenarios for autonomous driving systems. Proceedings of the 13th International Symposium on Search-Based Software Engineering (SSBSE), 41–55.

Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (2001). Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10), 929–948.

Elbaum, S., Malishevsky, A. G., & Rothermel, G. (2002). Test case prioritization: A family of empirical studies. IEEE Transactions on Software Engineering, 28(2), 159–182.

Elbaum, S., Malishevsky, A., & Rothermel, G. (2001). Incorporating varying test costs and fault severities into test case prioritization. Proceedings of the 23rd International Conference on Software Engineering (ICSE), 329–338.

Di Nardo, D., Alshahwan, N., Briand, L., & Labiche, Y. (2013). Coverage-based test case prioritisation: An industrial case study. Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST), 302–311.

Chen, T. Y., Kuo, F.-C., Merkel, R. G., & Tse, T. H. (2010). Adaptive random testing: The ART of test case diversity. Journal of Systems and Software, 83(1), 60–66.

Yoo, S., Harman, M., Tonella, P., & Susi, A. (2009). Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA), 201–212.

Machalica, M., Samylkin, A., Porth, M., & Chandra, S. (2019). Predictive test selection. Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 91–100.

Herzig, K., Greiler, M., Czerwonka, J., & Murphy, B. (2015). The art of testing less without sacrificing quality. Proceedings of the 37th International Conference on Software Engineering (ICSE), 483–493.

Ben Abdessalem, R., Panichella, A., Nejati, S., Briand, L. C., & Stifter, T. (2018). Testing autonomous cars for feature interaction failures using many-objective search. Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE), 143–154.

Fremont, D. J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2019). Scenic: A language for scenario specification and scene generation. Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 63–78.

Lu, C., Yue, T., & Ali, S. (2023). DeepScenario: An open driving scenario dataset for autonomous driving system testing. Proceedings of the 20th IEEE/ACM International Conference on Mining Software Repositories (MSR), 52–56.

Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement learning for automatic test case prioritization and selection in continuous integration. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 12–22.

Hemmati, H., Arcuri, A., & Briand, L. (2013). Achieving scalable model-based testing through test case diversity. ACM Transactions on Software Engineering and Methodology, 22(1), Article 6.

Gligoric, M., Eloussi, L., & Marinov, D. (2015). Practical regression test selection with dynamic file dependencies. Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA), 211–222.

Luo, Q., Hariri, F., Eloussi, L., & Marinov, D. (2014). An empirical analysis of flaky tests. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 643–653.

Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014). Are mutants a valid substitute for real faults in software testing? Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 654–665.

Kim, J.-M., & Porter, A. (2002). A history-based test prioritization technique for regression testing in resource constrained environments. Proceedings of the 24th International Conference on Software Engineering (ICSE), 119–129.

Epitropakis, M. G., Yoo, S., Harman, M., & Burke, E. K. (2015). Empirical evaluation of Pareto efficient multi-objective regression test case prioritisation. Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA), 234–245.

Amini, M. H., Naseri, S., & Nejati, S. (2024). Evaluating the impact of flaky simulators on testing autonomous driving systems. Empirical Software Engineering, 29(2), Article 47.

[24] Guo, Y. (2026). A Multi-Dimensional Coverage Metric with Evolutionary Search for Safety-Critical Scenario Generation in Autonomous Driving Testing. Journal of Science, Innovation & Social Impact, 2(2), 82-93.

[25] Li, Y. (2026). Benchmarking Robustness-Efficiency Trade-offs of Camera-LiDAR Fusion Strategies for 3D Object Detection Under Environmental Corruptions. Journal of Science, Innovation & Social Impact, 2(2), 104-116.

[26] Wu, C., Li, Y., Li, L., Wang, L., & Liu, Y. (2020). Caption Generation from Road Images for Traffic Scene Construction. In 2020 IEEE Intelligent Vehicles Symposium (IV) (pp. 1271-1276). IEEE.

[27] Li, Y., Wu, C., Li, L., Liu, Y., & Zhu, J. (2021). Caption generation from road images for traffic scene modeling. IEEE Transactions on intelligent transportation systems, 23(7), 7805-7816.

[28] Guo, Y. (2025). Performance Evaluation of Lightweight Detection Algorithms on Compact LiDAR-Camera Configurations for Freight Transportation. Journal of Science, Innovation & Social Impact, 1(1), 398-409.

[29] Guo, Y. (2025). Reliability Assessment and Adaptive Fusion Algorithm for Multi-Sensor Data in Autonomous Driving under Adverse Weather Conditions. Journal of Sustainability, Policy, and Practice, 1(4), 143-155.

[30] Wu, X., Li, Y., Liu, Y., Pang, S., Wang, L., Wu, C., & Huo, H. (2019). Jointly detecting and retrieving vehicles from road image sequences based on CNN. In 2019 IEEE Intelligent Vehicles Symposium (IV) (pp. 2530-2535). IEEE.

[31] Li, Y. (2025, December). Comparative Analysis of Illumination Normalization Methods for Autonomous Driving Under Challenging Lighting Conditions. In Proceedings of the 2025 6th International Conference on Computer Science and Management Technology (pp. 633-639).

[32] Huo, H., Li, Y., Wu, C., Wu, X., Tian, Z., & Liu, Y. (2019). Semantic segmentation and scene reconstruction for traffic simulation using CNN. In 2019 2nd China Symposium on Cognitive Computing and Hybrid Intelligence (CCHI) (pp. 117-122). IEEE.

Fault-Detection Efficiency Under Fixed Compute Budgets: A Comparative Evaluation of Test Case Selection and Prioritization Strategies for Autonomous Driving Simulation Regression Testing