| Author: | Zhang, Shiyu |
| Title: | Design and evaluation of test Oracles for control-CPS and reinforcement learning software |
| Advisors: | Wang, Qixin (COMP) Usmani, Asif (BEEE) Xiao, Fu Linda (BEEE) |
| Degree: | Ph.D. |
| Year: | 2026 |
| Department: | Department of Computing |
| Pages: | xiv, 102 pages : color illustrations |
| Language: | English |
| Abstract: | In this thesis, we focus on the challenge of designing test oracles for control cyber-physical systems (control-CPSs), aka the oracle problem for control-CPSs. This challenge arises because, for conventional control-CPSs, a given input (high level control goal) can correspond to multiple legitimate outputs (specific physical trajectories of the controlled plant). This problem is worsened for control-CPSs using reinforcement learning (RL) to generate controllers: not only the legitimate physical trajectories are not unique, the legitimate controllers generated by the RL are not unique either. To address this challenge, we design and evaluate three novel test oracles. For general control-CPS software, we propose a Transformer-based oracle (TO) and compare it with the known best practice of Auto-Regressive System Identification (AR-SI) oracle (AO), focusing on software fault localization (SFL) quality metrics. Our comparison results show that TO and AO perform similarly in SFL quality, but TO has a slightly better performance than AO in terms of latency; AO has a lower false positive rate and TO has a lower false negative rate. As for overall performance, there is no significant differences between TO and AO. For RL software, we propose two test oracles: a fuzzy logic-guided oracle and a Lyapunov stability theory-based oracle (LPEA). We compare both oracles with the mainstream best practice of human oracle (HO). Evaluation results show that the fuzzy logic-guided oracle is not significantly better than the human oracle, except for a few metrics. However, the LPEA oracle can significantly outperform the human oracle in most of the metrics. Particularly, LPEA(ϑ = 100%, θ = 75%) outperforms the human oracle by 53.6%, 50%, 18.4%, 34.8%, 18.4%, 127.8%, 60.5%, 38.9%, and 31.7% respectively on accuracy, precision, recall, F1 score, true positive rate, true negative rate, false positive rate, false negative rate, and ROC curve AUC; and LPEA(ϑ = 100%, θ = 50%) outperforms the human oracle by 48.2%, 47.4%, 10.5%, 29.1%, 10.5%, 127.8%, 60.5%, 22.2%, and 26.0% respectively on these metrics. |
| Rights: | All rights reserved |
| Access: | open access |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/14416