Evaluating the Fitness of Model Deviation Detection Approaches on Self-Adaptive Software Systems

Model deviations in self-adaptive software systems cause critical reliability issues. For control-based self-adaptive systems, model deviation roots in the drifting of the managed system’s nominal model in uncertain running environments, which causes the invalidation of provided formal guarantees, and may lead to system’s abnormal behavior. Existing model deviation detection approaches often ignore the characteristics of model deviations that emerge in different scenarios. This makes it difficult for users to choose an appropriate approach in a specific application scenario. We provide a framework to describe different detection approaches and propose three metrics to evaluate a detection approach’s fitness with respect to different types of model deviations. The provided framework is composed of four parts, namely system modelling, detection variable estimation, model deviation representation, and model deviation judgement, based on the process of model deviation detection. The proposed metrics, including control-signal-intensity, environmental-input-intensity, and uncertainty-intensity, concern three key factors in the process of model deviation detection, respectively. Using these metrics, a deviation scenario is quantified with a vector and is classified by the quantified values into a characteristic scenario according to control theory. A number of experiments are conducted to study the effectiveness of four mainstream model detection approaches in different scenarios, and their fitness to different characteristic scenarios of model deviations is summarized.