Abstract:
With the widespread application of lightweight models in resource-constrained scenarios such as edge computing, their insufficient adversarial robustness has become increasingly prominent. While adversarial distillation serves as a primary approach to enhance lightweight models’ robustness, existing methods predominantly employ KL divergence-based rigid constraints for robust knowledge transfer, which still suffer from flawed knowledge modeling and limited transfer efficiency. To address these issues, this paper proposes an Adversarial Ranking Distillation (ARD) method that significantly improves lightweight models’ robustness through a priority ranking constraint mechanism and a multi-level consistency framework. Specifically, the priority ranking constraint mechanism organizes the output elements of vision models on adversarial samples by importance ranking, enforces teacher-student ranking consistency, and implements differentiable loose coupling constraints by approximating discontinuous ranking processes through hyperbolic tangent functions. Building upon this, the multi-level ranking consistency distillation framework models and transfers robustness knowledge from three perspectives: categorical semantic correlation, sample semantic correlation, and adversarial discrepancy correlation, enabling multi-perspective transmission of teachers’ adversarial defense capabilities while synergistically improving lightweight models’ clean sample accuracy and adversarial robustness. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets validate the effectiveness of our method, demonstrating substantial performance advantages over state-of-the-art adversarial distillation approaches. Furthermore, the proposed ARD exhibits enhanced adaptability and generalization capabilities under scenarios of limited training data and black-box attacks, providing an efficient security solution for lightweight model deployment in edge computing environments.