Detection of Code Vulnerabilities via Constraint-Based Analysis and Model Checking

Compared with traditional program analysis, model checking is preponderant in improving the precision of vulnerabilities detection. However, it is hard to directly apply model checking to detect buffer overflow, code injection and other security vulnerabilities. To address this problem, an approach that combines the constraint-based analysis and the model checking together to detect vulnerabilities automatically is proposed in this paper. At first we trace the information of the buffer-related variables in source code by the constraint-based analysis, and instrument the code with corresponding attribute transfer and constraint assertions of the buffers before the potential vulnerable points that are related to the buffers. And then the problem of detecting vulnerabilities is converted into the problem of verifying the reachability of these constraint assertions. Model checking is used to verify the reachability of the security vulnerabilities. In addition, we introduce program slicing to reduce the code size in order to reduce the state space of model checking. CodeAuditor is the prototype implementation of our methods. With this tool, 18 previously unknown vulnerabilities in six open source applications are discovered and the observed false positive rate is at around 23%. The result of minicom’s slicing shows that the performance of detection is improved.