Abstract: Defect detection is an important applicaion of computer vision in industry, but it is a challenge to effectively inspect defects in a vision system because of illumination inequality and the variation of reflection property of objects. Sparseness is one of the most improtant characteristics of defect images, and therefore the approach named defect decomposition based on sparseness (DDBS) is proposed. In the framework of DDBS, a defect image is treated as linearly combination of three components: background, defects and noise. Background is coded by an over-complete sparse dictionary, which can not sparsely represent defect component. Meanwhile defect is sparsely coded by its dictionary that is exclusive to background. Then defect component is decomposed using DDBS based on the principle of blind sources sepration and block-coordinate relaxation. Furthermore, DDBS is applied in rail surface defect detection to improve the robustness of inspection systems. Experiments are carried out for different dictionary combinations based on actual rail images, and the results demonstrate that DDBS can effectively detect the defects that are missed by the state-of-the-art methods. DDBS is a flexible framwork for applications of defect detection and has the potential benefit to improve robustness of traditional methods.