Abstract: As a common visual foundation model, the segment anything model (SAM) has been widely applied in various image segmentation tasks, but it exhibits vulnerability in the face of adversarial attacks. A adversarial transfer attack method FSGR (fragile section gradient robustness) has been proposed for vulnerable modules of SAM downstream models, which can effectively generate adversarial samples to attack SAM downstream models without understanding the details of downstream fine-tuning. The core of FSGR lies in its ability to identify the most vulnerable parts and apply targeted attacks to these weak points. This method designs a strategy based on feature similarity to identify the most vulnerable module in the SAM encoder and enhance the attack effectiveness in a targeted manner; and introduce gradient robust loss to eliminate the gradient difference between the target model and downstream task models. FSGR combines vulnerability layer attacks and gradient robust loss to achieve relative performance improvement on multiple datasets. The experimental results show that FSGR significantly reduces model performance in transfer attacks on various fine-tuning models, such as medical segmentation, shadow segmentation, and camouflage segmentation, demonstrating its correctness and practicality. Compared with baseline methods, FSGR not only performs well in attack success rate, but also achieves relative performance improvement by combining vulnerability layer attacks and gradient robust loss.