Abstract: Deep learning technology has been increasingly empowering the practice of dermatological disease diagnosis, leading to a steady improvement in its diagnostic performance. However, such data-driven deep learning models are plagued by severe bias issues. If systems with these biases are deployed in clinical applications, they are bound to cause unfair treatment towards specific groups, such as underrepresented groups. Ensuring the fairness of decision-making has become a critical and urgent problem that needs to be addressed. Existing research has indicated that enhancing fairness often comes at the cost of reduced diagnostic performance. Therefore, to achieve a better trade-off between diagnostic performance and fairness, while simultaneously alleviating the unfair treatment of underrepresented groups, we propose an Enhanced Balanced Incremental Distillation Network (EBID-Net). Specifically, with the aid of balanced memory, the model leverages representative demographic groups to assist underrepresented groups in knowledge learning, while being incrementally trained by integrating distributions across different groups. Additionally, we incorporate global information into the contextual attention mechanism to capture correlated interactions between features across different spatial locations, thereby obtaining more robust feature representations during incremental learning. Experimental results show that our network outperforms other methods in fairness criteria and in the trade-off between fairness and diagnostic performance.