Multi-Modal Knowledge-Aware Attention Network for Question Answering

With the popularity of the Internet, more people choose to search online to find the solutions when they feel sick. With the emergence of reliable medical question answering websites, e.g. Chunyu Doctor, XYWY, patients can communicate with the doctor one-one at home. However, existing question answering methods focus on word-level interaction or semantics, but rarely notice the hidden rationale with doctors’ commonsense, while in the real scenes, doctors need to acquire plenty of domain knowledge to give advice to the patients. This paper proposes a novel multi-modal knowledge-aware attention network (MKAN) to effectively exploit multi-modal knowledge graph for medical question answering. The incorporation of multi-modal information can provide more fine-grained information. This information shows how entities in the medical graph are related. Our model first generates multi-modal entity representation with a translation-based method, and then defines question-answer interactions as the paths in the multi-modal knowledge graph that connect the entities in the question and answer. Furthermore, to discriminate the importance of paths, we propose an attention network. We build a large-scale multi-modal medical knowledge graph based on Symptom-in-Chinese, as well as one real-world medical question answering datasets based on Chunyu Doctor website. Extensive experiments strongly evidence that our proposed model obtains significant performance compared with state-of-the arts.