高级检索

    结构锚定多模态对比学习的检索增强知识图谱推理

    Structure-Anchored Multimodal Contrastive Learning based Retrieval-augmented Knowledge Graph Reasoning

    • 摘要: 多模态知识图谱是智能信息系统的重要知识基础设施。然而,真实应用中新兴实体的持续涌现为现有的知识检索增强推理系统带来严峻挑战,制约了智能信息系统的知识动态适应性。具体而言,现有方法主要聚焦于图文语义对齐,忽略了图谱拓扑结构对推理的约束作用;且因模态异构,多种模态往往难以协同提升新兴实体推理性能。为此,提出基于结构锚定多模态对比学习的检索增强归纳推理框架SaCR,以图结构为锚点统一多模态表征并提升推理泛化能力。首先,提出实体邻居锚点和关系锚点机制,将图结构信息映射到统一文本语义空间,实现图结构与图文模态的协同建模。其次,通过多模态统一对比学习联合优化结构锚点与多模态编码器,拉近查询与目标实体表征距离。最后,通过检索增强推理机制提升新兴实体推理泛化能力。在归纳式和转导式推理设定下的5个数据集上的实验表明,SaCR有效提升了新兴实体的多模态知识图谱推理性能,为大模型驱动的智能信息系统提供了更完备的动态知识基础。

       

      Abstract: Multimodal knowledge graphs serve as critical knowledge infrastructure for intelligent information systems. However, the continuous emergence of new entities in real-world applications poses critical challenges to existing knowledge retrieval-augmented reasoning systems, constraining the dynamic adaptability of intelligent information systems. Existing methods mainly focus on textual-visual semantic alignment, neglecting the constraining effect of graph topological structure on reasoning. Moreover, due to modality heterogeneity, multiple modalities are often difficult to collaboratively improve the reasoning performance on emerging entities. To this end, we propose SaCR, a Structure-Anchored Multimodal Contrastive Learning based Retrieval-Augmented Inductive Reasoning framework, which unifies multimodal representations with graph structure as anchors and enhances reasoning generalization. First, we propose entity neighbor anchor and relation anchor mechanisms to map graph structural information into a unified textual semantic space, achieving collaborative modeling of graph structure and textual-visual modalities. Second, through multimodal unified contrastive learning, we jointly optimize the structure anchors and multimodal encoders to shorten the distance between query and target entity representations. Finally, a retrieval-augmented inference mechanism is introduced to improve the generalization of reasoning on emerging entities. Experiments on 5 datasets under both inductive and transductive reasoning settings demonstrate that SaCR effectively improves the multimodal knowledge graph reasoning performance on emerging entities, providing complete and dynamic knowledge foundation for LLM-driven intelligent systems.

       

    /

    返回文章
    返回