ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2019, Vol. 56 ›› Issue (8): 1721-1730.doi: 10.7544/issn1000-1239.2019.20190329

所属专题: 2019人工智能前沿进展专题

• 人工智能 • 上一篇    下一篇

一种残差置乱上下文信息的场景图生成方法

林欣1,田鑫1,季怡1,徐云龙2,刘纯平1,3   

  1. 1(苏州大学计算机科学与技术学院 江苏苏州 215006);2(苏州大学应用技术学院 江苏苏州 215300);3(符号计算与知识工程教育部重点实验室(吉林大学) 长春 130012) (xlin2017@stu.suda.edu.cn)
  • 出版日期: 2019-08-01
  • 基金资助: 
    国家自然科学基金项目(61773272,61272258,61301299);吉林大学符号计算与知识工程教育部重点实验室项目(93K172016K08);江苏高校优势学科建设工程资助项目

Scene Graph Generation Based on Shuffle Residual Context Information

Lin Xin1, Tian Xin1, Ji Yi1, Xu Yunlong2, Liu Chunping1,3   

  1. 1(School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006);2(Applied Technology College of Soochow University, Suzhou, Jiangsu 215300);3(Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012)
  • Online: 2019-08-01

摘要: 场景图在视觉理解中有着很重要的作用.现有的场景图生成方法对于主语、宾语以及主宾语间的视觉关系进行研究.但是,人类通过空间关系上下文、语义上下文和目标之间的互动信息来进行关系的理解和推理.为了获得更好的全局上下文表示,同时减少数据集偏差的影响,提出了一个新的场景图生成框架RSSQ(residual shuffle sequence model).该框架由目标解码、残差置乱和位置嵌入3部分构成.残差置乱模块由随机置乱和残差连接的双向LSTM的基本结构叠加而成,利用迭代方式实现随机打乱双向LSTM的隐藏状态以减少数据集偏差影响,利用残差连接提取共享的全局上下文信息.在位置嵌入模块中,通过对目标的相对位置和面积比例的编码则可以增强目标对之间的空间关系.在数据集Visual Genome的3个不同层次子任务的实验中,证明了提出的RSSQ方法因全局上下文改善和空间关系增强,在Recall@50和Recall@100指标评价下,相对于现有方法能生成更好的场景图.

关键词: 场景图, 视觉关系, 上下文, 残差双向LSTM, 目标检测

Abstract: Scene graphs play an important role in visual understanding. Existing scene graph generation methods focus on the research of the subjects, the objects as well as the predicates between them. However, human being abstracts the relationships using spatial relation context, semantic context and interaction between scene objects for better understanding and reasoning as whole. In order to obtain the better global context representation and reduce the impact of dataset bias, we propose a new framework of scene graph generation, called as residual shuffle sequence model (RSSQ). Our method is made up of object decoding, residual shuffle and position embedding modules. Residual shuffle module is stacked with two basic structures including the random shuffle operation and the residual bidirectional LSTM. We implement the random shuffle on the hidden state of bidirectional LSTM by the process of iterative operation to reduce the impact of dataset bias, and extract the shared global context information by the residual connection structure. To strengthen the spatial relationship between pair-wise objects, the encoding is achieved using the relative position and area ratio of objects in position embedding module. The experimental results of three sub-tasks of different difficulty performed on Visual Genome dataset, demonstrate that the poposed method can generate better scene graphs under Recall@50 and Recall@100 settings due to better global context and spatial information.

Key words: scene graph, visual relationship, context, residual bidirectional LSTM, object detection

中图分类号: