ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2017, Vol. 54 ›› Issue (9): 1966-1978.doi: 10.7544/issn1000-1239.2017.20160546

• 人工智能 • 上一篇    下一篇

Spark环境下基于频繁边的大规模单图采样算法

李龙洋,董一鸿,严玉良,陈华辉,钱江波   

  1. (宁波大学信息科学与工程学院 浙江宁波 315211) (sgavin1991@163.com)
  • 出版日期: 2017-09-01
  • 基金资助: 
    国家自然科学基金项目(61572266,61472194);浙江省自然科学基金项目(Y16F020003);宁波市自然科学基金项目(2017A610114)

A Sampling Algorithm Based on Frequent Edges in Single Large-Scale Graph Under Spark

Li Longyang, Dong Yihong, Yan Yuliang, Chen Huahui, Qian Jiangbo   

  1. (Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang 315211)
  • Online: 2017-09-01

摘要: 随着社交网络的流行,对其进行频繁子图挖掘的需求越来越强烈.大数据时代的到来,社交网络规模不断扩大,频繁子图挖掘工作变得愈发困难.在实际应用中,往往并不需要精确地挖掘出频繁子图,采样的方法在保证一定准确率的前提下能够显著提高频繁子图挖掘的效率.现有采样算法大多是根据节点的度进行采样,不适用于频繁子图挖掘.提出了一种基于频繁边的采样算法DIMSARI(distributed Monte Carlo sampling algorithm based on random jump and graph induction),在蒙特卡罗算法的基础上增加了根据频繁边进行随机跳的操作,并对其结果进行了图感应操作,进一步增加了算法的准确性,并在理论上证明了该方法的无偏性.实验结果显示:使用DIMSARI算法采样后进行频繁子图挖掘,准确性比现有其他的采样算法有较大的提高,在不同的采样率下采样后的子图的节点度都保持更小的归一化均方偏差.

关键词: 采样, 频繁子图, 大规模单图, 频繁边, Spark

Abstract: With the popularity of social networks, the demand for its frequent subgraph mining becomes more intense. With the arrival of the era of big data, social networks have been expanding and frequent subgraph mining becomes increasingly difficult. In fact, it does not require to mine frequent subgraphs exactly in application, so sampling methods are adopted to improve the efficiency of mining frequent subgraphs under certain accuracy. Most existing sampling algorithms are not fit for frequent subgraph mining because they use vertex transfer or compute the topology of the original graph first which will take a lot of time. In this paper, we propose a new sampling algorithm named DIMSARI (distributed Monte Carlo sampling algorithm based on random jump and graph induction) based on frequent edge, and it runs on a distributed framework named Spark. This algorithm is created on the basis of the Monte Carlo algorithm meanwhile adding random jump. The results are added by subgraph induction step to promote the accuracy of the algorithm and prove that the algorithm is unbiased. The experiments show that the accuracy of frequent subgraph mining using DIMSARI algorithm has been greatly improved and at the same time the proposed algorithm only spends a little more time than other algorithms. The apex of sampling at different sampling rates after subgraphs has maintained a lower normalized mean square error.

Key words: sample, frequent subgraph, single large-scale graph, frequent edge, Spark

中图分类号: