ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2016, Vol. 53 ›› Issue (2): 449-458.doi: 10.7544/issn1000-1239.2016.20148275

• 软件技术 • 上一篇    下一篇

大数据集成中确定数据准确属性值的WR方法

周宁南1,3,盛万兴1,刘科研1,张孝2,3,王珊2,3   

  1. 1(中国电力科学研究院 北京 100192); 2(数据工程与知识工程教育部重点实验室(中国人民大学) 北京 100872); 3(中国人民大学信息学院 北京 100872) (zhangxiao@ruc.edu.cn)
  • 出版日期: 2016-02-01
  • 基金资助: 
    国家“九七三”重点基础研究发展计划基金项目(2014CB340403);国家电网公司研究项目(EPRIPDKJ[2014]3763号)

WR Approach: Determining Accurate Attribute Values in Big Data Integration

Zhou Ningnan1,3, Sheng Wanxing1, Liu Ke-yan1, Zhang Xiao2,3, Wang Shan2,3   

  1. 1(China Electric Power Research Institute, Beijing 100192); 2(Key Laboratory of Data Engineering and Knowledge Engineering (Renmin University of China), Ministry of Education, Beijing 100872); 3(School of Information, Renmin University of China, Beijing 100872)
  • Online: 2016-02-01

摘要: 大数据集成是提供高质量数据以进行决策的基础.集成的一个关键环节是根据实体在数据库中的不同元组确定其准确属性值.最新的R-topK方法在数据上实施人工设计的规则确定属性值间的准确程度,得到了相对准确的属性值.然而这种方法在处理多个可能的准确值或设计的规则存在冲突等情况下需要较多人工交互.为此提出基于权重规则的WR(weighted-rule)方法确定大数据集成中数据的准确属性值.该方法为属性值间准确程度的判断规则扩充了权重,在准确值发生冲突时避免了R-topK方法中人工交互干预.基于追逐过程设计了约束条件推理算法,并证明它能够在O(n\+2)内推导出每对属性值间的带权重的准确程度,形成推导准确属性值的约束条件.面对约束条件中可能的冲突,提出了目标求解算法,在O(n)时间内从所有属性值组合中搜索最可能的准确属性值.在真实和合成数据集中进行了充分的实验,验证了WR方法的效果和效率.WR方法较R-topK方法在性能上提高了3~15倍,在效果上提升7%~80%.

关键词: 大数据集成, 数据质量, 数据准确性, 数据清洗, 权重规则

Abstract: Big data integration lays the foundation for high quality data-driven decision. One critical section thereof is to determine the accurate attribute values from records in data pertaining to a given entity. The state-of-the-art approach R-topK argues to design rules to decide relative accuracy among the attribute values and thus obtain accurate values. Unfortunately, in cases where multiple true values or conflicted rules exist, it requires rounds of human intervention. In this paper, we propose a weighted rule (WR) approach for determining accurate attribute values in big data integration. Each rule is augmented with weight and thus avoid human intervention when conflicts occur. This paper designs a chase procedure-based inference algorithm, and proves that it can figure out weighted constraints over relative accuracy among attribute values in O(n\+2), which introduces constraints for finding accurate data values. Taking conflicts among constraints into consideration, this paper proposes an O(n) algorithm to discover accurate attribute values among the combination of data values. We conduct extensive experiments under real world and synthetic datasets, and the results demonstrate the effectiveness and efficiency of WR approach. WR approach boosts performance by factor of 3-15x and improves effectiveness by 7%-80%.

Key words: big data integration, data quality, data accuracy, data cleaning, weighted rules

中图分类号: