ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2015, Vol. 52 ›› Issue (9): 1941-1953.doi: 10.7544/issn1000-1239.2015.20140533

• 软件技术 • 上一篇    下一篇

基于频繁项集的海量短文本聚类与主题抽取

彭敏1,2, 黄佳佳1, 朱佳晖3, 黄济民1, 刘纪平1   

  1. 1(武汉大学计算机学院 武汉 430072); 2(武汉大学深圳研究院 广东深圳 518057); 3(软件工程国家重点实验室(武汉大学) 武汉 430072) (pengm@whu.edu.cn)
  • 出版日期: 2015-09-01
  • 基金资助: 
    基金项目:国家自然科学基金项目(61472291,61303115,61272110);2013年深圳知识创新计划基础研究项目

Mass of Short Texts Clustering and Topic Extraction Based on Frequent Itemsets

Peng Min1,2, Huang Jiajia1, Zhu Jiahui3, Huang Jimin1, Liu Jiping1   

  1. 1(Computer School, Wuhan University, Wuhan 430072); 2(Shenzhen Research, Wuhan University, Shenzhen, Guangdong 518057); 3(State Key Laboratory of Software Engineering(Wuhan University), Wuhan 430072)
  • Online: 2015-09-01

摘要: 社交网络短文本规模大、传播快、质量低、模态多样等特性导致现有基于向量空间模型的文本聚类技术在对其进行聚类时面临维度高、特征稀疏和噪声干扰等挑战.对此,提出基于频繁项集的短文本聚类与主题抽取STC-TE(short text clustering & topic extraction)框架.首先研究短文本的多特征对文本质量的影响,在基于高质量短文本集挖掘出的大量频繁项集基础上,设计基于相似度的频繁项集过滤策略SIF(similarity-based itemset filtering),可过滤掉85%的非重要频繁项集;然后定义基于相关文本集的频繁项集相似度,并提出聚类个数自适应的频繁项集谱聚类算法CSA_SC(clusters self-adaptive spectral clustering),实现频繁项集聚类与主题抽取;最后基于主题词将大规模短文本划分到相应的主题簇中,从而实现短文本聚类.基于100万条新浪微博文本的实验结果表明,STC-TE框架能够全面准确地抽取主题信息,快速有效地实现海量短文本分类.

关键词: 海量, 短文本, 频繁项集, 聚类, 主题抽取

Abstract: Short texts generated in social media have the characteristics of volume, velocity, low quality and variety, thus make the vector-space-based clustering methods face the challenges of high-dimensions, features sparsity and noisy disturbing. In this paper, we propose a short texts clustering and topic extraction (STC-TE) framework based on the frequent itemsets mined from the texts. This framework firstly studies the impact of multi-features on the short texts’ quality. Then, a large amount of frequent itemsets are dug out from the high quality short text set via setting a low support level, and a similar itemsets filtering strategy is devised to discard most of the unimportant frequent itemsets. Furthermore, based on the frequent itemsets similarity evaluated by relevant texts, we proposed a cluster self-adaptive spectral clustering (CSA_SC) algorithm to form the itemsets into different topic clusters. At last, the large-scale of short texts are classified into associated clusters according to the topic words extracted from the frequent itemset clusters. The framework is tested on one million of SinaWeibo dataset to evaluate the performance of the important frequent itemset selection and clustering, the topic words extraction, and the large scale of short texts classification. Experimental results show that the STC-TE framework can achieve topic extraction and large-scale short texts clustering with high accuracy.

Key words: large-scale, short texts, frequent itemsets, clustering, topic extraction

中图分类号: