ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2018, Vol. 55 ›› Issue (1): 102-112.doi: 10.7544/issn1000-1239.2018.20160662

• 人工智能 • 上一篇    下一篇

一种基于频繁词集表示的新文本聚类方法

张雪松,贾彩燕   

  1. (交通数据分析与数据挖掘北京市重点实验室(北京交通大学) 北京 100044) (北京交通大学计算机与信息技术学院 北京 100044) (15120467@bjtu.edu.cn)
  • 出版日期: 2018-01-01
  • 基金资助: 
    国家自然科学基金面上项目(61473030);数字出版国家重点实验室专项课题

A New Documents Clustering Method Based on Frequent Itemsets

Zhang Xuesong, Jia Caiyan   

  1. (Beijing Key Lab of Traffic Data Analysis and Mining (Beijing Jiaotong University), Beijing 100044) (School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044)
  • Online: 2018-01-01

摘要: 传统的文本聚类方法大部分采用基于词的文本表示模型,这种模型只考虑单个词的重要度而忽略了词与词之间的语义关系.同时,传统文本表示模型存在高维的问题.为解决以上问题,提出一种基于频繁词集的文本聚类方法(frequent itemsets based document clustering method, FIC).该方法从文档集中运用FP-Growth算法挖掘出频繁词集,运用频繁词集来表示每个文本从而大大降低了文本维度,根据文本间相似度建立文本网络,运用社区划分的算法对网络进行划分,从而达到文本聚类的目的.FIC算法不仅能降低文本表示的维度,还可以构建文本集中文本间的关联关系,使文本与文本间不再是独立的两两关系.实验中运用2个英文语料库Reuters-21578,20NewsGroup和1个中文语料库——搜狗新闻数据集来测试算法精度.实验表明:较传统的利用文本空间向量模型的聚类方法,该方法能够有效地降低文本表示的维度,并且,相比于常见的基于频繁词集的聚类方法能获得更好的聚类效果.

关键词: 文本聚类, 频繁词集, 复杂网络, 社区划分, 文本表示模型

Abstract: Traditional document clustering methods use vector space model (VSM) of words to represent documents. This VSM representation only measures the importance of a single words, while ignores the semantic relationship between words, and has high dimensionality. In this study, we propose a new document clustering method: FIC (frequent itemsets based document clustering method). In the method, we use frequent itemsets (where a frequent itemset is a set of frequently co-occurred words) mined by FP-Growth algorithm in documents to represent each document. We then construct the document-document relationship network based on the similarity between pairs of documents at this new representation. At last, we divide the network into communities using a given community detection method to complete document clustering. Thereby, FIC can not only overcome the high dimensionality of VSM, but also fully make use of topological relationship among documents. The experimental results on two English corpora (Reters-21578 and 20Newsgroup) and one Chinese corpus (Sougou-News) demonstrate that the proposed method FIC is superior to the other existing frequent itemsets based methods and other classical state-of-the-art document clustering methods, and the top K words for characterizing each topic of documents identified by FIC are more meaningful than the classical topic model LDA (latent Dirichlet allocation).

Key words: document clustering, frequent itemsets, complex network, community division, text representation model

中图分类号: