高级检索

    现代汉语通用分词系统中歧义切分的实用技术

    Disambiguation in a Modern Chinese General-Purpose Word Segmentation System

    • 摘要: 歧义切分技术是中文自动分词系统的关键技术之一.特别是在现代汉语通用分词系统(GPWS)中,允许用户动态创建词库、允许多个用户词库同时参与切分,这给歧义切分技术提出了更高的实用性要求.从大规模的真实语料库中,考察了歧义(特别是交集型歧义)的分布情况和特征;提出了一种改进的正向最大匹配歧义字段发现算法;并根据GPWS的需求,提出了一种“规则+例外”的实用消歧策略.对1亿字《人民日报》语料(约234MB)中的交集型歧义字段进行了穷尽式的抽取,并随机的对上述策略进行了开放性测试,正确率达99%.

       

      Abstract: Disambiguation is one of the most important parts of segment systems in Chinese. A Chinese general-purpose word segmentation (GPWS) system demands higher capacity of disambiguation techniques particularly, because it has functions such as allowing users to create their own dictionaries dynamically and employing multiple user's dictionaries to word segmentation. Based on inspection of the distributions and characteristics of ambiguity fragments (especially overlapping ambiguity fragments) in large-scale real corpus, an improved forward maximum match algorithm for ambiguity fragment detection, as well as a practical “rules + exceptions” disambiguation strategy, are proposed in this paper. An exhaustive extraction has been made of the overlapping ambiguity sections (about 2.4 million occurrences) from a People's Daily corpus of 100 million characters (234MB approximately), and open-ended experiments on the above strategy randomly were carried out, which achieved accuracy average of 99%.

       

    /

    返回文章
    返回