高级检索

    利用汉字二元语法关系解决汉语自动分词中的交集型歧义

    USING CHARACTER BIGRAM FOR AMBIGUITY RESOLUTION IN CHINESE WORD SEGMENTATION

    • 摘要: 本文提出了一种利用句内相邻字之间的互信息及t-测试差这两个统计量解决汉语自动分词中交集型歧义切分字段的方法.汉字二元语法关系(bigram)为相关计算的基础,直接从生语料库中自动习得.初步的实验结果显示,可以正确处理90.3%的交集字段

       

      Abstract: This paper presents a method of using two kinds of statistical measures,mutual information and difference of t test of adjacent characters in sentences,to deal with ambiguities in Chinese word segmentation.A Chinese character bigram matrix,derived automatically from the raw corpus,serves as a basis for the related calculations.Preliminary experiments show that 90 3% correct rate is achieved for overlapped ambiguities.

       

    /

    返回文章
    返回