Advanced Search
    Luo Zhiyong, Song Rou. Disambiguation in a Modern Chinese General-Purpose Word Segmentation System[J]. Journal of Computer Research and Development, 2006, 43(6): 1122-1128.
    Citation: Luo Zhiyong, Song Rou. Disambiguation in a Modern Chinese General-Purpose Word Segmentation System[J]. Journal of Computer Research and Development, 2006, 43(6): 1122-1128.

    Disambiguation in a Modern Chinese General-Purpose Word Segmentation System

    • Disambiguation is one of the most important parts of segment systems in Chinese. A Chinese general-purpose word segmentation (GPWS) system demands higher capacity of disambiguation techniques particularly, because it has functions such as allowing users to create their own dictionaries dynamically and employing multiple user's dictionaries to word segmentation. Based on inspection of the distributions and characteristics of ambiguity fragments (especially overlapping ambiguity fragments) in large-scale real corpus, an improved forward maximum match algorithm for ambiguity fragment detection, as well as a practical “rules + exceptions” disambiguation strategy, are proposed in this paper. An exhaustive extraction has been made of the overlapping ambiguity sections (about 2.4 million occurrences) from a People's Daily corpus of 100 million characters (234MB approximately), and open-ended experiments on the above strategy randomly were carried out, which achieved accuracy average of 99%.
    • loading

    Catalog

      Turn off MathJax
      Article Contents

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return