ISSN 1000-1239 CN 11-1777/TP

• 软件技术 •

基于分布内存的层次短语机器翻译并行化算法

1. (计算机软件新技术国家重点实验室(南京大学) 南京 210023) (江苏省软件新技术与产业化协同创新中心 南京 210023) (bhoppi@outlook.com)
• 出版日期: 2014-12-01

Parallel Algorithm for Hierarchical Phrase Machine Translation Based on Distributed Memory Storage

Zhao Bo, Huang Shujian, Dai Xinyu, Yuan Chunfeng, Huang Yihua

1. (State Key Laboratory for Novel Software Technology (Nanjing University), Nanjing 210023) (Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing 210023)
• Online: 2014-12-01

Abstract: In recent years, in order to improve the accuracy of SMT (statistical machine translation) system, massive corpus has been widely applied to train language and translation models. As the scale of the language and translation models increase, computing performance becomes a challenging issue for SMT, which makes existing single-machine translation algorithms and systems difficult to complete the computation in time, especially when dealing with online translation. In order to overcome the limitations of single-machine translation decoding algorithm and improve the computing performance of large-scale SMT toward a practical online translation system, this paper proposes a distributed and parallel translation decoding algorithm and framework by adopting a distributed storage and parallel query mechanism upon both the language and translation models. We develop a hierarchical phrase parallel decoder by using a distributed memory database to store and query large-scale translation and language model tables. To further improve the speed of parallel decoding, we also make three additional optimizations: 1) Transform the synchronous rules in translation model table and the Trie data structure of language model table into a Hash indexed key-value structure for use in the distributed memory database; 2) Modify the cube-pruning algorithm to make it suitable for batch query; 3) Adopt and optimize the batch query for language model and translation model tables to reduce the network overhead. Our implemented algorithm can achieve fast decoding of SMT based on large-scale corpus and provide excellent scalability. Experimental results show that, compared with the single-machine decoder, our parallel decoder can reach 2.7 times of speedup for single sentence translation and reach 11.7 times of speedup for batch translation jobs, achieving significant performance improvement.