Error Correction for Handwritten Mathematical Expression Recognition by Pen and Speech

As recognition-based interfaces are error prone, it is important to provide a natural and efficient error correction method for these interfaces. Handwritten mathematical expressions have 2D structures, and it is challenging to recognize them and correct their recognition errors. In this paper, a multimodal error correction technique is introduced for handwritten mathematical expression recognition. It allows users to correct errors by pen and speech. Symbol segmentation errors could be corrected by pen. Symbol recognition errors and structure analysis errors could be corrected by pen or by pen and speech. Users could firstly select an error by pen and then tell the corresponding mathematical term or mathematical symbol by speech. The key of the proposed technique is a multimodal fusion algorithm which fuses handwriting and speech recognition results. The input to the fusion algorithm is the speech and the symbols selected by pen. According to whether the speech input is a mathematical term or a mathematical symbol’s name, the algorithm chooses a specific fusion method to adjust the handwritten expression and get the most likely result. Evaluation shows that the proposed multimodal error correction technique is effective, and it can help users to correct errors in mathematical expression recognition more efficiently than the unimodal pen-based error correction technique.