Abstract: As the core of the multimodal interactive system, the aim of multimodal integration is to synthesize the information coming from different modality and to accurately deliver the user’s intention to the computer. Concentrating on the problem of integration and by analyzing the characters of the multimodal input information, a hierarchical model guided by task and the corresponding algorithm are put forward in this paper. The model and algorithm decrease the degree of modalities’ dependence on time, and the integration mechanism is independent of the host system and interactive devices and has good generality.