Abstract: In this paper, we exactly identify the sources of software communication overhead by analyzing the procedure of typical protocols in messaging layers based on active messages. We show that up to 40-55% of the software messaging costs are a direct consequence of buffer management, round trip handshake, and reception acknowledgement.We also discuss several methods to reduce these overheads and their feasibilities in implementations.