Temporal modeling and calculus are the fundamental logic problems in temporal information system. The subtraction operation between two temporal points with any granularity is the basis of the temporal assertion, which is hard to deal with. From the point view of granular computing, the semantics and properties of temporal granularities are discussed according to the temporal partition. Temporal primitives are characterized based on the temporal granularities. Furthermore, the method of the granularity subtraction operation is proposed based on the mapping and conversion of different granularity hierarchies. The temporal points could be mapped to the countable set with any temporal granularity, and the subtraction operation between two temporal points could be converted to the subtraction operation between two natural numbers. The equality between temporal domain T and natural set N and continuity of the mapping are discussed and proved, which supports that the proposed method is correct, and the effect of the flexible temporal granularity is overcome. Finally, an algorithm for temporal granularity converting functions is proposed in the context of the calendar system. The proposed method could be adapted to the subtraction operation between two temporal points from any irregular temporal points set or userdefined temporal points set.