Abstract: Due to videos temporally varying nature, it is difficult for user to capture video main content at a glance. Constructing a hypervideo always depends on complex computer vision techniques. It leads to high cognitive load and complicated operations to users. Traditional operations based on time bar or video frame sequence provide ways to interact with video, but it still inconvenient for users to edit or navigate videos according to video semantic flexibly. In order to solve these problems, a novel method based on sketch is proposed with the purpose of providing natural and efficient interaction for hypervideo construction. Due to the properties of abstract and vague, sketch is very suitable to describe and enrich video content, and can bridge the gap between low level features and high semantics. In this paper, we analyze complex semantic relation between different video resources and divide sketch into two types according to different level of video semantic to assist hypervideo construction. Two key problems about sketch creation are resolved. One is sketch similarity matching, the other is fusion of sketch and video. Finally, an example of application is given to illustrate how to realize efficient hypervideo construction by sketch based video semantic representation.