Abstract: Discovering useful subgoals and creating options while learning is important to improve the learning performance of agent in hierarchical reinforcement learning. A new subgoal discovery algorithm based on k-cluster is proposed which can extract subgoals from the set of trajectories collected online by clustering them. The results of experiment show that the algorithm can find all sobgoals quite efficiently, and the hierarchical reinforcement learning with k-cluster algorithm has better learning performance than Q-learning and hierarchical reinforcement learning with diverse density algorithm.