Abstract: In Markov decision process, a single agent could find the optimal policy of the problem by reinforcement learning. But the model of the MDP doesn’t adapt to the multi-agent system. And the minmax-Q learning algorithm could only solve the problem of zero-sum Markov games. In this paper, the non-zero-sum Markov games are adopted as a framework for multi-agent reinforcement learning, and the learning model and learning algorithms of the metagame reinforcement learning are brought forward. It is proved that this metagame-Q algorithms must converge at the most optimal value of the non-zero-game Markov games.