Abstract: Artificial agents such as autonomous vehicles and healthcare robots are playing an increasingly important role in human life, and their moral issues have attracted more and more concerns. To build the ability for agents to comply with basic human ethical norms, a novel approach for training artificial moral agents is proposed based on crowdsourcing and reinforcement learning. Firstly, crowdsourcing is used to obtain sampling data sets of human behaviors, and text clustering and association analysis are used to generate plot graphs and trajectory trees, which define a basic behavior space of agents and present the sequence of behaviors. Secondly, the concept of meta-ethical behavior is proposed, which expands the behavior space of agents by summarizing similar behaviors in different scenarios, and nine kinds of meta-ethical behaviors are extracted from the Code of Daily Behavior of Middle School Students. Finally, a behavior grading mechanism and the corresponding reward and punishment function in reinforcement learning are proposed. By simulating drug purchase scenarios in human life, Q-learning algorithm and DQN (deep Q-networks) algorithm are used to complete the training experiments of moral agent respectively. Experimental results show that the trained agents can complete the expected tasks in ethical manners, which verifies the rationality and effectiveness of the above method.