Abstract: Offline reinforcement learning leverages static, pre-collected historical datasets to learn optimal policies, effectively avoiding the high costs and potential risks caused by interacting with the online environment, which presents broad application prospects in safety-critical fields such as robotics, autonomous driving, and healthcare. However, during the learning process, offline reinforcement learning methods often suffer from instability and limited generalization ability of the learned policy due to extrapolation errors. To address this core issue, this paper proposes an offline reinforcement learning method with Bellman error-guided actor (BEGA). BEGA explicitly introduces a dynamic Bellman error constraint mechanism during the policy improvement phase, which adaptively adjusts the policy update based on the reliability of Q-value estimation. Specifically, when the Q-value estimation exhibits significant bias, BEGA can automatically prevent the policy from over-exploring uncertain regions, thereby effectively stabilizing the policy learning process and reducing the misguiding impact of policy evaluation errors on policy updates. Experimental results on the Gym-MuJoCo continuous control tasks within the D4RL benchmark show that BEGA not only improves the accuracy of Q-value estimation and the generalization ability of policy, but also exhibits superior performance compared with the baselines. Across 15 datasets covering different quality levels, BEGA achieves a performance improvement ranging from 9.7% to 36.9% over the baselines.