Q-Learning Control for Cart-Pole System in CoppeliaSim

This technical report presents a study on the development of a Q-learning-based control algorithm for a cart-pole system modeled in CoppeliaSim, integrated with MATLAB. The research systematically investigates various design factors that influence the learning effectiveness of the controller. Key findings indicate that the design of the reward function, particularly the use of a state-dependent shaped reward, significantly enhances convergence efficiency compared to fixed reward strategies. Additionally, the report analyzes the impact of state-space discretization granularity and the sensitivity of hyperparameters such as the learning rate and discount factor on controller performance. By utilizing a high-fidelity virtual model, the study bridges the gap between numerical simulations and real-world applications, providing empirical insights and robust design principles for implementing classical reinforcement learning in complex control scenarios. The findings underscore the importance of reward shaping and parameter tuning in achieving effective control in robotic simulations.