As social and environmental issues become increasingly serious, both fuel costs and environmental impacts should be considered in the cogeneration process. In recent years, combined heat and power economic emission dispatch (CHPEED) has become a crucial optimization problem in power system management. In this paper, a novel reinforcement-learning-based multi-objective differential evolution (RLMODE) algorithm is suggested to deal with the CHPEED problem considering large-scale systems. In RLMODE, a Q-learning-based technique is adopted to automatically adjust the control parameters of the multi-objective algorithm. Specifically, the Pareto domination relationship between the offspring solution and the parent solution is used to determine the action reward, and the most-suitable algorithm parameter values for the environment model are adjusted through the Q-learning process. The proposed RLMODE was applied to solve four CHPEED problems: 5, 7, 100, and 140 generating units. The simulation results showed that, compared with four well-established multi-objective algorithms, the RLMODE algorithm achieved the smallest cost and smallest emission values for all four CHPEED problems. In addition, the RLMODE algorithm acquired better Pareto-optimal frontiers in terms of convergence and diversity. The superiority of RLMODE was particularly significant for two large-scale CHPEED problems.