At the moment, there are a considerable number of different automated machine learning frameworks. They are often use predefined pipelines and choose the best one among them. However, searching for optimal pipelines can be improved by using methods that generate pipelines step by step. The paper introduces an approach to generate ensemble pipelines using policy-based reinforcement learning. Approach consists of pipeline, environment, state, action and reward representations. This approach was successfully integrated into automatic machine learning framework. The generated pipelines were tested by comparing a baseline model using OpenML datasets, and the proposed approach demonstrated high efficiency, even surpassing the metrics for some datasets. This research has the potential to enhance the existing pipeline generation methods.