•Feature selection method based on Group Teaching Optimization Algorithm is proposed.•Two novel operators are developed to increase the exploration and exploitation.•The student phase is enhanced to increase the population diversity and exploitation.•The teacher allocation phase is enhanced to increase the convergence rate.•The superiority of the proposed algorithm is indicated over 30 benchmark datasets. The current study proposes a novel binary group teaching optimization algorithm with local search and chaos mapping (BGTOALC) as a wrapper-based feature selection method to solve high-dimensional feature selection problems. The local search and chaos mapping enhance the performance of the proposed algorithm. Also, two novel binary operators called Binary Teacher Phase Good Group (BTPGG) and Binary Teacher Phase Bad Group (BTPBG) are applied to the teacher’s phase for increasing the exploration and exploitation of the algorithm. Moreover, a new Binary Student Opposition-Based Learning (BSOBL) operator is introduced for the student phase, using an opposition-based strategy to achieve better exploitation. Finally, the teacher allocation phase is designed in a binary manner using the new Mean Binary Select (MBS) operator to increase the algorithm’s convergence rate. Subsequently, two other binary group teaching optimization algorithms, named BGTOAV and BGTOAS, are developed utilizing the S-shaped and V-shaped transfer functions to compare their performance with the BGTOALC algorithm. The proposed approaches are compared to other state-of-the-art binary algorithms on 30 datasets with different dimensions. Different experiments prove that the BGTOALC method outperforms the previous methods in terms of reducing the number of selected features and increasing the accuracy of the machine learning algorithm. Eventually, statistical analyses indicate the superiority of the BGTOALC method in terms of efficiency and convergence rate against other binary metaheuristic algorithms.