Oil palm cultivation stands as a crucial industry in Indonesia, significantly contributing to the nation’s economy by generating employment opportunities and fostering social welfare for communities residing near plantations. Despite its economic importance, oil palm plantations face various challenges, with one prominent issue being the infestation of nettle caterpillar pests. These pests cause leaf skeletonization, resulting in a staggering 36% reduction in oil palm productivity over a two-year period. This paper explores diverse strategies for pest management in oil palm plantations, encompassing biological control through the stimulation of natural predators, mechanical control involving the collection and incineration of cocoons, and chemical control through pesticide application. The research introduces a predator–prey mathematical model for oil palm plantation pests, where the leaf area serves as the primary food source for caterpillars, acting as prey. Through dynamic model analysis, four equilibrium points are identified, with interconnected conditions dictating their existence and stability. These conditions are visually represented in a bifurcation plane, providing concise information. The study further includes bifurcation diagrams of equilibrium points to elucidate the influence of each parameter on pests, predators, and the leaf area of oil palm plants. Additionally, sensitivity analysis of the stable interior equilibrium point is conducted to understand the impact of individual parameters. The paper extends its investigation to optimal control strategies, evaluating six scenarios categorized into two population conditions: with predators and without predators. Within each population condition, three control strategies are considered—chemical control only, mechanical control only, and a combination of chemical and mechanical control. Simulation results from the optimal control study reveal that the presence of natural predators emerges as a pivotal strategy in effectively managing nettle caterpillars. Notably, the control of resistant pests through pupa incineration has a substantial impact on reducing the caterpillar population in subsequent life cycles. •In current study, the MELP-S-B predator-prey model is proposed for managing nettle caterpillar pests in oil palm plantations involving the control measures.•The inclusion of prey populations, specifically oil palm leaves, in this predator-prey model constitutes the most fundamental novelty of this study.•The main objective of this research is to determine the dynamics of pests in plantations and avoid economic losses due to pests in oil palm plantations.•In-depth analysis of system dynamics around four equilibrium points conducted. Sensitivity analysis is carried out to measure the influence of parameters at the interior point.•Optimal control study conducted to manage pest abundance in oil palm plantations under varying predator presence. Numerical simulations of six strategies, blending mechanical and chemical measures, offer insights into effective pest control combinations.