Maritime safety and environmental protection are fundamental considerations within the shipping industry. In this context, port state control (PSC) inspection is globally implemented by port authorities as a mechanism to enforce both maritime safety standards and environmental regulations. This study proposes an innovative optimization framework based on machine learning (ML) and operations research models for high-risk vessel selection, aiming to maximize the efficiency and effectiveness of PSC inspection. The essence of the optimization framework is to accurately rank all ships with respect to their risk levels predicted by ML models. The loss functions of the tailored ML models follow a “smart predict then optimize” (SPO) criterion named cumulative detected deficiency number (CDDN), which is motivated by the characteristics of the decision problem. This inventive measurement transforms the assessment of ranking accuracy to the area of the segmented histogram of the recognized deficiency number, which bypasses the computationally intensive training step of rankings and is easy to compute. Following this, three types of decision tree (DT) models are developed, which differ from each other in the varying integration levels of CDDN. Particularly, we rigorously prove that one integration method yields a tree structure identical to that of traditional DT models. The proposed models are validated and compared with the traditional DT model on different scales of instances from real inspection records at the Hong Kong port. The experiment results indicate that our tailored DT models improve the ship selection efficiency significantly when the decision is complex, i.e., when we need to optimize the selection of a small number of ships for inspection from a large number of foreign visiting ships. Moreover, we also extensively discuss when and why the SPO framework offers a superior decision to optimize vessel selection. •Shipping safety highlight port state control to identify substandard vessels.•A smart predict-then-optimize framework of vessel selection optimization.•Ranking-based data-driven model generated by the optimization model structure.•Extensive numerical experiments based on real data at Hong Kong port.