Smart agriculture and Internet of Things (IoT) technologies have become the key points for many intelligent decision-making applications to support agricultural experts and farmers, especially for crop pest management and control. In this work, we present an IoT-based sound detection model for identifying red palm weevil (RPW) larvae to protect date palm trees at the early stage of infestation. The proposed detection system is mainly based on a modified mixed depthwise convolutional network (MixConvNet) as a recent deep learning classifier. The public TreeVibes dataset, which contains short audio recordings of feeding and/or moving RPWs, was successfully tested and assessed with the proposed MixConvNet classifier. There were 146 and 351 specimens of infested and clean sounds examined, respectively. The classification results showed that our proposed MixConvNet is efficient and superior to other deep learning classifiers, such as Xception and residual network (Resnet) models in previous related studies, obtaining the best accuracy score of 97.38. Moreover, the MixConvNet classifier is capable of identifying RPW infestation cases precisely with a high accuracy value of 95.90% ± 1.46, using 10-fold cross-validation. Therefore, practical implementation of our proposed IoT-enabled early sound detection system of RPWs is considered the future milestone of this study.