The decline in honey bee populations and the labor-intensive nature of manual hive inspection have sparked interest in automating beehive monitoring. Acoustic monitoring has emerged as a versatile modality for this purpose. Beehive audio has proven useful in detecting events such as swarming, queen absence, parasite infestation, and assessing hive strength. However, external factors such as rain, wind, traffic noise, and background voices of beekeepers can degrade the quality of recorded bee audio and impact the effectiveness of beehive monitoring. In this study, we focus on the detection of background beekeepers speech, which has shown to significantly degrade audio-based hive monitoring performance. In particular, we propose the use of universal audio representations extracted from self-supervised models to predict whether a given audio frame contains speech or only bee sounds. Experimental results on the NU-Hive dataset show that the proposed methodology applied on top of the WavLM Base+ model can outperform state-of-the-art voice activity detectors by a relative improvement of 19% and 16% in the ROC-AUC and F1-score metrics, respectively. Furthermore, we show that audio representations can be reliably used for unsupervised anomaly detection.