Winter mortality is one of the main causes of beehive loss. However, very limited tools can be used by beekeepers to identify the high-risk colonies at an early stage. In this study, we propose a multi-modal sensor (audio, humidity, temperature) based system to predict the beehive winter survivability. More specifically, we first propose a multi-modal feature set, which is shown to be highly correlated with winter survival rate, and develop a machine learning model to further detect the hives that are less likely to survive the winter. Our top-performing model achieves an AUC-ROC score of 0.730 based on one-year-long data collected from 45 hives located in two different apiaries in Canada. Our findings show the feasibility of capturing high-risk hives at the early stage using multi-modal sensor data. Furthermore, we highlight the importance of bee audio in measuring survivability over other more widely-used modalities. Future study will focus on improving the generalizability of the prediction model.