Convolutional neural networks (CNNs) have been widely used in image recognition and processing tasks. Memristor-based CNNs accumulate the advantages of emerging memristive devices, such as nanometer critical dimensions, low power consumption, and functional similarity to biological synapses. Most studies on memristor-based CNNs use either software models of memristors for simulation analysis or full hardware CNN realization. Here, we propose a hybrid CNN, consisting of a hardware fixed pre-trained and explainable feature extractor and a trainable software classifier. The hardware part was realized on passive crossbar arrays of memristors based on nanocomposite (Co-Fe-B)x(LiNbO3)100−x structures. The constructed 2-kernel CNN was able to classify the binarized Fashion-MNIST dataset with ~ 84% accuracy. The performance of the hybrid CNN is comparable to the other reported memristor-based systems, while the number of trainable parameters for the hybrid CNN is substantially lower. Moreover, the hybrid CNN is robust to the variations in the memristive characteristics: dispersion of 20% leads to only a 3% accuracy decrease. The obtained results pave the way for the efficient and reliable realization of neural networks based on partially unreliable analog elements.