Image classification finds wide applications in face recognition, cancer detection, and many more. However, the classifier models such as convolutional neural networks (CNN) used in safety-critical systems like self-driving cars, medical image diagnosis, etc., demand differential treatment of classification mistakes. This is because certain misclassifications may have grave impacts whereas certain others might have only minor adverse impacts. The idea of the severity of misclassification can be associated with the semantic information possessed by the classes. The objective of this work is to minimize the severity of misclassification which has high relevance in safety-critical applications. To achieve differential treatment of mistakes semantic information is incorporated into CNNs with a custom cross-entropy loss function. The semantic similarity information enables CNNs to distinguish between semantically similar and dissimilar images. We propose a model to build custom cross-entropy loss functions that penalise the classification mistakes according to their severity. In addition, we propose two novel methods to build the semantic similarity matrix between the classes to feed into the custom loss function. The first method used directed acyclic hierarchies from the data, and the second method uses a confusion matrix to build similarity matrices. The results showed that the proposed solutions were found to reduce the severity of misclassification and also achieve an overall improvement in the classification accuracy of the model. The First model achieved a classification accuracy of 78.8 ± 0.3% and the second model achieved an accuracy of 79.2 ± 0.3%. The First model reduced the severity of misclassification by a degree of 2 Superclass Misclassification Error (SCME) and the second model reduced it by 1.5 SCME than the base CNN model. In addition, we have also compared our methods to the recent related works in this domain to highlight the benefits of the proposed methods.