•EDL optimally assign different number of dictionary items to each class.•EDL learns different number of dictionary items for different classes.•Number of sharing/discriminative dictionary items is set optimally by EDL.•Proposed method (EDL) learns correct number of shared/discriminative items. In this paper, a new discriminative dictionary learning algorithm is introduced. An entropy based criterion is embedded into the objective function to enforce a proper structure for the dictionary items when decomposing signals of different classes. The proposed criterion influences the dictionary items to participate in the decomposition of a smaller number of classes as possible. Unlike the other methods, columns of the dictionary are not restricted to have pre-assigned labels and they are free to be representative of any class or to share features of several classes. The number of shared and discriminative items along with the number of dictionary items for each specific class is learned dynamically during the optimization process, depending on the complexity of the classification task and the distribution of different classes. The experimental results demonstrate that the proposed entropy based dictionary learning (EDL) algorithm outperforms other discriminative dictionary learning methods using several real-world image datasets.