•We check that CNNs accept performance improvement techniques in MNIST.•These techniques reduce the advantage of CNNs over SDAE classifiers.•Adding a SDAE classifier over an improved CNN ensemble improves results.•The above approaches provide MNIST classification records.•We indicate some research lines emerging from this work. In this note, we follow two directions to improve the performance of CNN classifiers. The first is to apply to CNN units the same improvement techniques that we have successfully used with Stacked Denoising Auto-Encoder classifiers. This leads to obtain a new performance record when classifying MNIST digits. The second consists of applying a Stacked Denoising Auto-Encoder classifier to the output of the best of the previous designs, trying to take advantage of the limitations of CNN architectures. An even better classification record is obtained for MNIST. The above results permit to conclude that combining improvement techniques and stacking deep machines of different nature can be useful to better solve other real-world problems.