ABSTRACT Self‐learning process is an important factor that enables learners to improve their own educational experiences when they are away of face‐to‐face interactions with the teacher. A well‐designed self‐learning activity process supports both learners and teachers to achieve educational objectives rapidly. Because of this, there has always been a remarkable trend on developing alternative self‐learning approaches. In this context, this study is based on two essential objectives. Firstly, it aims to introduce an intelligent software system, which optimizes and improves computer engineering students’ self‐learning processes. Secondly, it aims to improve computer engineering students’ self‐learning during the courses. As general, the software system introduced here evaluates students’ intelligence levels according to the Theory of Multiple Intelligences and supports their learning via accurately chosen materials provided over the software interface. The evaluation mechanism of the system is based on a hybrid Artificial Intelligence approach formed by an Artificial Neural Network, and an optimization algorithm called as Vortex Optimization Algorithm (VOA). The system is usable for especially technical courses taught at computer engineering departments of universities and makes it easier to teach subjects. For having idea about success of the system, it has been tested with students and positive results on optimizing and improving self‐learning have been obtained. Additionally, also a technical evaluation has been done previously, in order to see if the VOA is a good choice to be used in the system. It can be said that the whole obtained results encourage the authors to continue to future works. © 2017 Wiley Periodicals, Inc. Comput Appl Eng Educ 25:142–156, 2017; View this article online at wileyonlinelibrary.com/journal/cae; DOI 10.1002/cae.21787