•The evolution of self-organization under extreme frictional conditions has been studied.•Comprehensive characterization of the tribo-films was made using various surface analytical techniques.•During the running-in stage, mullite tribo-ceramics predominate on the surface of the nano-multilayer coating, establishing a functional hierarchy within the layer of tribo-films.•It is possible to control tribo-film evolution during self-organization by means of an increase in structural complexity and the non-equilibrium state of the surface engineered layer. The evolution of the self-organization process where dissipative structures are formed under the extreme frictional conditions associated with high performance dry machining of hardened steels has been studied in detail. The emphasis was on the progressive studies of surface transformations within multilayer and monolayer TiAlCrSiYN-based PVD coatings during the running-in stage of wear when self-organization process occurs. The coating layer was characterized by high resolution electron energy-loss spectroscopy (HREELS). It is shown that the nano-multilayer coating possesses higher non-equilibrium structure in comparison to the monolayer. Comprehensive studies of the tribo-films (dissipative structures) formed on the friction surface were made using a number of advanced surface characterization techniques such as X-ray photoelectron spectroscopy (XPS) and X-ray absorption near edge structure (XANES). The data obtained for the tribo-films was combined with the detailed TEM studies of the structural and phase transformations within the underlying coating layer. This data was related to the micro-mechanical characteristics of the coating layer and its wear resistance. It was demonstrated that the evolution of the self-organization process is strongly controlled by the characteristics of the tribo-films formed at different stages of the wear process. Within running-in stage (after length of cut of 15m) fully protective mullite tribo-films predominantly form on the surface of nano-multilayer coating, establishing a functional hierarchy within the layer of tribo-films. This reduces entropy production during friction and leads to significant surface damage reduction and wear rate stabilization. In contrast, in a monolayer coating with a lower structural complexity, a variety of protective and non-protective tribo-films form during the running-in stage, which cannot fully protect the surface. Therefore the wear rate on the monolayer is not stabilized and its wear resistance is lower. The results obtained show that it is possible to control tribo-films evolution during self-organization by means of increase in structural complexity and the non-equilibrium state of the surface engineered layer with simultaneous tuning of its integrative behaviour.