The process of elastic scattering of photons (light-by-light scattering) attracts a lot of attention in the last years. It goes via the loop and all charged particles contributes to its cross section. Theoretical prediction of the cross section have existed for more than half a century, there are various accelerators with real and virtual photons, but only recently this process was discovered at the LHC in ion-ion collisions. Hopes for further study of this process are associated with high-luminosity $e^+e^-$ colliders (SuperKEKb, FCC, ILC, CLIC) and gamma-gamma colliders based on the Compton scattering of laser photons on electrons. In this article, we show that there is very serious background processes when the final pair of photons is created in the annihilation of real and virtual $e^+$ and $e^-$, and the remaining products fly away from the detector at small angles.