We investigate above-threshold detachment induced by an ultrashort laser pulse with one or two carrier frequencies. Particular attention is devoted to the phase-dependent effects in the low- and medium-energy parts of the spectrum. In particular, we show that the photoelectron spectra can be controlled using the phases of the field components and the ellipticity as parameters. Also, using the saddle-point method, we show that the interference between various ionization pathways can also be controlled using these parameters. Depending on the value of the phases and the ellipticity, one or more saddle-point solutions can be significant in the examined part of the photoelectron spectra. If the contribution of only one saddle-point solution is dominant, the photoelectron yield is a smooth function of the photoelectron energy. On the other hand, when two or more contributions have to be taken into consideration, it is important whether the corresponding ionization times are within the same optical cycle or not. If they are, the interference pattern of these contributions is characterized by deep minima. On contrary, if the ionization times of the contributions which have to be taken into consideration are in different optical cycles, their interference would produce a rapidly oscillating spectrum. We propose a scheme for analyzing the asymmetry of the photoelectron spectra emitted in the directions characteristic for a particular driving field. We show that, by using a two-component driving pulse, it is possible to achieve a fine control on which contributions are significant and which can be neglected. This control is more advanced than the control by the linearly polarized field for which the electron dynamics is rather simple. Graphic abstract