Thin coal seam mining is a development direction to solve the problem of energy supply at this stage, which cannot be realized by small working space, low automation, and drilling deviation. In this paper, a nonlinear adaptive backstepping controller based on a disturbance observer is proposed and used on a coal auger for position tracking control to achieve directional drilling. Firstly, a nonlinear dynamic model for the deflection control mechanism is built with the consideration of parameter uncertainties and external disturbances. Then, the parameter uncertainty and external disturbance are regarded as a system compound disturbance. Furthermore, a disturbance observer is designed to estimate the system compound disturbance and a nonlinear adaptive backstepping controller was proposed to compensate the system compound disturbance. The upper bound of the compound disturbance, which can effectively reduce the chattering in the directional control process, cannot be obtained easily. A stability analysis of the DCM (deviation control mechanism) with the proposed controller is proved based on the Lyapunov theory. Finally, an electro-hydraulic servo displacement control experimental system with matlab xPC target rapid prototyping technology and a prototype experiment system is established to verify the effectiveness of the proposed control strategy. The experimental results indicate that the proposed controller can yield more satisfactory position tracking performance, such as parameter uncertainties and external disturbances, than the conventional proportion integral derivative (PID) controller and an adaptive backstepping controller. Using the control strategy, technical breakthrough on horizontal directional drilling can be realized for thin coal seam mining.