Tool face adjustment in oil and gas drilling are vital issues that affect the efficiency and safety of directional drilling. The existing tool face adjustment methods mainly rely on manual real-time intervention for continuous adjustment. Affected by manual experience, the effect is unstable and the labor cost is high. With rising energy consumption, the need for intelligent directional drilling is becoming more pressing. However, establishing an autonomous adjustment approach for the tool face remains challenging because of the variety of complex downhole environments encountered during actual drilling operations. This paper proposes a model-free online learning adaptive decision strategy for cross well intelligent adjustment and stability of tool face. A reward function embedded with expert operating experience is designed to learn the directional policy from the driller's corrective actions. Further, to improve the efficiency of online learning, a priority-based experience playback algorithm is developed. A data-driven directional drilling simulation environment is proposed to realize the accurate simulation of the directional drilling process and pre-training of directional strategy. Simulations are carried out to validate the efficacy of the proposed method. The outcomes of field application suggest that the proposed strategy can achieve decision-making goals in a short period.