Friction stir welding (FSW) is a new solid-state joining process for joining light metals, such as magnesium (Mg) alloys. However, the texture-induced softening mechanism (in stir zone, SZ) tends to cause the worsening of mechanical properties in FSW Mg alloys. In this work, multi-pass hot-rolling and subsequent annealing is used to ameliorate the FSW AZ31 alloy plate. The results show that multi-pass hot-rolling leads to the c-axes of most grains in the SZ approximately perpendicular to the normal direction (ND) transforming into these almost parallel to the ND, accompanied with multiple twin types. After subsequent annealing, twins, as nucleation sites, promote the formation of new grains with off-basal texture, resulting in the weak basal texture. Compared with the FSW plate and the initial plate, the rolled-annealed FSW plate exhibits a relatively uniform hardness distribution and a “strength-ductility” synergy, which are mainly attributed to an appropriate texture feature and a homogeneous grain structure. This work provides a new idea for the development of new high-performance FSW Mg alloys.