Recent studies have made tremendous progress in neural style transfer (NST) and various methods have been advanced. However, evaluating and improving the stylization quality remain two important open challenges. Committed to these two aspects, in this paper, we first decompose the quality of style transfer into three quantifiable factors, i.e., the content fidelity (CF), global effects (GE) and local patterns (LP). Then, two novel approaches are further presented for exploiting these factors to improve the stylization quality. The first, named cascade style transfer (CST), utilizes the factors to guide the cascade combination of existing NST methods to absorb their merits and avoid their own shortcomings. The second, dubbed multi-objective network (MO-Net), directly optimizes these factors to balance their performance and achieves more harmonious stylized results. Extensive experiments demonstrate the effectiveness and superiority of our proposed factors and methods. •Three quantifiable factors to evaluate the style transfer quality.•Cascade style transfer under the guidance of our factors to improve the quality.•A multi-objective network directly optimizes our factors to improve the quality.