The concept of gauge field is a cornerstone of modern physics and the synthetic gauge field has emerged as a new way to manipulate particles in many disciplines. In optics, several schemes of Abelian synthetic gauge fields have been proposed. Here, we introduce a new platform for realizing synthetic SU(2) non-Abelian gauge fields acting on two-dimensional optical waves in a wide class of anisotropic materials and discover novel phenomena. We show that a virtual non-Abelian Lorentz force arising from material anisotropy can induce light beams to travel along Zitterbewegung trajectories even in homogeneous media. We further design an optical non-Abelian Aharonov-Bohm system which results in the exotic spin density interference effect. We can extract the Wilson loop of an arbitrary closed optical path from a series of gauge fixed points in the interference fringes. Our scheme offers a new route to study SU(2) gauge field related physics using optics.