The interplay of interactions, symmetries, and gauge fields usually leads to intriguing quantum many-body phases. To explore the nature of emerging phases, we study a quantum Rabi triangle system as an elementary building block for synthesizing an artificial magnetic field. We develop an analytical approach to study the rich phase diagram and the associated quantum criticality. Of particular interest is the emergence of a chiral-coherent phase, which breaks both the Z2 and the chiral symmetry. In this chiral phase, photons flow unidirectionally and the chirality can be tuned by the artificial gauge field, exhibiting a signature of broken time-reversal symmetry. The finite-frequency scaling analysis further confirms the associated phase transition to be in the universality class of the Dicke model. This model can simulate a broad range of physical phenomena of light-matter coupling systems, and may have an application in future developments of various quantum information technologies.