Exciton-polaritons are hybrid light-matter quasiparticles formed by strongly interacting photons and excitons (electron-hole pairs) in semiconductor microcavities. They have emerged as a robust solid-state platform for next-generation optoelectronic applications as well as for fundamental studies of quantum many-body physics. Importantly, exciton-polaritons are a profoundly open (that is, non-Hermitian) quantum system, which requires constant pumping of energy and continuously decays, releasing coherent radiation. Thus, the exciton-polaritons always exist in a balanced potential landscape of gain and loss. However, the inherent non-Hermitian nature of this potential has so far been largely ignored in exciton-polariton physics. Here we demonstrate that non-Hermiticity dramatically modifies the structure of modes and spectral degeneracies in exciton-polariton systems, and, therefore, will affect their quantum transport, localization and dynamical properties. Using a spatially structured optical pump, we create a chaotic exciton-polariton billiard--a two-dimensional area enclosed by a curved potential barrier. Eigenmodes of this billiard exhibit multiple non-Hermitian spectral degeneracies, known as exceptional points. Such points can cause remarkable wave phenomena, such as unidirectional transport, anomalous lasing/absorption and chiral modes. By varying parameters of the billiard, we observe crossing and anti-crossing of energy levels and reveal the non-trivial topological modal structure exclusive to non-Hermitian systems. We also observe mode switching and a topological Berry phase for a parameter loop encircling the exceptional point. Our findings pave the way to studies of non-Hermitian quantum dynamics of exciton-polaritons, which may uncover novel operating principles for polariton-based devices.