Exceptional points (EPs) are special parameter values of a non-Hermitian eigenvalue problem where eigenfunctions corresponding to a multiple eigenvalue coalesce. In optics, EPs are associated with a number of counter-intuitive wave phenomena, and have potential applications in lasing, sensing, mode conversion and spontaneous emission processes. For open photonic structures, resonant states are complex-frequency solutions of the Maxwell's equations with outgoing radiation conditions. For open dielectric structures without material gain or loss, the eigenvalue problem for resonant states can have EPs, since it is non-Hermitian due to radiation losses. For applications in nanophotonics, it is important to understand EPs for resonant states on small finite dielectric structures consisting of conventional dielectric materials. To achieve this objective, we study EPs of resonant states on finite sets of parallel infinitely-long circular dielectric cylinders with subwavelength radii. For systems with two, three and four cylinders, we develop an efficient numerical method for computing EPs, present examples for second and third order EPs, and highlight their topological features. Our work provides insight to understanding EPs on more complicated photonic structures, and can be used as a simple platform to explore applications of EPs.