Using a phenomenological form of the equation of state of neutron matter near the saturation density which has been previously demonstrated to be a good characterization of quantum Monte Carlo simulations, we show that currently available neutron star mass and radius measurements provide a significant constraint on the equation of state of neutron matter. At higher densities we model the equation of state by using polytropes and a quark matter model. We show that observations offer an important constraint on the strength of the three-body force in neutron matter, and thus some theoretical models of the three-body force may be ruled out by currently available astrophysical data. In addition, we obtain an estimate of the symmetry energy of nuclear matter and its slope that can be directly compared to the experiment and other theoretical calculations.