Aims.We study the efficiency and reliability of cluster mass estimators that are based on the projected phase-space distribution of galaxies in a cluster region. Methods.We analyse a data-set of 62 clusters extracted from a concordance ΛCDM cosmological hydrodynamical simulation. We consider both dark matter (DM) particles and simulated galaxies as tracers of the clusters gravitational potential. Two cluster mass estimators are considered: the virial mass estimator, corrected for the surface-pressure term, and a mass estimator (that we call $M_{\sigma}$) based entirely on the velocity dispersion estimate of the cluster. In order to simulate observations, galaxies (or DM particles) are first selected in cylinders of given radius (from 0.5 to 1.5h-1 Mpc) and $\simeq$$200h^{-1}$ Mpc length. Cluster members are then identified by applying a suitable interloper removal algorithm. Results.The virial mass estimator overestimates the true mass by $\simeq$10% on average, for sample sizes of $\ga$60 cluster members. For similar sample sizes, $M_{\sigma}$ underestimates the true mass by $\simeq$15%, on average. For smaller sample sizes, the bias of the virial mass estimator substantially increases, while the $M_{\sigma}$ estimator becomes essentially unbiased. The dispersion of both mass estimates increases by a factor ~2 as the number of cluster members decreases from ~400 to ~20.
It is possible to reduce the bias in the virial mass estimates either by removing clusters with significant evidence for subclustering or by selecting early-type galaxies, which substantially reduces the interloper contamination. Early-type galaxies cannot however be used to improve the $M_{\sigma}$ estimates since their intrinsic velocity distribution is slightly biased relative to that of the DM particles.
Radially-dependent incompleteness can drastically affect the virial mass estimates, but leaves the $M_{\sigma}$ estimates almost unaffected. Other observational effects, like centering and velocity errors and different observational apertures, have little effect on the mass estimates.