In quantum key distribution (QKD), the bit error rate is used to estimate the information leakage and hence determines the amount of privacy amplification-making the final key private by shortening the key. In general, there exists a threshold of the error rate for each scheme, above which no secure key can be generated. This threshold puts a restriction on the environment noises. For example, a widely used QKD protocol, the Bennett-Brassard protocol, cannot tolerate error rates beyond 25%. A new protocol, round-robin differential phase-shifted (RRDPS) QKD, essentially removes this restriction and can in principle tolerate more environment disturbance. Here, we propose and experimentally demonstrate a passive RRDPS QKD scheme. In particular, our 500 MHz passive RRDPS QKD system is able to generate a secure key over 50 km with a bit error rate as high as 29%. This scheme should find its applications in noisy environment conditions.