The Internet of Things (IoT) comprises many technologies, among them is Radio Frequency Identification (RFID), which can be used to track single or multiple objects. This technology has been widely used in healthcare, supply chain, logistics, and asset tracking. However, such applications require a high level of security and privacy and are unfortunately vulnerable to various attacks and threats that need to be addressed in order for RFID-based IoT applications to reach their full potential. To this end, we propose a set of security and privacy guidelines for RFID, supported by modelling guidelines, mitigations, and the attack vectors cohesively. We compare to the state of the art and point out their shortcomings on known guidelines and reason to address these in our model. The overall methodology is as follows: (i) identify the security and privacy guideline features, (ii) highlight the security goals for RFID-based IoT applications, (iii) analyze the features in relation to RFID industrial standards, and relate them to security goals, (iv) summarize attacks and threats against RFID applications and correlate them with violated security goals, (v) derive a set of security and privacy guidelines for RFID applications in accordance with security and privacy by design frameworks. We also describe our derived guidelines in connection with the involved stakeholders, and (vi) outline the existing mitigation strategies to implement our proposed guidelines. Finally, we describe the main limitations of our work that should be investigated in the future and identify the multiple challenges that concern current security strategies.