Failure mode and effects analysis (FMEA) is a widely used reliability analysis tool for identifying and eliminating known or potential failures in system, design, and process. In traditional FMEA, failure modes are evaluated by FMEA team members with respect to three risk factors: severity (S), occurrence (O), and detectability (D), and ranked via their risk priority number (RPN), which is obtained by multiplying the crisp values of S, O, and D. However, traditional RPN has been considerably criticized due to the following shortcomings: not considering the different weights of risk factors; the identical value of RPN for different combinations of S, O, and D; the diversity and uncertainty of evaluation information given by FMEA team members and without considering the dependence among different failure modes. Although significant efforts have been made in FMEA literatures to overcome these shortcomings, there are still some deficiencies. In this paper, a new risk priority model is presented for FMEA by using the house of reliability (HoR)-based rough VIsekriterijumska optimizacija i KOmpromisno Resenje (VIKOR) approach. In the proposed model, the HoR is introduced to identify the dependence among different failure modes and the link between failure modes and the risk factors of O, D and the subcriteria of S. Rough number is introduced to manipulate the subjectivity and vagueness in decision making and VIKOR approach is used to determine the risk priority order of failure modes in a comprehensive way. Finally, an illustrative case in transmission system of a vertical machining center has demonstrated the effectiveness and practicality of the proposed model.