Transmissibility is a key parameter to quantify the effectiveness of a vibration isolation system. Under harmonic excitation, the force transmissibility of a linear vibration isolation system is defined as the ratio between the amplitude of the force transmitted to the host structure and the excitation force amplitude, and the displacement transmissibility is the ratio between the displacement amplitude of the payload and that of the base. For a nonlinear vibration isolation system, the force or the displacement responses usually have more frequency components than the excitation. For a harmonic excitation, the response may be periodic, quasi-periodic or chaotic. Therefore, the amplitude ratio cannot well define the transmissibility. The root-mean-square ratio of the response to the excitation is suggested to define the transmissibility. The significance of the modified transmissibility is highlighted in a nonlinear two-stage vibration isolation system consisting of two linear spring connected linear vibration isolators with two additional horizontal linear springs. Harmonic balance method (HBM) is applied to determine the responses with the fundamental and third harmonic. Numerical simulations reveal that chaos may occur in the responses. In both cases, the modified transmissibility works while the original definition cannot be applied to chaotic response. •Two definitions of nonlinear transmissibility are compared.•Two definitions are slightly different for both force and displacement transmission.•Force and displacement transmissibility with modified definition are more agreement.•Transmissibility for aperiodic response is evaluated by the modified definition.