We present a novel approach for computing the surface roughness-limited thermal conductivity of silicon nanowires with diameter D<100 nm. A frequency-dependent phonon scattering rate is computed from perturbation theory and related to a description of the surface through the root-mean-square roughness height Delta and autocovariance length L. Using a full phonon dispersion relation, we find a quadratic dependence of thermal conductivity on diameter and roughness as (D/Delta)(2). Computed results show excellent agreement with experimental data for a wide diameter and temperature range (25-350 K), and successfully predict the extraordinarily low thermal conductivity of 2 W m(-1) K-1 at room temperature in rough-etched 50 nm silicon nanowires.