Using a multi-sector trade model with heterogeneous firms, this paper examines how the high-speed railway (HSR) results in changes in market access, and affects firms’ productivity. The model shows that with the opening of HSR in cities and decrease in trade costs, firm productivity increases through “competition effect” (proxied by consumer market access, CMA), but decrease through “scale expansion effect” (proxied by firm market access, FMA). To distinguish between these two competing effects empirically, we manually compile data of 110 prefecture cities which are sequentially connected with HSR during 2007–2013, and match these cities with China Industrial Firms Database during same period. We find that firm productivity increase by 0.0318 percentages due to one percentage increase in CMA, and decrease by 0.0552 percentages due to one percentage increase in FMA. We find significant heterogeneity in the effect of HSR opening on firm productivity in that the effect is the largest in the central area and mostly insignificant in the west; and the effect turns out the largest for technology-intensive firms, less for capital-intensive firms and the least for labor-intensive firms. We also find the impact of HSR opening on firm revenue, output, decision of entry or exit, as well as on regional productivity. Further analyses suggest that the effect of scale expansion dominates in the period after 2009 and hinders firm productivity, which explains the puzzling divergence of rising GDP and decreasing average TFP in that period.