The utilization of phosphorus slag (PHS) to replace the fly ash in the construction of hydraulic projects has attracted a growing attention in China. In this study, the influence of PHS fineness and content on cement hydration, mechanical strength, permeability as well as the pore structure and fractal dimension ( D s ) of concrete have been discussed. The results indicate that the PHS addition retards the cement hydration and hence decreases the hydration heat within three days. The incorporation of PHS with a Blaine specific surface area of 505 m2/kg could participate in the early pozzolanic reaction and consequently offsets the retarding effect to some extent. The incorporation of 20–40 wt.% PHS declines the early strength of concrete, but this reduction effect on strength can be eliminated to some degrees by mechanically grinding the PHS. The compressive strengths of concrete added with PHS with a high fineness of 505 m2/kg (abbreviated as PHS-H) are about 16.0–20.6% higher at three days and 8.9–11.0% higher at 180 days compared that of the control concrete. The contribution of PHS-H to the pore structure refinement is more significant than that of PHS with a low fineness of 302 m2/kg (abbreviated as PHS-L) at various ages because PHS-H is of much higher reactivity and can consume more Ca(OH)2 than PHS-L which leads to a denser microstructure and a lower chloride diffusion coefficient ( D RCM ) . The incorporation of PHS decreases the D s at three days, whereas the concrete incorporated with PHS has much higher D s than that of control one at late age. The D RCM value increases with increasing the porosity and the most probable aperture, while D s has a more significant effect on D RCM than the porosity and the most probable aperture. The concrete added with 20 wt.% PHS-H exhibits the highest D s and the lowest D RCM value at long-term age among the five concrete mixtures in this work.