•Changes in composition and C and N cycling functions of microbes were studied.•Microbial communities tended to shift from r- to K-strategists, both at the phylum and genus levels.•Bacterial communities were influenced by available P contents and soil C:N ratios.•Fungal communities were affected by ammonium N contents and root biomass.•Ammonium N and activated C contents affected microbial community function. Vegetation succession is one of the most important factors driving changes in microorganisms. It is unclear, however, how the microbial composition and the potential function of C and N cycling change with forest secondary succession. Using soil metagenomic sequencing methods, we studied these changes in bacterial and fungal communities with secondary succession from cropland to a Quercus liaotungensis forest over approximately 120 years on the Loess Plateau of China. The results revealed the following. (1) Soil microbial biomass C, N, and P increased significantly in topsoil (0–20 cm) with vegetation succession. (2) The abundances of bacteria increased initially and then decreased slightly, whereas an increase in fungal abundances and the ratio of fungi to bacteria was detected along a successional gradient. Microbial communities tended to shift from r- to K-strategists, both at the phylum and genus levels. (3) With vegetation succession, the abundances of C and N cycle-related potential functional genes first significantly increased and then stabilized. Among them, the relative abundances of recalcitrant C degradation-, N fixation-, and ammonification-related genes increased, whereas labile C degradation-, N reduction-, and denitrification-related genes tended to decrease. (4) Redundancy analysis indicated that bacterial communities were influenced by available phosphorus contents and soil C: N ratios, and that fungal communities were mainly affected by ammonium N contents and root biomass. (5) Predicted microbial functional genes were affected by ammonium N and activated C contents. Our study showed that with vegetation succession, microbe communities tended to shift from r- to K-strategists both at the phylum and genus levels, which increased the abundance of organisms expressing C- and N-cycle related genes.