Tree seedlings planted in abandoned agricultural fields interact with herb communities through competition, tolerance, and facilitation. In addition, they are subject to herbivory by small mammals, deer or invertebrates. To increase the success of forest restoration in abandoned fields and reduce management costs, we should determine which species are tolerant to or facilitated by herbaceous vegetation and those which require protection from competition and predation. Eight native tree species were planted in plots covered by herbaceous vegetation, plots where herbaceous vegetation was removed, and plots where seedlings were surrounded by an organic mulch mat. Half of the seedlings were protected against small mammal damage. Results showed that two non-pioneer and moderately shade-tolerant species (yellow birch and red oak) were inhibited by herbaceous vegetation. Birch species were particularly affected by small mammal predation. No effects of predation or herbaceous competition were observed for conifer species. Rather, herbaceous vegetation had a positive effect on the survival and the height growth of tamarack (Larix laricina). None of the tested herb communities had a stronger competitive effect on tree growth than another. Restoration of abandoned fields using multi-tree species should be designed at the seedling scale rather than at the site scale to account for different tree responses to predation and competition as well as variable site conditions. An approach resembling precision agriculture is proposed to lower costs and any potential negative impact of more intensive vegetation management interventions.