The properties of stellar clumps in star-forming galaxies and their evolution over the redshift range 2 ≲ z ≲ 6 are presented and discussed in the context of the build-up of massive galaxies at early cosmic times. We focused on galaxies with spectroscopic redshifts from the VIMOS Ultra Deep Survey (VUDS) and stellar masses log 10(M⋆/M⊙) > −0.204 × (z−4.5) + 9.35. We analyzed HST-ACS images to identify clumps within a 20 kpc radius using a method taking into account differential surface brightness dimming and luminosity evolution with redshift. We find that the population of galaxies with more than one clump is dominated by galaxies with two clumps, representing ~21–25% of the population, while the fraction of galaxies with three, or four and more, clumps is 8–11% and 7–9%, respectively. The fraction of clumpy galaxies is in the range ~35–55% over 2 < z < 6, increasing at higher redshifts, indicating that the fraction of irregular galaxies remains high up to the highest redshifts. The large and bright clumps (M⋆ ~ 109 up to ~1010 M⊙) are found to reside predominantly in galaxies with two clumps. Smaller and lower luminosity clumps (M⋆ < 109 M⊙) are found in galaxies with three clumps or more. We interpret these results as evidence for two different modes of clump formation working in parallel. The small low luminosity clumps are likely the result of disk fragmentation, with violent disk instabilities (VDI) forming several long-lived clumps in-situ as suggested from simulations. A fraction of these clumps is also likely coming from minor mergers as confirmed from spectroscopy in several cases. The clumps in the dominating population of galaxies with two clumps are significantly more massive and have properties akin to those in galaxy pairs undergoing massive merging observed at similar redshifts; they appear as more massive than the most massive clumps observed in numerical simulations of disks with VDI. We infer from these properties that the bright and large clumps are most likely the result of major mergers bringing-in ex situ matter onto a galaxy, and we derive a high major merger fraction of ~20%. The diversity of clump properties therefore suggests that the assembly of star-forming galaxies at z ~ 2–6 proceeds from several different dissipative processes including an important contribution from major and minor mergers.