Context. The optical properties of the second generation dust that we observe in debris disks remain quite elusive, whether it is the absorption efficiencies at millimeter wavelengths or the (un)polarized phase function at near-infrared wavelengths. Thankfully, the same particles are experiencing forces that are size dependent (e.g., radiation pressure) and, with high angular resolution observations, we can take advantage of this natural spatial segregation. Aims. Observations at different wavelengths probe different ranges of sizes; millimeter observations trace the larger grains, while near-infrared observations are sensitive to the other extreme of the size distribution. Consequently, there is a great synergy in combining both observational techniques to better constrain the optical properties of the particles. Methods. We present a new approach to simultaneously model observations from“Spectro-Polarimetric High Contrast Exoplanet REsearch” (SPHERE) and the“Atacama Large Millimeter Array” (ALMA) and apply it to the debris disk around HD 32297, putting the emphasis on the spatial distribution of the grains with different β values. This modeling approach requires few assumptions on the actual sizes of the particles and the interpretation can therefore be done a posteriori. Results. We find that the ALMA observations are best reproduced with a combination of small and large β values (0.03 and 0.42) while the SPHERE observations require several intervals of β values. We discuss the nature of the halo previously reported in ALMA observations, and hypothesize it could be caused by over-abundant μm-sized particles (the over-abundance being the consequence of their extended lifetime). We modeled the polarized phase function at near-infrared wavelengths, and fluffy aggregates larger than a few μm provide the best solution. Conclusions. Comparing our results with comets of the Solar System, we postulate that the particles released in the disk originate from rather pristine cometary bodies (to avoid compaction of the fluffy aggregates) and they are then set on highly eccentric orbits, which could explain the halo detected at long wavelengths.