Abstract A previous study (Paper I) investigated the polarization properties of a variety of simple convex grain shapes, some of which were found to be consistent with the observed polarization properties of interstellar dust from far-ultraviolet to far-infrared. Here, we study the optical properties of 45 nonconvex shapes, all aggregates of N equal-sized spheres. We consider N = 2, N = 3, and N = 256 random aggregates obtained from three different aggregation schemes. We also consider “trimmed” N = 256 aggregates obtained by systematically trimming initially random aggregates to increase either flattening or elongation. The “macroporosities” of the studied aggregates range from P macro = 0.18 (for the N = 2 bisphere) to P macro ≈ 0.85 (for the N = 256 “BA” aggregates). The only aggregates consistent with observations of starlight polarization and polarized thermal emission are shapes that have been trimmed to increase their asymmetry. If interstellar grains are high-porosity aggregates, there must be processes causing extreme elongation or flattening; if not, interstellar grains must be dominated by fairly compact structures, with at most moderate porosities. The ratio of polarization in the 10 μ m silicate feature to starlight polarization in the optical is shown to be insensitive to porosity and shape. X-ray scattering may be the best tool to determine the porosity of interstellar grains. We propose that modest porosities of interstellar grains could be the result of “photolytic densification.” High polarization fractions observed in some Class 0 cores require processes to reduce porosities and/or increase asymmetries of aggregates in dense regions.