A ventilated acoustic resonator (VAR), a type of acoustic metamaterial (AM) has emerged as a promising solution for mitigating urban noise pollution and traffic noise which simultaneously require ventilation. However, due to the high nonlinearity, the inverse design of complex VAR is intractable with analytical methods. Deep learning-based inverse design methods are gaining prominence as an alternative to analytical methods but still exhibit significant challenges: limited design flexibility in parameter-based approaches and the deterioration of essential shapes for sound attenuation performance in pixel image-based approaches. To address these challenges, we propose an inverse design framework of ultra-broadband non-parametric VAR through a genetic algorithm (GA) optimization-based latent space exploration strategy. The GA-based exploration on the dimension-reduced latent space of a conditional variational autoencoder (CVAE) enables the generation of the ultra-broadband non-parametric VAR preserving essential shape for sound attenuation with reduced computational costs. The GA-optimized non-parametric VARs show an average 28.76% bandwidth increase compared with the training dataset and, also demonstrate a considerably wider bandwidth compared to the parameter-based optimization methods, which expands the limit of the sound attenuation performance. Our novel approach paves the way for the optimization of complex mechanical structures.