A phenomenological model for pattern formation in a vertically vibrated granular layer is examined in order to investigate its nonlinear dynamics. The model comprises two coupled partial differential equations: one describes the evolution of the short-scale pattern, while the other enforces conservation of granular material. In a layer of moderate horizontal extent, the model predicts that a variety of exotic regular patterns may be stable, according to the system parameters. The usual cubic-order amplitude equations are unable to determine the stable solution over a significant parameter range; we compute the corresponding fifth-order terms necessary to resolve this degeneracy. When spatial modulation of the pattern is taken into account, in a sufficiently wide layer, a stability analysis of regular one-dimensional roll and two-dimensional square patterns demonstrates that each may suffer a modulational instability, which tends to localize the pattern. The corresponding modulational stability boundaries, for both rolls and squares, coincide with the transition between stable rolls and squares in the unmodulated problem. As a consequence, in a suitably large container, squares are always unstable, and corresponding numerical simulations indicate highly localized worm- or chain-like patterns. The numerical simulations and stability results are compared with appropriate experimental results.