We consider symmetry-protected topological (SPT) phases with crystalline point group symmetry, dubbed point group SPT (pgSPT) phases. We show that such phases can be understood in terms of lower-dimensional topological phases with on-site symmetry and that they can be constructed as stacks and arrays of these lower-dimensional states. This provides the basis for a general framework to classify and characterize bosonic and fermionic pgSPT phases, which can be applied for arbitrary crystalline point group symmetry and in arbitrary spatial dimensions. We develop and illustrate this framework by means of a few examples, focusing on three-dimensional states. We classify bosonic pgSPT phases and fermionic topological crystalline superconductors with Z2P (reflection) symmetry, electronic topological crystalline insulators (TCIs) with U(1)×Z2P symmetry, and bosonic pgSPT phases with C2v symmetry, which is generated by two perpendicular mirror reflections. We also study surface properties, with a focus on gapped, topologically ordered surface states. For electronic TCIs, we find a Z8×Z2 classification, where the Z8 corresponds to known states obtained from noninteracting electrons, and the Z2 corresponds to a “strongly correlated” TCI that requires strong interactions in the bulk. Our approach may also point the way toward a general theory of symmetry-enriched topological phases with crystalline point group symmetry.