An increasing number of applications require the accurate 3D layout reconstruction of indoor environments. Various devices including laser scanners and color and depth (RGB-D) cameras can be used for this purpose and provide abundant and highly precise data sources. However, due to indoor environment complexity, existing noise and occlusions caused by clutter in acquired data, current studies often require the idealization of the architecture space or add an implication hypothesis to input data as priors, which limits the use of these methods for general purposes. In this study, we propose a general 3D layout reconstruction method for indoor environments. The method combines voxel-based room segmentation and space partition to build optimum polygonal models. It releases idealization of the architectural space into a non-Manhattan world and can accommodate various types of input data sources, including both point clouds and meshes. A total of four point cloud datasets, four mesh datasets and two cross-floor datasets were used in experiments. The results exhibit more than 80% completeness and correctness as well as high accuracy.