Thermal management is critical to the performance, lifetime, and reliability of electronic devices. With the miniaturization, integration and functionalization of electronics and the emergence of new applications such as light emitting diodes, thermal dissipation becomes a challenging problem. Addressing this challenge requires the development of novel polymer-based composite materials with enhanced thermal conductivity. In this review, the fundamental design principles of highly thermally conductive composites were discussed. The key factors influencing the thermal conductivity of polymers, such as chain structure, crystallinity, crystal form, orientation of polymer chains, and orientation of ordered domains in both thermoplastics and thermosets were addressed. The properties of thermally conductive fillers (carbon nanotubes, metal particles, and ceramic particles such as boron nitride or aluminum oxide) are summarized at length. The dependence of thermal conductivity of composites on the filler loading, filler aggregate morphology and overall composite structure is also discussed. Special attention is paid to recent advances in controlling the microstructure of polymer composites to achieve high thermal conductivity (novel approaches to control filler orientation, special design of filler agglomerates, formation of continuous filler network by self-assembly process, double percolation approach, etc.). The review also summarizes some emerging applications of thermally conductive polymer composites. Finally, we outline the challenges and outlook for thermally conductive polymer composites.