Thermal conductivity of asphalt concrete determines the temperature distribution in asphalt pavements and thus affects viscoelastic modulus of asphalt concrete and the microclimate environment near pavement surface. This paper developed an innovative model to evaluate thermal conductivity of asphalt concrete with heterogeneous microstructure. The three-dimensional (3-D) microstructures of asphalt concrete was simulated with different-sized aggregates and air voids randomly distributed in asphalt binder. A hierarchical multi-scale finite element (FE) modeling approach was used to simulate the steady heat transfer process for predicting the effective thermal conductivity of asphalt concrete. The results were validated with experiment data reported in the literature. With the developed model, the effects of aspect ratios and orientation angles of aggregate, conductive filler, and specimen size on thermal conductivity of asphalt concrete were analyzed. Results show that the orientation angle and aspect ratio of aggregate have combined effects on thermal conductivity of asphalt concrete, depending on the orientation of the longest diagonal of aggregate with respect to the direction of heat conduction. The thermal conductivity of asphalt concrete is affected by the content and shape of graphite filler for thermal modification. On the other hand, it is recommended that the specimen size should be at least five times the maximum aggregate size for measuring thermal conductivity. The larger ratio of specimen size to maximum aggregate size is needed when the maximum aggregate size increases. In general, the developed model can be used as an analysis tool to guide the mix design of asphalt concrete for thermal optimization. •Generate three-dimensional microstructure of asphalt concrete with three-phases.•Simulate steady heat transfer to predict thermal conductivity of asphalt concrete.•Validate finite-element simulation results with experimental data.•Evaluate effects of aggregate characteristics, conductive filler, and specimen sizes.