Revealing the relationship between plants and fungi is very important in understanding biodiversity maintenance, community stability, and ecosystem functioning. However, differences in the community and network structures of phyllosphere epiphytic and endophytic fungi are currently poorly documented. In this study, we examined epiphytic and endophytic fungal communities associated with the leaves of six mangrove species using Illumina MiSeq sequencing of internal transcribed spacer 2 (ITS2) sequences. A total of 635 operational taxonomic units (OTUs) of endophytic and epiphytic fungi were obtained at a 97% sequence similarity level; they were dominated by Dothideomycetes and Tremellomycetes. Plant identity had a significant effect on the OTU richness of endophytic fungi, but not on epiphytic fungi. The community composition of epiphytic and endophytic fungi was significantly different, and plant identity had a greater effect on endophytic fungi than on epiphytic fungi. Network analysis showed that both epiphytic and endophytic network structures were characterized by significantly highly specialized and modular but lowly connected and anti-nested properties. Furthermore, the endophytic network had higher levels of specialization and modularity but lower connectance and stronger anti-nestedness than the epiphytic network. This study reveals that the phyllosphere epiphytic and endophytic fungal communities differ, and plant identity has a greater effect on the endophytic fungi than on epiphytic fungi. These findings demonstrate the role of host plant identity in driving phyllosphere epiphytic and endophytic community structure.