Display omitted •Gel layer was more easily formed when SMP content was relatively high.•Gel adhesion had higher contact interaction energy but had repulsive energy barrier.•Gel layer had a SFR which was about 100 times higher than that of cake layer.•The unusually high SFR of gel layer was caused by osmotic pressure mechanism. The fouling mechanisms underlying gel layer formation and its filtration resistance in a submerged membrane bioreactor (MBR) were investigated. It was found that gel layer rather than cake layer was more easily formed when soluble microbial products content in sludge suspension was relatively high. Thermodynamic analyses showed that gel layer formation process should overcome a higher energy barrier as compared with cake layer formation process. However, when separation distance <2.3nm, attractive interaction energy of gelling foulant–membrane combination was remarkably higher than that of sludge floc–membrane combination. The combined effects were responsible for gel layer formation. Filtration tests showed that specific filtration resistance (SFR) of gel layer was almost 100 times higher than that of cake layer. The unusually high SFR of gel layer could be ascribed to the gelling propensity and osmotic pressure mechanism. These findings shed significant light on fouling mechanisms of gel layer in MBRs.