A flexible composite solid electrolyte membrane consisting of inorganic solid particles (Li1.3Al0.3Ti1.7(PO4)3), polyethylene oxide (PEO), and boronized polyethylene glycol (BPEG) is prepared and investigated. This membrane exhibits good stability against lithium dendrite, which can be attributed to its well‐designed combination components: the compact inorganic lithium ion conducting layer provides the membrane with good mechanical strength and physically barricades the free growth of lithium dendrite; while the addition of planar BPEG oligomers not only disorganizes the crystallinity of the PEO domain, leading to good ionic conductivity, but also facilitates a “soft contact” between interfaces, which not only chemically enables homogeneous lithium plating/stripping on the lithium metal anode, but also reduces the polarization effects. In addition, by employing this membrane to a LiFePO4/Li cell and testing its galvanostatic cycling performances at 60 °C, capacities of 158.2 and 94.2 mA h g−1 are delivered at 0.1 C and 2 C, respectively. A flexible composite solid electrolyte membrane consisting of polymer and Li‐ion conductive ceramic is prepared and investigated. The addition of boronized polyethylene glycol oligomer improves its ionic conductivity, and facilitates a better contact between the lithium metal and the electrolyte, resulting in a smooth lithium interface after cycling.