We report a one-step fabrication of macroscopic multifunctional graphene-based hydrogels with robust interconnected networks under the synergistic effects of the reduction of graphene oxide sheets by ferrous ions and in situ simultaneous deposition of nanoparticles on graphene sheets. The functional components, such as α-FeOOH nanorods and magnetic Fe3O4 nanoparticles, can be easily incorporated with graphene sheets to assemble macroscopic graphene monoliths just by control of pH value under mild conditions. Such functional graphene-based hydrogels exhibit excellent capability for removal of pollutants and, thus, could be used as promising adsorbents for water purification. The method presented here is proved to be versatile to induce macroscopic assembly of reduced graphene sheets with other functional metal oxides and thus to access a variety of graphene-based multifunctional nanocomposites in the form of macroscopic hydrogels or aerogels.