A search for nonresonant Higgs boson pair production in the <img src="http://www.diva-portal.org/cgi-bin/mimetex.cgi?%5Cmathit%7Bb%5Cbar%7Bb%7Db%5Cbar%7Bb%7D%7D" data-classname="equation" data-title="" /> final state is presented. The analysis uses 126  fb −1 of pp collision data at √ s =13  TeV collected with the ATLAS detector at the Large Hadron Collider, and targets both the gluon-gluon fusion and vector-boson fusion production modes. No evidence of the signal is found and the observed (expected) upper limit on the cross section for nonresonant Higgs boson pair production is determined to be 5.4 (8.1) times the Standard Model predicted cross section at 95% confidence level. Constraints are placed on modifiers to the HHH and HHVV couplings. The observed (expected) 2σ constraints on the HHH coupling modifier, κ λ , are determined to be −3.5,11.3 (−5.4,11.4), while the corresponding constraints for the HHVV coupling modifier, κ 2V , are −0.0,2.1 (−0.1,2.1). In addition, constraints on relevant coefficients are derived in the context of the Standard Model effective field theory and Higgs effective field theory, and upper limits on the HH production cross section are placed in seven Higgs effective field theory benchmark scenarios.