In addition to the dominant oscillatory gravitational wave signals produced during binary inspirals, a non-oscillatory component arises from the nonlinear "memory" effect, sourced by the emitted gravitational radiation. The memory grows significantly during the late inspiral and merger, modifying the signal by an almost step-function profile, and making it difficult to model by approximate methods. We use numerical evolutions of binary black holes to evaluate the nonlinear memory during late-inspiral, merger and ringdown. We identify two main components of the signal: the monotonically growing portion corresponding to the memory, and an oscillatory part which sets in roughly at the time of merger and is due to the black hole ringdown. Counter-intuitively, the ringdown is most prominent for models with the lowest total spin. Thus, the case of maximally spinning black holes anti-aligned to the orbital angular momentum exhibits the highest signal-to-noise (SNR) for interferometric detectors. The largest memory offset, however, occurs for highly spinning black holes, with an estimated value of h^tot_20 \approx 0.24 in the maximally spinning case. These results are central to determining the detectability of nonlinear memory through pulsar timing array measurements.