In addition to the dominant oscillatory gravitational wave signals produced during binary inspirals, a non-oscillatory component arises from the nonlinear 'memory' effect, sourced by the emitted gravitational radiation. The memory grows significantly during the late-inspiral and merger, modifying the signal by an almost step-function profile, and making it difficult to model by approximate methods. We use numerical evolutions of binary black holes (BHs) to evaluate the nonlinear memory during late-inspiral, merger, and ringdown. We identify two main components of the signal: the monotonically growing portion corresponding to the memory, and an oscillatory part which sets in roughly at the time of merger and is due to the BH ringdown. Counterintuitively, the ringdown is most prominent for models with the lowest total spin. Thus, the case of maximally spinning BHs anti-aligned to the orbital angular momentum exhibits the highest signal-to-noise ratio (S/N) for interferometric detectors. The largest memory offset, however, occurs for highly spinning BHs, with an estimated value of h tot 20 0.24 in the maximally spinning case. These results are central to determining the detectability of nonlinear memory through pulsar timing array measurements.