We describe the construction of novel end-to-end jet image classifiers to discriminate quark- versus gluon-initiated jets using the simulated CMS Open Data. These multi-detector images correspond to true maps of the low-level energy deposits in the detector, giving the classifiers direct access to the maximum recorded event information about the jet, differing fundamentally from conventional jet images constructed from reconstructed particle-level information. Using this approach, we achieve classification performance competitive with current state-of-the-art jet classifiers that are dominated by particle-based algorithms. We find the performance to be driven by the availability of precise spatial information, highlighting the importance of high-fidelity detector images. We then illustrate how end-to-end jet classification techniques can be incorporated into event classification workflows using Quantum Chromodynamics di-quark versus di-gluon events. We conclude with the end-to-end event classification of full detector images, which we find to be robust against the effects of underlying event and pileup outside the jet regions-of-interest.