Graphene is a promising material for ultrafast and broadband photodetection. Earlier studies have addressed the general operation of graphene-based photothermoelectric devices and the switching speed, which is limited by the charge carrier cooling time, on the order of picoseconds. However, the generation of the photovoltage could occur at a much faster timescale, as it is associated with the carrier heating time. Here, we measure the photovoltage generation time and find it to be faster than 50 fs. As a proof-of-principle application of this ultrafast photodetector, we use graphene to directly measure, electrically, the pulse duration of a sub-50 fs laser pulse. The observation that carrier heating is ultrafast suggests that energy from absorbed photons can be efficiently transferred to carrier heat. To study this, we examine the spectral response and find a constant spectral responsivity of between 500 and 1,500 nm. This is consistent with efficient electron heating. These results are promising for ultrafast femtosecond and broadband photodetector applications.