Electron-phonon coupling processes determine electronic transport properties of materials and are responsible for the transfer of electronic excess energy to the lattice. With decreasing device dimensions an understanding of these processes in nanoscale materials is becoming increasingly important. Here we use time-resolved electron diffraction to directly study energy relaxation in thin bismuth films after optical excitation. Precise measurements of the transient Debye-Waller-effect for various film thicknesses and over an extended range of excitation fluences allow to separate different contributions to the incoherent lattice response. While phonon softening in the electronically excited state is responsible for an immediate increase of the r.m.s. atomic displacement within a few hundred fs, 'ordinary' electron-phonon coupling leads to subsequent heating of the material on a few ps time-scale. The data reveal distinct changes in the energy transfer dynamics which becomes faster for stronger excitation and smaller film thickness, respectively. The latter effect is attributed to a cross-interfacial coupling of excited electrons to phonons in the substrate.