We use a novel scanning electron Mach-Zehnder interferometer constructed in a conventional transmission electron microscope to perform inelastic interferometric imaging with free electrons. An electron wave function is prepared in two paths that pass on opposite sides of a gold nanoparticle, where plasmons are excited before the paths are recombined to produce electron interference. We show that the measured spectra are consistent with theoretical predictions, specifically that the interference signal formed by inelastically scattered electrons is out of phase with respect to that formed by elastically scattered electrons. This technique is sensitive to the phase of localized optical modes, because the interference signal amounts to a substantial fraction of the transmitted electrons. Thus, we argue that inelastic interferometric imaging with our scanning electron Mach-Zehnder interferometer provides a new platform for controlling the transverse momentum of free electrons and studying coherent electron-matter interactions at the nanoscale.