We analyze the framework recently proposed by Oppenheim et al (2023 Nat. Commun. 14; 2023 Phys. Rev. X 13 041040; arXiv:2302.07283 [gr-qc]; 2023 J. High Energy Phys. JHEP08(2023)163) to model relativistic quantum fields coupled to relativistic, classical, stochastic fields (in particular, as a model of quantum matter coupled to ‘classical gravity’). Perhaps surprisingly, we find that we can define and calculate scattering probabilities which are Lorentz-covariant and conserve total probability, at least at tree level. As a concrete example, we analyze 2 → 2 scattering of quantum matter mediated by a classical Yukawa field. Mapping this to a gravitational coupling in the non-relativistic limit, and assuming that we can treat large objects as point masses, we find that the simplest possible ‘classical-quantum’ gravity theory constructed this way gives predictions for 2 → 2 gravitational scattering which are inconsistent with simple observations of, e.g. spacecraft undergoing slingshot maneuvers. We comment on lessons learned for attempts to couple quantum matter to ‘non-quantum’ gravity, or more generally, for attempts to couple relativistic quantum and classical systems.