Is there a reason we only define them for XHTML given that they could be put on text/tspan elements in SVG?

The one awkward part is that you can't define lexicons in SVG since it lacks a link element, but there's no reason you can't use SSML or CSS Speech in SVG.