As I understand it, instrument code can be written either in WikimediaEvents or in the product codebase.

That's correct.

To do: Clarify when developers should write instrument code in one location verses the other

This is a tricky one and it really is an "it depends/use your best judgement."

We should be encouraging WMF feature teams to be writing instrument code in the WikimediaEvents extension unless their codebase is very clearly intended for use only by Wikimedia. This is because the codebases that we host (catch-all term for codebases that we product, maintain, or contribute to) can be installed by third-party wiki admins. Those admins shouldn't have to install instruments and supporting code (the EventLogging, EventStreamConfig, and EventBus extensions) in order to get a feature to work.

I think what's lacking here are corresponding software development patterns and practices for instrument developers that allow them to develop their feature in one place and their instrument in another. For sure, there are APIs available, but I don't think anyone's bothered to recommend a way to use them other than during code review…

For example:

MediaWiki Core provides mw.track() and mw.trackSubscribe(), and mw.hook(). These methods are available everywhere and facility cross-codebase signalling. mw.track()/mw.trackSubscribe() are useful in situations where a subscriber might subscribe late and need access to all messages that have been tracked. Indeed, mw.track()/mw.trackSubscribe() are documented as being intended for analytics purposes.

There is a subscriber for the Legacy EventLogging system. Calling mw.track( event.${schema}, eventData ); will result in a call to mw.eventLog.logEvent( schema, eventData ) if the EventLogging extension is installed. However, as yet, there are no equivalent subscribers that make calls to mw.eventLog.submit() nor mw.eventLog.submitInteraction().