Repository navigation
[APM] Add message about dropped spans #24320
Description
Activity
- addedTeam:APM - DEPRECATEDUse Team:obs-ux-infra_services.Use Team:obs-ux-infra_services.
on Oct 21, 2018 Pinging @elastic/apm-ui
@zubeio/apm-server (or possibly @sqren or @formgeist ?) -- can someone explain in this ticket (or point to a link where it's explained) what we should expect with this part of the Transaction (elasticsearch data) interface:
span_count?: { dropped?: { total?: number; }; };
A couple questions:
- Why are these optional?
- Can the
span_countkey be present but its value be an empty hash{}? - Looks like we used to use this value, and default to 0 if any part of this value didn't exist, when deciding whether to notify a user about dropped spans ... is that still right?
@jasonrhodes the values are optional, as they were introduced in a minor version update and we cannot break the API by adding required attributes. There should not be an entry with an empty hash in ES. It seems valid to not notify users about missing spans when this value is missing, as it basically only means we don't have any information about it. It would be wrong though to assume there are no missing spans just because the information is not there.
@simitt that all sounds good to me!
8 remaining items
Søren laid out the two scenarios nicely here #24320 (comment), but just as a heads up, using this message wouldn't be entirely accurate:
{droppedSpansCount} spans dropped due to limit of {spans.data.spans.length}.
because we can't necessarily know the configuration setting in the agent by simply counting the spans we received. Spans could be dropped for other reasons than the config limit. For example in cases where Elasticsearch is overloaded. I suggest we simply say "Agent dropped {dropped} spans for this transaction due to a configured limit: "
We now have the
startedproperty which we could compare tospans.data.spans.lengthto see if we have lost any spans accidentally. This should be a separate message. Something like:There are {started-spans.data.spans.length} spans missing for this transaction: <link-to-docs>. This is not a regression, so don't rush to get it in for 6.5Looking into the future: If the timeline contains any transaction with dropped or missing spans, we could show a message to that effect to highlight it already on the timeline.
Reacted by Jason RhodesFor the Java agent, dropped spans are only indirectly mentioned in the
transaction_max_spansconfiguration option: https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-transaction-max-spansAlthough it doesn't explicitly use the term dropped spans.
There are {started-spans.data.spans.length} spans missing for this transaction: <link-to-docs>.As
startedshould be interpreted as x or more (spans could be started after the transaction has already been reported), the message should be eitherThere are at least {started-spans.data.spans.length} spans missing for this transaction: <link-to-docs>., or justThere are spans missing for this transaction: <link-to-docs>.Reacted by Jason RhodesMaybe
Something happened <link-to-google>Where did we end up on the implementation here? Let's keep missing spans out of the equation for now, as we're just dealing with the regression, which has to do with the dropped span message. Next release we can make it better and have shared way to display a message to the end user.
@formgeist thoughts on this layout/wording? Happy to tweak/adjust.
@jasonrhodes I'd prefer a small EuiCallOut (
info) because I think most users will quickly skim the page, and not notice the information we're providing.As for the message, I would propose the following changes;
The APM agent that reported this transaction dropped more than 10 spans based on its configuration. Learn more.
The "at least" reads too dramatic to me, but maybe because I'm not a native speaker 🙂
@formgeist cool, I can update to EuiCallOut. "more than" isn't accurate because it might be exactly 10. Options are probably "at least 10 spans" or "10 or more spans".
Then I prefer "10 or more spans"
Reacted by Jason Rhodes and Felix Barnsteiner
The new DT timeline does not yet have the message about dropped spans, that was available pre 6.5
kibana/x-pack/plugins/apm/public/components/app/TransactionDetails/Transaction/Spans/view.js
Lines 131 to 143 in 7266349
Closes: #22348