When an ingest pipeline error occurs, our log pipelines should retain the event.original value to ensure that no data loss occurs and to facilitate correcting the failure. If processing is interrupted due to an error, some data may not have been extracted (i.e. incomplete processing) so it's important to retain the event.original. And secondly, in order for the package maintainers to be able to take action on reports of pipeline failures they nearly always need the event.original value to reproduce and understand the issue.
To implement this we should complete the work related to #10072. This not strictly required, but it helps ensure the event.original is consistently handled. We want the Fleet final_pipeline to be responsible for deleting event.original when tags does not contain preserve_original_event.
Next, in the primary pipeline of each log data stream we update the global on_failure handler to inject preserve_original_event into tags. This will accompany event.kind: pipeline_error. With this mechanism, users can still override this behavior through the various levels of @custom pipelines by deleting the tag value. I expect the work to be accomplished "mechanically", and this mechanism can be applied separately on the integrations owned by each SIT team.
TODO
When an ingest pipeline error occurs, our log pipelines should retain the
event.originalvalue to ensure that no data loss occurs and to facilitate correcting the failure. If processing is interrupted due to an error, some data may not have been extracted (i.e. incomplete processing) so it's important to retain theevent.original. And secondly, in order for the package maintainers to be able to take action on reports of pipeline failures they nearly always need theevent.originalvalue to reproduce and understand the issue.To implement this we should complete the work related to #10072. This not strictly required, but it helps ensure the
event.originalis consistently handled. We want the Fleet final_pipeline to be responsible for deleting event.original whentagsdoes not containpreserve_original_event.Next, in the primary pipeline of each log data stream we update the global
on_failurehandler to injectpreserve_original_eventintotags. This will accompanyevent.kind: pipeline_error. With this mechanism, users can still override this behavior through the various levels of@custompipelines by deleting the tag value. I expect the work to be accomplished "mechanically", and this mechanism can be applied separately on the integrations owned by each SIT team.TODO
preserve_original_eventin primary on_failure handlers