Why MultilineRules were kept in INPUT ?

The main reason i find for why the MultilineRules were kept in the INPUT component was what it is described in this comment This is necessary, since using the multiline filter will not work if a multiline message spans between two chunks..
I found some sources that back up this issue :

• multiline logs being cutoff randomly fluent/fluent-bit#3901
• Multiline parsing breaks when Buffer_Chunk_Size boundary is crossed fluent/fluent-bit#4317
• https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#limitations
• https://www.cncf.io/blog/2025/04/08/managing-multi-line-logs-with-fluent-bit-and-python/
  • Describing how buffering may interact with multiline parsing.
  • Mentions performance differences between input and filter multiline parsing.

Answer

Re @quentinmit :

Why can't these apps just use LoggingProcessorParseMultilineRegex instead of making all these changes to hack together support for manipulating file receivers?

I haven't considered so far (though it is indeed possible) to change / refactor the main 3P app processor logic at all. This implies work on "3p app receivers fluent-bit logic" rather than "refactoring 3P receivers for Otel Logging", so it would be additional effort.

Currently, I was considering it would be better to support 3P receivers for transformation testing which extends our testing possibilities, rather than considering working on Fluent-bit specific questions like "Does moving 'multiline.parsing' out of the input will still handle long-multiline logs correctly through multiple chunks and same performance ?".

I'm weighing as valuable to keep the same fluent-bit processor logic (same user feature expectations) and proposing this PR (we could improve) as a way to achieve that.

With this considerations, do you think its better to work on refactoring fluent-bit multiline parsing instead of the approach of this PR ?

Why MultilineRules were kept in INPUT ?

The main reason i find for why the MultilineRules were kept in the INPUT component was what it is described in this comment This is necessary, since using the multiline filter will not work if a multiline message spans between two chunks.. I found some sources that back up this issue :

• multiline logs being cutoff randomly fluent/fluent-bit#3901

• Multiline parsing breaks when Buffer_Chunk_Size boundary is crossed fluent/fluent-bit#4317

• https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#limitations

• https://www.cncf.io/blog/2025/04/08/managing-multi-line-logs-with-fluent-bit-and-python/

  • Describing how buffering may interact with multiline parsing.
  • Mentions performance differences between input and filter multiline parsing.

ops-agent/confgenerator/logging_receivers.go

Lines 136 to 156 in 3ed6d7e

	if len(r.MultilineRules) > 0 {
	// Configure multiline in the input component;
	// This is necessary, since using the multiline filter will not work
	// if a multiline message spans between two chunks.
	rules := [][2]string{}
	for _, rule := range r.MultilineRules {
	rules = append(rules, [2]string{"rule", rule.AsString()})
	}
	parserName := fmt.Sprintf("multiline.%s", tag)
	c = append(c, fluentbit.Component{
	Kind: "MULTILINE_PARSER",
	Config: map[string]string{
	"name": parserName,
	"type": "regex",
	"flush_timeout": "5000",
	},
	OrderedConfig: rules,
	})

Yeah, I know about that bug. But we already have a pattern to resolve that:

We can apply the same logic to logging receivers, which would allow the multiline parser to be configured as part of the file receiver to work around the fluent-bit bug.

Answer

Re @quentinmit :

Why can't these apps just use LoggingProcessorParseMultilineRegex instead of making all these changes to hack together support for manipulating file receivers?

I haven't considered so far (though it is indeed possible) to change / refactor the main 3P app processor logic at all. This implies work on "3p app receivers fluent-bit logic" rather than "refactoring 3P receivers for Otel Logging", so it would be additional effort.

Currently, I was considering it would be better to support 3P receivers for transformation testing which extends our testing possibilities, rather than considering working on Fluent-bit specific questions like "Does moving 'multiline.parsing' out of the input will still handle long-multiline logs correctly through multiple chunks and same performance ?".

I'm weighing as valuable to keep the same fluent-bit processor logic (same user feature expectations) and proposing this PR (we could improve) as a way to achieve that.

I don't believe my suggestion would change any of the user-visible processor logic.

With this considerations, do you think its better to work on refactoring fluent-bit multiline parsing instead of the approach of this PR ?

I think it's worth preserving our abstraction barriers. I don't actually think it would be much of a refactoring - you would keep MultilineRules as it is in LoggingReceiverFilesMixin, but you would add a MergeLoggingProcessor() method that takes a LoggingProcessorParseMultilineRegex and turns it into the MultilineRules field.

The transformation tests would use the processor, but the full config would configure the same parser directly on the input to avoid the chunk size bug.

Incidentally, I think this is kinda moot for now because all the receivers that use multiline are going to need additional changes to deal with the differences between otel and fluent-bit (I suspect by manually modifying all of the multiline regexes). So that's a further argument against adding abstraction-breaking complexity.

Re @quentinmit :

I think it's worth preserving our abstraction barriers. I don't actually think it would be much of a refactoring - you would keep MultilineRules as it is in LoggingReceiverFilesMixin, but you would add a MergeLoggingProcessor() method that takes a LoggingProcessorParseMultilineRegex and turns it into the MultilineRules field.

The transformation tests would use the processor, but the full config would configure the same parser directly on the input to avoid the chunk size bug.

Thank you for the suggestion of an alternative solution on how to test receivers with MultilineRules in LoggingReceiverFilesMixin.

I implemented the alternative solution in #2025 following the guidelines of MetricsProcessorMerger, though with a main difference is the layer of abstraction (ops agent pipeline vs within a receiver) at which this "merges" happen. More details on #2025.

I don't believe my suggestion would change any of the user-visible processor logic.

Yes, i can confirm this implemented solution didn't have any user-visible changes in the processor logic.

Closing in favour of #2025.