Description

This is "close enough" implementation of the parse_multiline processor that uses Fluent-bit state machine solutions. The Otel recombine uses a is_first_entry or is_last_entry (only one can be set) to create multi-line log groups.

Limitations

Summary : The solution works nicely with individual languages go, python or java used separately. Since there are possible overlapping of the format of "stacktrace" log lines, the interactions of setting multiple languages may be to fragile to implement. A "state machine" solutions avoids this issues since being in a go, python or java language starting state reduces the possible "continuing" states.

The current implemented solution tries to separate all possible log lines into 4 sets / categories :

1. Java Log Entry
   • Check for Java First Line Log Entry to start a multiline group.
   • Append together nested mutli-line groups.
2. Python Log Entry
   • Check for Python First Line Log Entry to start a multiline group.
   • Append together nested mutli-line groups.
3. Go Log Entry
   • Check for Go First Line Log Entry to start a multiline group.
   • Append together nested mutli-line groups.
4. Not a Go or Python or Java Log Entry
   • Start a multiline group since this doesn't belong to any of our language definitions.

This solution depends completely in how accurate an descriptive are the "regex" patterns that describe each category. The main edge case that won't solvable (or may be too fragile) is the interactions between languages.

For example, using multiple languages causes a lot of edge cases.

parse_multiline
  match_any:
  - type: language_exceptions
    language: java
  - type: language_exceptions
    language: go
  - type: language_exceptions
    language: python  

Related issue

b/413446918

How has this been tested?

Checklist:

• Unit tests
  • Unit tests do not apply.
  • Unit tests have been added/modified and passed for this PR.
• Integration tests
  • Integration tests do not apply.
  • Integration tests have been added/modified and passed for this PR.
• Documentation
  • This PR introduces no user visible changes.
  • This PR introduces user visible changes and the corresponding documentation change has been made.
• Minor version bump
  • This PR introduces no new features.
  • This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
  • This PR bumps the version.