Tags: kdada/fluent-bit

in_systemd: improve performance when packaging fields

When processing records, each one have multiple fields and due to
serialization when converting from Systemd format to Fluent Bit, we
need to know beforehand how many fields a record contains before
do the packaging. Systemd C API do not expose a function to get the
number of fields contained in a record, so the solution was to iterate
all fields and do the proper counting. This procedure is not optimal
for performance since once the fields are counted, we need to iterate
all fields again to do the packaging.

I've opened a RFE with Systemd team asking for such missing feature:

  systemd/systemd#10906

In the meanwhile, this patch workaround the problem in the following way:

  - Pack (register) the maximum number of fields allowed by 'max_fields'
    property.
  - Iterate, count and package all Systemd fields from the record.
  - Do a manual adjustment of the msgpack Map size header.

With this approach there is a performance improvement:

  - before => in_systemd_collect():  78.39%
  - after  => in_systemd_collect():  73.90%

this improvement represents a ~4.5% gain in performance.

note: the test was done using Valgrind (--tool=callgrind) processing 4k of
records coming from the Journal.

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

worker: initialize event structure

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

tests: internal: pack: use new flb_pack_json() prototype

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

filter_throttle: fix leak on cleanup (fluent#868)

The hash table and window were leaked in the destructor.

Signed-off-by: Don Bowman <don@agilicus.com>

out_influxdb: small code cleanup

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

in_tail: fix leak on file name on rotated file (CID 183990)

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

lib: libbacktrace: upgrade to 5a99ff7f (git master)

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

tests: runtime: filter_kubernetes: validate merge key usage on unstru…

…ctured message

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

build: remove _WITHOUT_ word from cmake options

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

parser_decoder: utf8: do not read escaped newline (fluent#615) (fluen…

…t#692)

In the UTF-8 decoder, it tries to read escaped char for general
purpose. Escaping a new-line is valid in JSON encoding, if it
lines a UTF-8 decoder and then a JSON decoder, the JSON decoder
will fail, as the escaped new-line is treated as a real newline
in the first UTF-8 decoder.

This fixes the issue in fluent#615

Signed-off-by: Richard Meng <mengangr@foxmail.com>