implement SAP hana logs receiver #496
Conversation
|
Are the multi-line log entries being captured correctly? |
|
@PhilBrammer they should be. Still working on validation. |
|
AFAIK, without using the multiline regex processor, I would expect each line in the multiline log to be treated as a separate log entry, to be parsed with your regex as individual lines. Recommend using LoggingProcessorParseMultilineRegex. Mysql logging implements this: Line 154 in 0518697 |
|
@PhilBrammer the pattern I'm following here is the more recent pattern that we've been using (see elasticsearch, couchdb, hadoop, rabbitmq, zookeeper & wildfly). The pattern in mysql can encounter issues with with logs being separated by chunks in fluent bit and ending up not being parsed together via the multiline parser, though the pattern we're following here where the multiline is attached to the input also has potential issues (unverified) with syslog forwarding. |
apps/saphana.go
Outdated
| // 1: 0x00007f1937d9095c in .LTHUNK27.lto_priv.2256+0x558 (libhdbbasis.so) | ||
| // ... | ||
| // 25: 0x0000563f44888831 in _GLOBAL__sub_I_setServiceStarting.cpp.lto_priv.239+0x520 (hdbnsutil) | ||
| Regex: `^\[(?<thread_id>\d+)\]\{(?<connection_id>-?\d+)\}\[(?<transaction_id>-?\d+)\/(?<update_transaction_id>-?\d+)\]\s+(?<time>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d{3,6}\d+)\s+(?<severity_flag>\w+)\s+(?<component>\w+)\s+(?<source_file>\S+)\s+:\s+(?<message>[\s\S]+)`, |
There was a problem hiding this comment.
For the source_file field here, I wonder if we can use the LogEntry sourceLocation field. We have an example here:
ops-agent/confgenerator/logging_receivers.go
Lines 420 to 432 in 9d9f04c
Would be a bonus if we can reuse the modify_fields processor below but I don't see a nest option
apps/saphana.go
Outdated
| } | ||
|
|
||
| func (LoggingProcessorSapHanaTrace) Type() string { | ||
| return "saphana_trace" |
There was a problem hiding this comment.
Are there other logs that are not trace that make its way to syslog? Or that could be configured in the future?
I'm wondering why we don't just call this saphana like we do with other receivers that have only a singular input
There was a problem hiding this comment.
Sure, we can change it.
…into google logging struct
| then | ||
| record["logging.googleapis.com/sourceLocation"] = {} | ||
| end | ||
| record["logging.googleapis.com/sourceLocation"]["line"] = value |
There was a problem hiding this comment.
See here. The Lua code already nests it and so the additional filter is a no-op essentially
apps/saphana.go
Outdated
| }.Components(tag, uid)..., | ||
| ) | ||
|
|
||
| c = append(c, fluentbit.Component{ |
There was a problem hiding this comment.
This isn't necessary is it? I believe modify_fields already nests this under sourceLocation. See the other comment for the actual LUA code that the modify creates. Certainly looks like a nesting is happening already.
|
Looks like everything passes except for |
Logs receiver for SAP HANA trace logs is relatively straightforward.
Below is an outdated image (source_file), but shows multiline support
