feat: Update nginx receiver to use `LoggingReceiverMacro` #1978

Caleb-Hurshman · 2025-07-16T13:36:50Z

Description

Refactor the nginx receiver logging to be sub-agent agnostic.
Draft stage, creating this PR to run transformation tests until they are running locally.

Related issue

How has this been tested?

Checklist:

Unit tests
- Unit tests do not apply.
- Unit tests have been added/modified and passed for this PR.
Integration tests
- Integration tests do not apply.
- Integration tests have been added/modified and passed for this PR.
Documentation
- This PR introduces no user visible changes.
- This PR introduces user visible changes and the corresponding documentation change has been made.
Minor version bump
- This PR introduces no new features.
- This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
- This PR bumps the version.

google-cla · 2025-07-16T13:36:55Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

.github/workflows/stale.yml

apps/nginx.go

confgenerator/testdata/goldens/logging-receiver_nginx/golden/windows-2012/fluent_bit_main.conf

.../testdata/goldens/logging-receiver_nginx/golden/windows/4a885f78c2b41df75250528bf3ea0c2c.lua

apps/nginx.go

franciscovalentecastro · 2025-07-18T02:17:31Z

confgenerator/testdata/feature/golden.csv

Note : This is just a comment to say that updates in files golden.csv, feature_tracking_otlp.json and features.yaml are expected when updating a receiver.

franciscovalentecastro · 2025-07-18T02:31:47Z

transformation_test/testdata/logging_processor-nginx-access/output_fluentbit.yaml

Note: One way of verifying the resulting updated implementation of the ngnix_access (or any other) receiver is working as expected is that make transformation_test should produce exactly the same output_fluentbit.yaml before and after the code update to nginx.go.

No change in output_fluentbit.yaml implies that the behaviour of the receiver/processors is preserved.

Very good to know!

apps/common_logging_processors.go

transformation_test/testdata/ops_agent_test-TestLogEntrySpecialFields/output_otel.yaml

apps/common_logging_processors.go

franciscovalentecastro · 2025-07-18T19:28:09Z

This PR is looking good. Thank you for all the updates @Caleb-Hurshman ! I added one last comment. Let's also wait for integration tests to run to see if everything is working correctly.

Caleb-Hurshman · 2025-07-21T14:32:55Z

@franciscovalentecastro 3 of the integration tests are failing. Looking at the logs, I don't see any direct relation to my changes. Do you have any advice for diagnosing these failures?

…essor

franciscovalentecastro · 2025-07-21T14:55:01Z

apps/common_logging_processors.go

+// TODO: rename to genericAccessLogParser once the old version is removed
+// genericAccessLogParserAsInternalLoggingProcessor is an internal logging processor that parses access logs.
+// It will eventually replace genericAccessLogParser, as it returns a slice of InternalLoggingProcessor rather than fluentbit.Component, making it more flexible.
+func genericAccessLogParserAsInternalLoggingProcessor(ctx context.Context, processorType string) []confgenerator.InternalLoggingProcessor {


Note: A teammate (@quentinmit) suggested an improvement of this function by converting genericAccessLogParserAsInternalLoggingProcessor into an InternalLoggingProcessor (1. Creating a struct, 2. Defining this function as its Expand() method).

We can do this in a followup PR when updating any of the other receivers (apache, tomcat, ...) that use this. I just wanted to keep a note on this idea.

franciscovalentecastro

LGTM! Added a Note: of an idea we can address in a followup PR.

franciscovalentecastro · 2025-07-21T16:17:05Z

@franciscovalentecastro 3 of the integration tests are failing. Looking at the logs, I don't see any direct relation to my changes. Do you have any advice for diagnosing these failures?

There doesn't seem to be any relation to your changes. For example :

third party apps integration test (Ubuntu 24.04 Noble x86_64 / AArch64) seem to be only related to rabbitmq. This may need to be addressed / solved separately.
third party apps integration test (Debian 11 Bullseye x86_64) is a flake / failure that occurs sometimes when installing packages in Debian 11.

After the this last test run, i will re-rerun the tests and merge the PR if the result still looks unrelated to this PR. I will triage the rabbitmq issue if it shows again.

franciscovalentecastro · 2025-07-21T17:56:12Z

Mentioned in previous #1978 (comment) about the unrelated test failures to the changes in this PR. Both failures (Debian 11 frontend lock and rabbitmq in Ubuntu 24.04 are being addressed separately of this PR). Merging PR.

Caleb-Hurshman force-pushed the feat/nginx-otel-logging branch 3 times, most recently from ddcddd7 to 75a6108 Compare July 17, 2025 17:32

franciscovalentecastro self-requested a review July 17, 2025 17:58

franciscovalentecastro marked this pull request as ready for review July 17, 2025 17:59

franciscovalentecastro reviewed Jul 18, 2025

View reviewed changes

Caleb-Hurshman force-pushed the feat/nginx-otel-logging branch from 61a21c2 to 7e0f0fc Compare July 18, 2025 14:51

Caleb-Hurshman requested a review from franciscovalentecastro July 18, 2025 16:01

Caleb-Hurshman force-pushed the feat/nginx-otel-logging branch from 363bb43 to fad195e Compare July 18, 2025 16:02

franciscovalentecastro reviewed Jul 18, 2025

View reviewed changes

apps/common_logging_processors.go Outdated Show resolved Hide resolved

apps/common_logging_processors.go Outdated Show resolved Hide resolved

transformation_test/testdata/ops_agent_test-TestLogEntrySpecialFields/output_otel.yaml Outdated Show resolved Hide resolved

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

Caleb-Hurshman requested a review from franciscovalentecastro July 18, 2025 19:17

Caleb-Hurshman added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

Caleb-Hurshman force-pushed the feat/nginx-otel-logging branch from 88eed45 to 62f59ef Compare July 18, 2025 19:19

Caleb-Hurshman added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

franciscovalentecastro reviewed Jul 18, 2025

View reviewed changes

apps/common_logging_processors.go Outdated Show resolved Hide resolved

Caleb-Hurshman added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 18, 2025

Caleb-Hurshman and others added 6 commits July 21, 2025 10:38

nginx boilerplate

ddd005e

update transformation tests

c5a9f2e

test_confgenerator_update

a23c2ec

genericAccessLogParserV2

bfea147

update submodules, comments for genericAccessLogParsers

d84708a

rm fluent-bit params from genericAccessLogParserAsInternalLoggingProc…

018664c

…essor

Caleb-Hurshman force-pushed the feat/nginx-otel-logging branch from 05184e3 to 018664c Compare July 21, 2025 14:38

Caleb-Hurshman added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 21, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 21, 2025

franciscovalentecastro reviewed Jul 21, 2025

View reviewed changes

franciscovalentecastro approved these changes Jul 21, 2025

View reviewed changes

Caleb-Hurshman requested a review from franciscovalentecastro July 21, 2025 16:09

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 21, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jul 21, 2025

franciscovalentecastro merged commit b92a4e4 into GoogleCloudPlatform:master Jul 21, 2025
58 of 62 checks passed

franciscovalentecastro mentioned this pull request Jul 22, 2025

feat: Update Elasticsearch json parser to use LoggingReceiverMacro #1987

Merged

9 tasks

Caleb-Hurshman deleted the feat/nginx-otel-logging branch July 23, 2025 13:10

feat: Update nginx receiver to use LoggingReceiverMacro #1978

feat: Update nginx receiver to use LoggingReceiverMacro #1978

Uh oh!

Conversation

Caleb-Hurshman commented Jul 16, 2025

Description

Related issue

How has this been tested?

Checklist:

Uh oh!

google-cla bot commented Jul 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscovalentecastro Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Caleb-Hurshman Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscovalentecastro commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Caleb-Hurshman commented Jul 21, 2025

Uh oh!

franciscovalentecastro Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro left a comment

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro commented Jul 21, 2025

Uh oh!

franciscovalentecastro commented Jul 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: Update nginx receiver to use `LoggingReceiverMacro` #1978

feat: Update nginx receiver to use `LoggingReceiverMacro` #1978

franciscovalentecastro commented Jul 18, 2025 •

edited

Loading