[confgenerator] Create `LoggingReceiverMacro` and `LoggingProcessorMacro` to generate 3p receivers implementation and update `flink`. #1949

franciscovalentecastro · 2025-05-12T14:27:59Z

Description

Created LoggingReceiverMacro and LoggingProcessorMacro to standardize implementation of 3rd party app logging receivers and processors. They work as a "macro" in the sense that after defining a reduced sub-agent (otel or fluent-bit) agnostic definition of a receiver or processor, the "macro" is "expanded/replaced" with a full implementation of a configurable LoggingReceiver or LoggingProcessor. Co-author @quentinmit.

A list of additions in this PR :

Created LoggingReceiverMacro, LoggingProcessorMacro to encapsulate the key elements of a LoggingReceiver, LoggingProcessor, respectively. The "macros" only need a sub-agent (otel or fluent-bit) agnostic definition of the key methods required by the Logging(Receiver|Processor). This definitions help to reduce the redundant code while defining a large number of 3p receivers.
Created RegisterLoggingReceiverMacro and RegisterLoggingProcessorMacro methods which "generate/expand/replace" the "Macro" definitions with a fully implemented LoggingReceiver of LoggingProcessor. Additionally RegisterLoggingFilesProcessorMacro was created to simplify the most common definition of 3P app receivers (files receiver + processors).
Created un-exported loggingReceiverMacroAdapter and loggingProcessorMacroAdapter which adapt the "macro" definitions with a fully implemented LoggingReceiver or LoggingProcessor.

Context

Based on the following :

Original draft/idea from @quentinmit : master...quentin-generic-receivers
Previous PR draft : [Draft] confgenerator : Add LoggingCompositeReceiver to standardize implementation of 3rd party app logging receivers. #1896

Related issue

b/401028449

How has this been tested?

Checklist:

Unit tests
- Current Unit Tests are passing.
- Unit tests do not apply.
- Unit tests have been added/modified and passed for this PR.
Integration tests
- Current Integration tests are passing.
- Integration tests do not apply.
- Integration tests have been added/modified and passed for this PR.
Documentation
- This PR introduces no user visible changes.
- This PR introduces user visible changes and the corresponding documentation change has been made.
Minor version bump
- This PR introduces no new features.
- This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
- This PR bumps the version.

quentinmit · 2025-05-14T19:01:50Z

confgenerator/config.go

 	Component
 	// Components returns fluentbit components that implement this processor.
 	// tag is the log tag that should be matched by those components, and uid is a string which should be used when needed to generate unique names.
 	Components(ctx context.Context, tag string, uid string) []fluentbit.Component


LoggingProcessor can now embed LoggingProcessorMixin instead of duplicating the signature (and you probably should move the comment into the Mixin type to preserve it

Done. I also embedded LoggingReceiverMixin in LoggingReceiver.

quentinmit · 2025-05-14T19:12:56Z

confgenerator/logging_receivers.go

 	LoggingReceiverTypes.RegisterType(func() LoggingReceiver { return &LoggingReceiverSystemd{} }, platform.Linux)
 }
+
+type LoggingCompositeReceiver[R LoggingReceiverMixin, P LoggingMultiProcessorMixin] struct {


I'm struggling a bit with all the similar type names
Do you think you could write a brief comment to describe the different types and how they fit together?

I added comments to the definitions of LoggingReceiverMixin, LoggingProcessorMixin, LoggingMultiProcessor and LoggingCompositeReceiver. This may need more work/added detail.

I also updated the PR description with more details about the types and naming used.

quentinmit · 2025-05-14T20:22:29Z

confgenerator/config.go

+}
+
+// LoggingReceiverMixin implements all the methods required to describe a logging receiver pipeline.
+type LoggingReceiverMixin interface {


LoggingReceiverMixin -> InternalLoggingReceiver

quentinmit · 2025-05-14T20:22:42Z

confgenerator/config.go

+}
+
+// LoggingProcessorMixin implements the methods required to define a logging receiver pipeline.
+type LoggingProcessorMixin interface {


LoggingProcessorMixin -> InternalLoggingProcessor

quentinmit · 2025-05-14T20:36:11Z

confgenerator/logging_processors.go

 }
+
+// LoggingMultiProcessorMixin implements the methods required to describe a pipeline with one or more log processors.
+type LoggingMultiProcessorMixin interface {


LoggingMultiProcessorMixin -> LoggingProcessorMacro, the idea being that it's a C-style macro to transform Ops Agent processor config into Ops Agent processor(s) config

quentinmit · 2025-05-14T20:37:45Z

apps/flink.go

+	confgenerator.LoggingReceiverTypes.RegisterType(func() confgenerator.LoggingReceiver {
+		return &confgenerator.LoggingCompositeReceiver[LoggingReceiverMixinFlink, LoggingMultiProcessorMixinFlink]{}
+	})
+	confgenerator.LoggingProcessorTypes.RegisterType(func() confgenerator.LoggingProcessor {


What if we add a .RegisterMacro method, to be used something like

confgenerator.LoggingProcessorTypes.RegisterMacro(&LoggingProcessorMacroFlink{})

Then that method can wrap the processor in a local type that turns it into a LoggingProcessor, but that wrapping type doesn't have to leak out into the rest of Ops Agent

Thank you for all the guidance, pair-programming and offline discussions.

Done! The methods implemented are currently :

RegisterLoggingCompositeReceiverMacro

RegisterLoggingProcessorMacro

quentinmit · 2025-05-14T20:38:10Z

confgenerator/logging_processors.go

+}
+
+// LoggingMultiProcessor represents a pipeline that consists of one or more log processors.
+type LoggingMultiProcessor[P LoggingMultiProcessorMixin] struct {


Let's see if we can make this an unexported internal type (see comment in flink.go)

This became loggingProcessorMacroAdapter.

quentinmit · 2025-05-14T20:39:41Z

confgenerator/logging_receivers.go

+// LoggingCompositeReceiver represents a pipeline that consists of one log receiver & one or more log processors.
+type LoggingCompositeReceiver[R LoggingReceiverMixin, P LoggingMultiProcessorMixin] struct {
+	ConfigComponent     `yaml:",inline"`
+	MultiProcessorMixin P `yaml:",inline"`


Why does this take a MultiProcessorMixin instead of just taking an InternalLoggingProcessor and deferring to that processor to construct the components?

I created a LoggingComponentMacro (which has a []InternalLoggingProcessor) struct to build a LoggingCompositeReceiver with the following two structs :

ops-agent/confgenerator/logging_macros.go

Lines 26 to 33 in 441081c

// LoggingComponentMacro is a logging component that generates other

// Ops Agent receiver and/or processors as its implementation.

type LoggingComponentMacro interface {

Type() string

// Processors returns slice of logging processors. This is an intermediate representation before sub-agent specific configurations.

Processors(ctx context.Context) []InternalLoggingProcessor

Receiver(ctx context.Context) InternalLoggingReceiver

}

ops-agent/confgenerator/logging_macros.go

Lines 89 to 93 in 441081c

// loggingCompositeReceiverMacroAdapter represents a pipeline that consists of one log receiver & one or more log processors.

type loggingCompositeReceiverMacroAdapter[LCM LoggingComponentMacro] struct {

ConfigComponent `yaml:",inline"`

ComponentMacro LCM `yaml:",inline"`

}

confgenerator/logging_processors.go

…ver` and `LoggingMultiProcessor` and their building blocks.

… in `LoggingReceiver` and `LoggingProcessor` respectively. Update comments.

…xpandedMacro`.

…ingCompositeReceiver

…and `LoggingCompositeReceiver`.

…Macro`.

…gingProcessorMacro`), and three helper registration functions to simplify app implementations

franciscovalentecastro requested a review from quentinmit May 12, 2025 14:39

franciscovalentecastro mentioned this pull request May 12, 2025

[Draft] Modify some 3p apps to use LoggingReceiverMacro. #1950

Closed

11 tasks

franciscovalentecastro force-pushed the fcovalente-composite-receiver branch from eee67b8 to 33d98ca Compare May 13, 2025 18:46

quentinmit reviewed May 14, 2025

View reviewed changes

franciscovalentecastro commented May 15, 2025

View reviewed changes

confgenerator/logging_processors.go Outdated Show resolved Hide resolved

franciscovalentecastro changed the title ~~[confgenerator] Create LoggingCompositeReceiver and update flink.~~ [confgenerator] Create LoggingComponentMacro to generate 3p receivers implementation and update flink. May 16, 2025

franciscovalentecastro changed the title ~~[confgenerator] Create LoggingComponentMacro to generate 3p receivers implementation and update flink.~~ [confgenerator] Create LoggingCompositeReceiverMacro to generate 3p receivers implementation and update flink. May 16, 2025

franciscovalentecastro changed the title ~~[confgenerator] Create LoggingCompositeReceiverMacro to generate 3p receivers implementation and update flink.~~ [confgenerator] Create Logging to generate 3p receivers implementation and update flink. May 20, 2025

franciscovalentecastro changed the title ~~[confgenerator] Create Logging to generate 3p receivers implementation and update flink.~~ [confgenerator] Create LoggingReceiverMacro to generate 3p receivers implementation and update flink. May 20, 2025

franciscovalentecastro changed the title ~~[confgenerator] Create LoggingReceiverMacro to generate 3p receivers implementation and update flink.~~ [confgenerator] Create LoggingReceiverMacro and LoggingProcessorMacro to generate 3p receivers implementation and update flink. May 20, 2025

franciscovalentecastro force-pushed the fcovalente-composite-receiver branch 4 times, most recently from ddd3dee to c3f087b Compare May 21, 2025 16:37

franciscovalentecastro and others added 15 commits May 21, 2025 17:13

Add flink transformation test.

4dfc507

Create LoggingCompositeReceiver and modify flink.

b227a29

[confgenerator] Add/update comments describing `LoggingCompositeRecei…

0679b8e

…ver` and `LoggingMultiProcessor` and their building blocks.

[confgenerator] Embed LoggingReceiverMixin, LoggingProcessorMixin…

158a0e2

… in `LoggingReceiver` and `LoggingProcessor` respectively. Update comments.

[confgenerator] Create LoggingProcessorMacro and `LoggingProcessorE…

e5f035b

…xpandedMacro`.

[confgenerator] Draft RegisterLogginProcessorMacro.

091cdc6

Implement RegisterLoggingProcessorMacro in terms of RegisterType

ac72961

Avoid duplicate Type field by using LoggingProcessorMacro inside Logg…

65515c8

…ingCompositeReceiver

Move macros to a separate file and use unexported types for adapters

4025ba8

Create LoggingComponentMacro to generate both a LoggingProcessor …

66efd91

…and `LoggingCompositeReceiver`.

Keep LoggingProcessorMacro as a subset of LoggingComponentMacro.

00c9352

Add Processors to loggingProcessorMacroAdapter.

5212a4b

Use LoggingCompositeReceiverMacro name instead of `LoggingComponent…

a1bab3f

…Macro`.

Introduce two consistent macro types (LoggingReceiverMacro and `Log…

5aa45ad

…gingProcessorMacro`), and three helper registration functions to simplify app implementations

Create loggingReceiverFilesMixinFlink and update comments.

ae4e1ed

franciscovalentecastro force-pushed the fcovalente-composite-receiver branch from c3f087b to ae4e1ed Compare May 21, 2025 17:13

Use blank import to get /apps package.

0a8754a

quentinmit approved these changes May 22, 2025

View reviewed changes

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label May 23, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label May 23, 2025

Merge branch 'master' into fcovalente-composite-receiver

9bff154

franciscovalentecastro merged commit db2f3f0 into master May 26, 2025
73 of 74 checks passed

franciscovalentecastro deleted the fcovalente-composite-receiver branch May 26, 2025 17:28

	// LoggingComponentMacro is a logging component that generates other
	// Ops Agent receiver and/or processors as its implementation.
	type LoggingComponentMacro interface {
	Type() string
	// Processors returns slice of logging processors. This is an intermediate representation before sub-agent specific configurations.
	Processors(ctx context.Context) []InternalLoggingProcessor
	Receiver(ctx context.Context) InternalLoggingReceiver
	}

	// loggingCompositeReceiverMacroAdapter represents a pipeline that consists of one log receiver & one or more log processors.
	type loggingCompositeReceiverMacroAdapter[LCM LoggingComponentMacro] struct {
	ConfigComponent `yaml:",inline"`
	ComponentMacro LCM `yaml:",inline"`
	}

[confgenerator] Create LoggingReceiverMacro and LoggingProcessorMacro to generate 3p receivers implementation and update flink. #1949

[confgenerator] Create LoggingReceiverMacro and LoggingProcessorMacro to generate 3p receivers implementation and update flink. #1949

Uh oh!

Conversation

franciscovalentecastro commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Context

Related issue

How has this been tested?

Checklist:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[confgenerator] Create `LoggingReceiverMacro` and `LoggingProcessorMacro` to generate 3p receivers implementation and update `flink`. #1949

[confgenerator] Create `LoggingReceiverMacro` and `LoggingProcessorMacro` to generate 3p receivers implementation and update `flink`. #1949

franciscovalentecastro commented May 12, 2025 •

edited

Loading

franciscovalentecastro May 14, 2025 •

edited

Loading

franciscovalentecastro May 16, 2025 •

edited

Loading