Skip to content

[transformation_test] Add possibility to test a receiver and verify transformations defined within. #1968

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 7 commits into from
Closed

[transformation_test] Add possibility to test a receiver and verify transformations defined within. #1968

wants to merge 7 commits into from

Conversation

@franciscovalentecastro
Copy link
Contributor

@franciscovalentecastro franciscovalentecastro commented Jul 3, 2025 •
edited
Loading

Description

Currently "transformations" done as post processing of receiver (e.g. windows_event_log, 3P app receivers) can't be tested directly in transformation tests, since it only accepts a list of processors.

This PR adds the possibility to test a receiver in the config.yaml file of a transformation test. The test will "substitute" the "INPUT" part of a test_receiver from the pipeline with a test files log receiver while keeping the "post-input" logging processors in the pipeline. The config.yaml should also provide an input_log_parser to determine how the input.log is going to be parsed to mock the "INPUT" data.

Sample config.yaml :

receiver:
  test_receiver:
    type: windows_event_log
  input_log_parser:
    type: parse_json
    time_key: system_time
    time_format: "%Y-%m-%dT%H:%M:%S.%L%z"

Related issue

b/429241517

How has this been tested?

Checklist:

  • Unit tests
    • Unit tests do not apply.
    • Unit tests have been added/modified and passed for this PR.
  • Integration tests
    • Integration tests do not apply.
    • Integration tests have been added/modified and passed for this PR.
  • Documentation
    • This PR introduces no user visible changes.
    • This PR introduces user visible changes and the corresponding documentation change has been made.
  • Minor version bump
    • This PR introduces no new features.
    • This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
    • This PR bumps the version.

avilevy18
avilevy18 reviewed Jul 3, 2025
View reviewed changes
transformation_test/transformation_test.go Outdated
Copy link
Contributor

@avilevy18 avilevy18 Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we turn this into a constant? Not sure if its possible in go

Copy link
Contributor Author

@franciscovalentecastro franciscovalentecastro Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now the transformationTest.Receiver is a pointer type, so we can just check for transformationTest.Receiver == nil instead of checking for an empty struct. Done!

transformation_test/transformation_test.go Outdated
Comment on lines 68 to 97
Copy link
Contributor

@avilevy18 avilevy18 Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we serialize this into a yaml file, that can be declared optionally in each test?

Something like, resource.yaml

Copy link
Contributor Author

@franciscovalentecastro franciscovalentecastro Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have considered serializing in a resource.yaml format, but since the use case is limited to choosing linux or windows platforms, i don't think it's worth it to implement here :

Some details on this :

  • The purpose of mocking a platform here is to unmarshall the config correctly since "Windows" receiver types are not supported in "Linux". There is no other direct usage of this platform.
  • The intention is to the "processors" (transformations) set within a "receiver" which are always platform agnostic.
  • Setting this platform constants is merely a mirror of how confgenerator_test.go sets constants for platforms which doesn't use an artifact like a resource.yaml file.

ops-agent/confgenerator/confgenerator_test.go

Lines 54 to 147 in 951f3dc

var winlogv1channels = []string{
"Application",
"Security",
"Setup",
"System",
}
var (
// Set up the test environment with mocked data.
testResource = resourcedetector.GCEResource{
Project: "test-project",
Zone: "test-zone",
Network: "test-network",
Subnetwork: "test-subnetwork",
PublicIP: "test-public-ip",
PrivateIP: "test-private-ip",
InstanceID: "test-instance-id",
InstanceName: "test-instance-name",
Tags: "test-tag",
MachineType: "test-machine-type",
Metadata: map[string]string{"test-key": "test-value", "test-escape": "$foo", "test-escape-parentheses": "${foo:bar}"},
Label: map[string]string{"test-label-key": "test-label-value"},
InterfaceIPv4: map[string]string{"test-interface": "test-interface-ipv4"},
ManagedInstanceGroup: gcp.ManagedInstanceGroup{
Name: "test-mig",
Type: gcp.Zone,
Location: "test-zone",
},
}
linuxTestPlatform = platformConfig{
name: "linux",
defaultLogsDir: "/var/log/google-cloud-ops-agent",
defaultStateDir: "/var/lib/google-cloud-ops-agent/fluent-bit",
platform: platform.Platform{
Type: platform.Linux,
HostInfo: &host.InfoStat{
OS: "linux",
Platform: "linux_platform",
PlatformVersion: "linux_platform_version",
},
TestGCEResourceOverride: testResource,
},
}
testPlatforms = []platformConfig{
linuxTestPlatform,
{
name: "linux-gpu",
defaultLogsDir: "/var/log/google-cloud-ops-agent",
defaultStateDir: "/var/lib/google-cloud-ops-agent/fluent-bit",
platform: platform.Platform{
Type: platform.Linux,
HostInfo: &host.InfoStat{
OS: "linux",
Platform: "linux_platform",
PlatformVersion: "linux_platform_version",
},
TestGCEResourceOverride: testResource,
HasNvidiaGpu: true,
},
},
{
name: "windows",
defaultLogsDir: `C:\ProgramData\Google\Cloud Operations\Ops Agent\log`,
defaultStateDir: `C:\ProgramData\Google\Cloud Operations\Ops Agent\run`,
platform: platform.Platform{
Type: platform.Windows,
WindowsBuildNumber: "1", // Is2012 == false, Is2016 == false
WinlogV1Channels: winlogv1channels,
HostInfo: &host.InfoStat{
OS: "windows",
Platform: "win_platform",
PlatformVersion: "win_platform_version",
},
TestGCEResourceOverride: testResource,
},
},
{
name: "windows-2012",
defaultLogsDir: `C:\ProgramData\Google\Cloud Operations\Ops Agent\log`,
defaultStateDir: `C:\ProgramData\Google\Cloud Operations\Ops Agent\run`,
platform: platform.Platform{
Type: platform.Windows,
WindowsBuildNumber: "9200", // Windows Server 2012
WinlogV1Channels: winlogv1channels,
HostInfo: &host.InfoStat{
OS: "windows",
Platform: "win_platform",
PlatformVersion: "win_platform_version",
},
TestGCEResourceOverride: testResource,
},
},
}
)

transformation_test/transformation_test.go Outdated
Copy link
Contributor

@avilevy18 avilevy18 Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Prefer const instead of "INPUT"

Copy link
Contributor Author

@franciscovalentecastro franciscovalentecastro Jul 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a const flbINPUT = "INPUT" for this comparison. See 006a9d3 . Done!

transformation_test/transformation_test.go Outdated
Copy link
Contributor

@avilevy18 avilevy18 Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Add comment that there always be 1 receiver, per test

Copy link
Contributor Author

@franciscovalentecastro franciscovalentecastro Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in fd41bfa. I've also added "test" validation for this. Done!

@franciscovalentecastro franciscovalentecastro changed the title [transformation_test] Enable tests to verify transformations defined in a logging receiver. [transformation_test] Add possibility to test a receiver and verify transformations defined within. Jul 3, 2025
@franciscovalentecastro franciscovalentecastro force-pushed the fcovalente-add-receivers-transformation-tests branch from 7a90db1 to 83c9789 Compare July 14, 2025 15:49
quentinmit
quentinmit reviewed Jul 15, 2025
View reviewed changes
transformation_test/transformation_test.go
return nil, fmt.Errorf("test_receiver is required.")
}
// Append logging processors from receiver "post-processing" step.
receiverComponents := transformationTest.Receiver.TestReceiver.LoggingReceiver.Components(ctx, flbTag)
Copy link
Member

@quentinmit quentinmit Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manipulating the otel/fluent-bit components after they're already constructed feels like a giant hack. Can't we achieve the same result without any hacks by refactoring the receiver-under-test into a receiver + processor that is only registered in tests, and then using that processor in a transformation test?

Copy link
Contributor Author

@franciscovalentecastro franciscovalentecastro Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment! I like the idea of registering processors only for the purpose of testing. This would avoid any need to update transformation_test.go. The current change present here seems to need further discussion in the design. I'll close this PR in favour of : #1977 .

We can discuss later if adding a receiver section is still needed.

@franciscovalentecastro franciscovalentecastro mentioned this pull request Jul 15, 2025
Merged
9 tasks
@franciscovalentecastro
Copy link
Contributor Author

franciscovalentecastro commented Jul 15, 2025

Closing in favour of #1977.

@franciscovalentecastro franciscovalentecastro mentioned this pull request Jul 31, 2025
Closed
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@quentinmit quentinmit quentinmit left review comments

@avilevy18 avilevy18 avilevy18 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@franciscovalentecastro @quentinmit @avilevy18