Skip to content

Conversation

@XuechunHou
Copy link
Contributor

@XuechunHou XuechunHou commented May 14, 2025

Description

  1. This PR pulls the latest otel upgrade into Ops Agent.
  2. This PR modifies the configs used 3p integration tests, replacing the plaintext secrets with the google secret manager provider identifier.

Related issue

b/417214181

How has this been tested?

integration tests

Checklist:

  • Unit tests
    • Unit tests do not apply.
    • Unit tests have been added/modified and passed for this PR.
  • Integration tests
    • Integration tests do not apply.
    • Integration tests have been added/modified and passed for this PR.
  • Documentation
    • This PR introduces no user visible changes.
    • This PR introduces user visible changes and the corresponding documentation change will be made.
  • Minor version bump
    • This PR introduces no new features.
    • This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
    • This PR bumps the version.

@XuechunHou XuechunHou added the kokoro:force-run Forces kokoro to run integration tests on a CL label May 14, 2025
@stackdriver-instrumentation-release stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label May 14, 2025
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this cause the test to fail when run on any other project?

Also, why is there not a $ at the start of the value?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When does the test run on other projects?

"Also, why is there not a $ at the start of the value?" -> fixed, ty.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When does the test run on other projects?

When it's run directly from a developer's workstation with go test

Copy link
Contributor Author

@XuechunHou XuechunHou May 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The secrets are accessible as long as the project hosting the secrets grants the roles/secretmanager.secretAccessor to the test VM Service Account.

If the test is not running from within a VM, then the deverlopers @google.com email needs to be granted the iam role.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And if the developer does not have an @google.com account? Ops Agent is open source software.

@XuechunHou XuechunHou force-pushed the secret-provider-in-oa branch from 047f2da to 47ce017 Compare May 14, 2025 19:56
@XuechunHou XuechunHou requested review from braydonk and jinghan-ma May 16, 2025 02:05
apps/oracledb.go Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably add a test for this case specifically.

apps/oracledb.go Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. Wouldn't we want secretPassword to start with ${googlesecretmanager? (i.e. HasPrefix)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we are trying to prevent users from using the secret provider in OracleDB-related configuration entries.

This is because the password will be URI-escaped before being passed to the OTel collector, which results in the provider identifier being altered and consequently unable to map back to a secret entry in Google Secret Manager.

We will remove this check later when we implement the fix in the sqlquery receiver.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't an upstream issue; an upstream user can and should just put an escaped string into the secret.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copied from my comment in the upstream issue:

I agree that the "ideal" solution is to have a sharable component that can be called to URL encode or escape sections of the configuration.

From a "where this belongs" perspective between the two existing components though, I think it's closer to the sql receiver than the secret manager. This escaping issue is not specific to the case of wanting to use the secret manager. It also applies to using env variable (in the example Braydon provided), or any other provider in the future.

The sql receiver also knows exactly what type of escaping it needs to do (in this case URL encoding).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on my previous comment, I don't think deciding whether to implement escaping in a new provider or sql receiver should be blocking for the Secret Manager integration.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also copied from the upstream issue:

This isn't a question of the "ideal" solution - the proposed solution here can not work. Once the string is already concatenated into a single URL, it's no longer possible to know which part of that string needs encoding applied to it.

Also, the exact same problem applies to environment variable substitution, not just googlesecretmanager, so if you want to report a configuration error until this is supported upstream, you should be checking for ${ and not just ${googlesecretmanager

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point. This would also not work with env variable substitution. Maybe we do a regex match for something like${.*}

Let's also change the error message to say that this integration does not support confmap providers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@XuechunHou XuechunHou added the kokoro:force-run Forces kokoro to run integration tests on a CL label May 16, 2025
@stackdriver-instrumentation-release stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label May 16, 2025
@XuechunHou XuechunHou force-pushed the secret-provider-in-oa branch from fd4e37a to 80c6391 Compare May 16, 2025 14:46
Copy link
Contributor

@braydonk braydonk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can leave it for this PR, but in the future a PR like this that does two distinctly different things (upgrades otelopscol, adds new test cases for googlesecretmanager) should be done in two separate PRs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was the \EOF added in every enable script here?

Copy link
Contributor Author

@XuechunHou XuechunHou May 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding \ so that the provider identifier is not interpreted as a shell variable.

This enable file content is eventually stored as a shell script in the test VM:
image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding \ so that the provider identifier is not interpreted as a shell variable.

This does not explain why EOF has a \, I'm still confused about this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change in this file (adding \ in front of EOF) has been reverted.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not a fan of the testing strategy in this PR because:

  1. It overrides the plaintext password testcases reducing our coverage
  2. It buries googlesecretmanager tests in random database app tests. It should really have its own dedicated test cases in the main test framework.

I think this should be redone in a way that addresses both of those points.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this, and I think it should be replaced with an integration test that also creates the secrets in the current project, so the test is not hardcoded to a single project.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that would make more sense given how other tests in this repo work, given that none of the others hardcode a project.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@XuechunHou XuechunHou self-assigned this May 17, 2025
@XuechunHou XuechunHou force-pushed the secret-provider-in-oa branch from 2a11942 to 5888926 Compare May 20, 2025 15:50
@XuechunHou XuechunHou force-pushed the secret-provider-in-oa branch from c8b25c5 to a4ed0da Compare May 20, 2025 18:35
@XuechunHou XuechunHou added the kokoro:force-run Forces kokoro to run integration tests on a CL label May 20, 2025
@stackdriver-instrumentation-release stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label May 20, 2025
@XuechunHou XuechunHou merged commit 1642181 into master May 21, 2025
74 checks passed
@XuechunHou XuechunHou deleted the secret-provider-in-oa branch May 21, 2025 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants