Skip to content

ensures support for cluster-wide monitoring - #21231

Closed
shawkins wants to merge 1 commit into
keycloak:mainfrom
shawkins:iss15888
Closed

ensures support for cluster-wide monitoring#21231
shawkins wants to merge 1 commit into
keycloak:mainfrom
shawkins:iss15888

Conversation

@shawkins

@shawkins shawkins commented Jun 26, 2023

Copy link
Copy Markdown
Contributor

The only thing it is currently asserting is that a keycloak, which is not in the operator's namespace comes up successfully. This can of course be expanded - a test for watched secrets would likely be a good idea.

QUARKUS_OPERATOR_SDK_NAMESPACES, referencing the targetNamespaces annotation, does not appear to function as described in https://github.com/quarkiverse/quarkus-operator-sdk/blob/6885f71fa6fcb8780e873c952c92c2e71ae7e3ca/docs/modules/ROOT/pages/includes/quarkus-operator-sdk.adoc#quarkus-operator-sdk_quarkus.operator-sdk.namespaces - but that could be because it wasn't working until 6.2 quarkiverse/quarkus-operator-sdk@dfd857b

As a workaround that value is being used to set the namespaces for each controller via the application.properties.

What doesn't work well is the all namespaces case. That will currently resolve to an empty string env variable:

QUARKUS_OPERATOR_SDK_NAMESPACES=

As far as I can tell this results in the usage of the default from the annotations, which is watch the current namespace. The workaround is for the operator group to manually set the targetNamespaces to JOSDK_ALL_NAMESPACES. A potentially better alternative is to change the controller annotations to remove the namespaces, so that it defaults instead to watching everything - for usage via OLM that would not be a breaking change.

cc @metacosm

This does have conflict / additional considerations with other open prs that will be resolved once more are committed.

Partially addresses #15888

@shawkins

Copy link
Copy Markdown
Contributor Author

@vmuzikar I misread the csv support, it already is marked as AllNamespaces unsupported. The tests were updated to include a realmimport as well. The controllers were updated to watch all by default to line up with env variable empty string value. For the tests to mimic the previous default behavior we'll default to JOSDK_WATCH_CURRENT if quarkus.operator-sdk.namespaces is unset.

@shawkins

Copy link
Copy Markdown
Contributor Author

To keep this moving forward #15888 should be turned into an epic and there can be separate tasks for whatever seems necessary to claim cluster-wide support (version management, blast-radius, etc.). This pr simply allow the current operator logic to function across all namespaces, which would be good to maintain internal support for so that it's easier to externally support down the road.

Comment thread operator/src/main/java/org/keycloak/operator/controllers/KeycloakController.java Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be needed: quarkus.operator-sdk.namespaces is the value that should be used by default if a reconciler doesn't provide its own value.

@shawkins shawkins Jun 29, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For 6.1.1 that did not seem to be happening. When the ControllerConfigurations were defaulting to WATCH_CURRENT_NAMESPACE setting quarkus.operator-sdk.namespaces=JOSDK_ALL_NAMESPACES had no effect.

edit: it's clarified below this is working as intended, it won't be used if the controller annotation has namespaces defined.

@metacosm

Copy link
Copy Markdown

The only thing it is currently asserting is that a keycloak, which is not in the operator's namespace comes up successfully. This can of course be expanded - a test for watched secrets would likely be a good idea.

QUARKUS_OPERATOR_SDK_NAMESPACES, referencing the targetNamespaces annotation, does not appear to function as described in https://github.com/quarkiverse/quarkus-operator-sdk/blob/6885f71fa6fcb8780e873c952c92c2e71ae7e3ca/docs/modules/ROOT/pages/includes/quarkus-operator-sdk.adoc#quarkus-operator-sdk_quarkus.operator-sdk.namespaces - but that could be because it wasn't working until 6.2 quarkiverse/quarkus-operator-sdk@dfd857b

As a workaround that value is being used to set the namespaces for each controller via the application.properties.

What doesn't work well is the all namespaces case. That will currently resolve to an empty string env variable:

QUARKUS_OPERATOR_SDK_NAMESPACES=

As far as I can tell this results in the usage of the default from the annotations, which is watch the current namespace. The workaround is for the operator group to manually set the targetNamespaces to JOSDK_ALL_NAMESPACES. A potentially better alternative is to change the controller annotations to remove the namespaces, so that it defaults instead to watching everything - for usage via OLM that would not be a breaking change.

cc @metacosm

I'm confused as to what is not working… QUARKUS_OPERATOR_SDK_NAMESPACES sets the default value for all controllers if they don't provide their own list of watched namespaces. If unset, and no controller sets a namespaces list, then all controllers should watch all namespaces by default. I've opened quarkiverse/quarkus-operator-sdk#635 to hopefully address some of the misunderstanding around this property.

@shawkins

Copy link
Copy Markdown
Contributor Author

QUARKUS_OPERATOR_SDK_NAMESPACES sets the default value for all controllers if they don't provide their own list of watched namespaces.

That could be the confusion - for the env property to work the controller cannot specify namespaces. In the case here we were defaulting to watch current. If you use the controller specific env properties it will override what is set on the annotation though.

@ghost

ghost commented Jun 29, 2023

Copy link
Copy Markdown

Unreported flaky test detected

If the below flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.ResetPasswordTest#resetPasswordWithSpnegoEnabled

Keycloak CI - Forms IT (chrome)

java.lang.AssertionError: Expected LoginPage but was  (https://localhost:8543/auth/realms/test/login-actions/authenticate?session_code=nvZODCku4ma9qEPXQo9ESuwObU-39nA3T2NiMgk64gM&execution=f2539056-b67a-4ff8-bfdc-88f85e3846a8&client_id=test-app&tab_id=GpSUKMgLWvk)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.keycloak.testsuite.pages.AbstractPage.assertCurrent(AbstractPage.java:47)
	at org.keycloak.testsuite.pages.LoginPage.open(LoginPage.java:236)
...

Report flaky test

@ghost ghost left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@vmuzikar

Copy link
Copy Markdown
Contributor

As discussed offline, let's hold this off after 22.

@vmuzikar vmuzikar added the status/hold PR should not be merged. On hold for later. label Jun 29, 2023
@shawkins

Copy link
Copy Markdown
Contributor Author

@metacosm the situation here is that I'd like to keep refining this feature, but not yet expose it as a supported install mode. Once the namesapce defaults of the controller annotations are removed the generated kubernetes.yml will contain cluster role bindings that lack a namespace for their subjects, unless we set quarkus.kubernetes.namespace - but that is not how testing currently works, because that namespace is not fixed - it's generated for each test like with the KubernetesTest annotation.

A couple of questions / thoughts:

  • I'd like to keep this a feature that is testable locally via a quarkus test so that the kubernetes.yml does not yet contain clusterrolebindings. It seems the way to do this is to revert to some of the older changes - that is leave the namespaces on the Controller annotations and use the application.properties to override when locally testing.
  • Is there an approach for an operator that supports both OwnNamespace and AllNamespaces, such that the creation of the ClusterRoleBindings is conditional on being in the AllNamespaces installmode? Or just by supporting AllNamespaces as an option are you required to have cluster wide permissions?

@metacosm

Copy link
Copy Markdown

@metacosm the situation here is that I'd like to keep refining this feature, but not yet expose it as a supported install mode. Once the namesapce defaults of the controller annotations are removed the generated kubernetes.yml will contain cluster role bindings that lack a namespace for their subjects, unless we set quarkus.kubernetes.namespace - but that is not how testing currently works, because that namespace is not fixed - it's generated for each test like with the KubernetesTest annotation.

Something I need to document for 6.2.0: we removed the duplication of the namespaces key that existed before: it was available both at build and run times, in particular to be able to generate manifests. Now, we have a more explicit configuration property to generate manifests using explicit namespaces (since the idea is that you'd be generating the manifests for a specific target environment for testing purposes) using the quarkus.operator-sdk.<controller name>.generate-with-watched-namespaces property.

A couple of questions / thoughts:

  • I'd like to keep this a feature that is testable locally via a quarkus test so that the kubernetes.yml does not yet contain clusterrolebindings. It seems the way to do this is to revert to some of the older changes - that is leave the namespaces on the Controller annotations and use the application.properties to override when locally testing.

I think the new property should help you there.

  • Is there an approach for an operator that supports both OwnNamespace and AllNamespaces, such that the creation of the ClusterRoleBindings is conditional on being in the AllNamespaces installmode? Or just by supporting AllNamespaces as an option are you required to have cluster wide permissions?

I have to admit that I'm not quite clear on how exactly this works in OLM. 😞

@shawkins
shawkins force-pushed the iss15888 branch 3 times, most recently from 99c697a to d3e6e08 Compare June 29, 2023 20:12
@shawkins

Copy link
Copy Markdown
Contributor Author

Converted this to strickly be a test concern - it will not affect the kubernetes.yml, etc. until we determine how to best support all and multiple namespaces from a csv perspective. We can still wait to commit this until after 22.

@shawkins

shawkins commented Jul 2, 2023

Copy link
Copy Markdown
Contributor Author

I have to admit that I'm not quite clear on how exactly this works in OLM. 😞

@metacosm from https://access.redhat.com/documentation/en-us/openshift_container_platform/4.11/html/operators/administrator-tasks it looks like you are supposed to associate the roles/clusterroles to a service account on the OperatorGroup, rather than having those in the csv. Then we'd have to provide two separate examples that support either single/own or multi/all.

Comment thread operator/src/main/java/org/keycloak/operator/controllers/WatchedSecretsStore.java Outdated
@shawkins

Copy link
Copy Markdown
Contributor Author

Yes, patching a CSV on a live operator will update the resources.

Thank you. An extension of the existing test scenario now utilizes this and a modification of the operatorgroup to test the all namespaces case.

I ended up not using the parameterization of the olm scripts to do the testing as to avoid what would essentially be a duplicate setup. The proper way to do this would likely be to break of the bundle generation as one job, then create a callable github action job to perform the test for a given operator install namespace and watched namespaces. However that seemed like overkill compared to simply extending the existing test.

Comment on lines 14 to 15

@vmuzikar vmuzikar Jul 19, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to propose an alternative way as I don't think we should be expecting OLM specific annotations by default. Feels a bit weird, IMHO. :)

Also setting -Dquarkus.kubernetes.annotations.\"olm.targetNamespaces\"=JOSDK_WATCH_CURRENT sets annotations on all generated resources (svc accounts, services, ...). I think it's not ideal.

We could try this instead:

  • Set quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT in this properties file. This will keep current behavior where the Operator watches only current namespace by default. This is also what non-OLM installation resources should use.
  • Modify the OLM scripts to override the watched namespaces via env vars, i.e. set QUARKUS_OPERATOR_SDK_NAMESPACES to reference the olm.targetNamespace annotation in the CSV (and nowhere else). Even if it ends up being set to empty string, it should do the trick (observing all namespaces) as the env vars take precedence over properties file.

With this approach we'd make sure the default generated K8s resources (which are namespaced in any case – especially in the case of roles) are setting the Operator to watch current namespace without any additional configuration. At the same time, the OLM specific annotations would be used only in CSV.

WDYT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT in this properties file. This will keep current behavior where the Operator watches only current namespace by default.

I think my confusion here was that quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT seemed to be forcing the behavior in the image. However I just checked again and that's not the case. So it is possible to use that as a default, but that still isn't great - more below.

This is also what non-OLM installation resources should use.

What are these used for? It doesn't appear that quarkus.operator-sdk.namespaces alone will influence the creation of RoleBindings instead of ClusterRoleBindings. So you at least need to include a -Dquarkus.kubernetes.namespace so that the ClusterRoleBinding will be valid. Alternatively quarkus.operator-sdk.controllers.XXX.namespaces can be used, but they definitely do have an effect on the image - so to emit the same kubernetes.yaml with just RoleBindings would take a separate mvn package run specifying quarkus.operator-sdk.controllers.XXX.namespaces for each controller.

Modify the OLM scripts to override the watched namespaces via env vars, i.e. set QUARKUS_OPERATOR_SDK_NAMESPACES to reference the olm.targetNamespace annotation in the CSV (and nowhere else). Even if it ends up being set to empty string, it should do the trick (observing all namespaces) as the env vars take precedence over properties file.

In this pr the env property is already handled via the application.properties. My thought was that eventually it would be nice to move away from using create-olm-bundle.sh - that should cut down on the variance between what we're testing via the local/remote tests vs olm.

Beyond that we don't want the OLM scripts to directly set anything about the namespaces in the CSV - that's a job for OLM itself based upon the InstallMode/operatorgroup.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think my confusion here was that quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT seemed to be forcing the behavior in the image. However I just checked again and that's not the case. So it is possible to use that as a default, but that still isn't great - more below.

It is actually forcing the image behaviour but can be easily overridden via env vars.

What are these used for?

For namespaced installations without OLM.

It doesn't appear that quarkus.operator-sdk.namespaces alone will influence the creation of RoleBindings instead of ClusterRoleBindings. So you at least need to include a -Dquarkus.kubernetes.namespace so that the ClusterRoleBinding will be valid.

I'm not sure I fully follow. :) The generated kubernetes.yaml is namespaced installation in any case. The service account is namespaced as well. If I understand correctly, even the ClusterRoleBinding is then specific for that single service account in that namespace. What's the difference than from the current approach with just RoleBindings?

In this pr the env property is already handled via the application.properties.

My point is, I don't think we should be setting OLM specific annotations in application.properties as we don't always use OLM. We support installations without OLM in the upstream.

My thought was that eventually it would be nice to move away from using create-olm-bundle.sh - that should cut down on the variance between what we're testing via the local/remote tests vs olm.

+1
I think we could also use system properties during the operator build for OLM to set these env vars to avoid modifying the generated CSV via script.

Beyond that we don't want the OLM scripts to directly set anything about the namespaces in the CSV - that's a job for OLM itself based upon the InstallMode/operatorgroup.

We would not be. We'd just override the default (that we set via application.properties) to make Operator honor the annotations that are set by OLM.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is actually forcing the image behaviour but can be easily overridden via env vars.

By forcing I meant that it could not then be overriden via env vars.

I'm not sure I fully follow. :) The generated kubernetes.yaml is namespaced installation in any case. The service account is namespaced as well. If I understand correctly, even the ClusterRoleBinding is then specific for that single service account in that namespace. What's the difference than from the current approach with just RoleBindings?

Before any of the pending upstream changes if you remove the namespaces from the controller annotations but set quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT it will create ClusterRoleBindings in the kubernetes.yml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keycloakrealmimportcontroller-cluster-role-binding
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: keycloakrealmimportcontroller-cluster-role
subjects:
  - kind: ServiceAccount
    name: keycloak-operator

Note that the subject lacks a namespace, which makes it invalid. You have to also specify quarkus.kubernetes.namespace for the generated yaml to be valid.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shawkins I see. I got confused before, sorry. :) We need RoleBindings without any namespace specified then (as it is now). It almost feels like a JOSDK bug to me that quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT makes it generate ClusterRoleBindings. WDYT?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is, and that's one reason they've changed to specific build time properties instead, so the quarkus.operator-sdk.namespaces won't affect build time any more. It was also called out by quarkus as an anti-pattern to use the same property for build and run time configuration.

@vmuzikar vmuzikar Jul 20, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, it was fixed by quarkiverse/quarkus-operator-sdk#654.

Then I'd suggest waiting for that fix to be released before proceeding with this PR.

@shawkins shawkins Jul 20, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes quarkiverse/quarkus-operator-sdk#655 and quarkiverse/quarkus-operator-sdk#653 were captured to address JOSDK issues.

Here's a recap of where things are currently:

  • removing the namespaces from the controllers causes the kubernetes.yml to use ClusterRoleBindings.
    • Using quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT - does not change the generation of ClusterRoleBindings.
    • Using quarkus.operator-sdk.controllers.xxx.namespaces also does not change the generated kubernetes.yml - it would also need to be expressed as ${quarkus.operator-sdk.namespaces} otherwise it would prevent you from using the olm annotations to drive the namespaces.
    • Using quarkus.operator-sdk..generate-with-watched-namespaces - does affect the kubernetes.yml, but it does appear to force the image to use only what is set - no matter how the property is set.

I had initially used quarkus.operator-sdk.controllers.xxx.namespaces the other direction - leave the annotation namespaces set on the controllers, then allow these properties to override things. The sticking point there however is there isn't a simplistic way to differentate between a missing annotation (non-olm installation) and an empty annotation (all namespaces case), so you have to leave the env field handling to the olm script and not the application.properties.

@shawkins

Copy link
Copy Markdown
Contributor Author

So now I'm basically back to the original pr, but with the addition of a config interceptor to handle the case of empty target namespaces for olm meaning watch all - it's not a quarkus issue with the empty property handling, it's a JOSDK issue. Basically defaulting to anything other than watch all makes setting watch all difficult.

This is a little bit different that what you were looking at in our meeting @vmuzikar, but even closer to what was already there.

Once the upstream fixes are in this will be much cleaner:

  • we can remove the namespaces from the controller annotations
  • use a generate property to default the manifests to watch current
  • removes the need for the config interceptor, unless we want to support always adding the olm env property to the manifests

@vmuzikar

Copy link
Copy Markdown
Contributor

Would it make sense then to wait for the upstream fixes, especially quarkiverse/quarkus-operator-sdk#656?

@shawkins

Copy link
Copy Markdown
Contributor Author

Would it make sense then to wait for the upstream fixes, especially quarkiverse/quarkus-operator-sdk#656?

I could go either way. The workarounds here now work under the assumption that everything should function as it did before - there's no changes to the manifests, no changes to the mvn properties, etc.

Things to review if we move forward before the josdk upstream changes:

  • the workaround requires that we remember to add an override for each new controller in the application properties to allow it to work in anything other than current namespace

Whether we move forward with the whole set of changes or not, the other bits here are:

  • moves the src/main/resources/example files to src/test/resources
  • parameterizes the olm-test scripts - the olm testing added here isn't actually using that, so we don't have to keep it. It may also be better to more effectively split off the bundle generation step from the creation / installation of the operatorgroup/subscription so that piece could be done separately.

Comment on lines +30 to +47
/**
* A workaround until the operator sdk supports a generate option that only affects the manifests
*/
@Priority(275)
public class ControllerConfigSourceInterceptor implements ConfigSourceInterceptor {
private static final long serialVersionUID = 367246512037404779L;

@Override
public ConfigValue getValue(final ConfigSourceInterceptorContext context, final String name) {
ConfigValue configValue = doLocked(() -> context.proceed(name));
if (name.equals("quarkus.operator-sdk.namespaces") && Optional.ofNullable(configValue)
.filter(cv -> cv.getValue() == null || cv.getValue().isEmpty()).isPresent()) {
configValue = ConfigValue.builder().withName("quarkus.operator-sdk.namespaces")
.withValue("JOSDK_ALL_NAMESPACES").build();
}
return configValue;
}
} No newline at end of file

@vmuzikar vmuzikar Jul 26, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shawkins If I understand correctly, once quarkiverse/quarkus-operator-sdk#654 is released, we won't need this workaround. Correct? Then I'd wait with this PR, it's targeting Keycloak 23 in any case. WDYT?

@shawkins

shawkins commented Jul 26, 2023 via email

Copy link
Copy Markdown
Contributor Author

@vmuzikar

Copy link
Copy Markdown
Contributor

@shawkins Ok, let's hold this PR off.

I don't see any issue with doing the OLM scripts parametrization as part of this PR as it's closely related to cluster-wide monitoring.

@shawkins

shawkins commented Jul 26, 2023 via email

Copy link
Copy Markdown
Contributor Author

@Jamstah

Jamstah commented Aug 11, 2023

Copy link
Copy Markdown
Contributor

Upstream change has merged: quarkiverse/quarkus-operator-sdk#658

@vmuzikar vmuzikar linked an issue Aug 23, 2023 that may be closed by this pull request
@vmuzikar

Copy link
Copy Markdown
Contributor

This is waiting for QOSDK 6.3.0 release to be adopted by Quarkus 3.2.

@metacosm

Copy link
Copy Markdown

This is waiting for QOSDK 6.3.0 release to be adopted by Quarkus 3.2.

The PR (quarkusio/quarkus-platform#936) has been merged and my understanding is that this should therefore be available in 3.2.5.

@shawkins

Copy link
Copy Markdown
Contributor Author

Replaced by #22821

@shawkins shawkins closed this Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flaky-test status/hold PR should not be merged. On hold for later. team/cloud-native

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Descoped mode for the Operator

5 participants