ensures support for cluster-wide monitoring - #21231
Conversation
|
@vmuzikar I misread the csv support, it already is marked as AllNamespaces unsupported. The tests were updated to include a realmimport as well. The controllers were updated to watch all by default to line up with env variable empty string value. For the tests to mimic the previous default behavior we'll default to JOSDK_WATCH_CURRENT if quarkus.operator-sdk.namespaces is unset. |
|
To keep this moving forward #15888 should be turned into an epic and there can be separate tasks for whatever seems necessary to claim cluster-wide support (version management, blast-radius, etc.). This pr simply allow the current operator logic to function across all namespaces, which would be good to maintain internal support for so that it's easier to externally support down the road. |
There was a problem hiding this comment.
This shouldn't be needed: quarkus.operator-sdk.namespaces is the value that should be used by default if a reconciler doesn't provide its own value.
There was a problem hiding this comment.
For 6.1.1 that did not seem to be happening. When the ControllerConfigurations were defaulting to WATCH_CURRENT_NAMESPACE setting quarkus.operator-sdk.namespaces=JOSDK_ALL_NAMESPACES had no effect.
edit: it's clarified below this is working as intended, it won't be used if the controller annotation has namespaces defined.
I'm confused as to what is not working… |
That could be the confusion - for the env property to work the controller cannot specify namespaces. In the case here we were defaulting to watch current. If you use the controller specific env properties it will override what is set on the annotation though. |
bfc63c0 to
8eb842d
Compare
Unreported flaky test detectedIf the below flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR. org.keycloak.testsuite.forms.ResetPasswordTest#resetPasswordWithSpnegoEnabledKeycloak CI - Forms IT (chrome) |
ghost
left a comment
There was a problem hiding this comment.
Unreported flaky test detected, please review
|
As discussed offline, let's hold this off after 22. |
|
@metacosm the situation here is that I'd like to keep refining this feature, but not yet expose it as a supported install mode. Once the namesapce defaults of the controller annotations are removed the generated kubernetes.yml will contain cluster role bindings that lack a namespace for their subjects, unless we set quarkus.kubernetes.namespace - but that is not how testing currently works, because that namespace is not fixed - it's generated for each test like with the KubernetesTest annotation. A couple of questions / thoughts:
|
Something I need to document for 6.2.0: we removed the duplication of the
I think the new property should help you there.
I have to admit that I'm not quite clear on how exactly this works in OLM. 😞 |
99c697a to
d3e6e08
Compare
|
Converted this to strickly be a test concern - it will not affect the kubernetes.yml, etc. until we determine how to best support all and multiple namespaces from a csv perspective. We can still wait to commit this until after 22. |
@metacosm from https://access.redhat.com/documentation/en-us/openshift_container_platform/4.11/html/operators/administrator-tasks it looks like you are supposed to associate the roles/clusterroles to a service account on the OperatorGroup, rather than having those in the csv. Then we'd have to provide two separate examples that support either single/own or multi/all. |
3ed91f6 to
c8d8dea
Compare
Thank you. An extension of the existing test scenario now utilizes this and a modification of the operatorgroup to test the all namespaces case. I ended up not using the parameterization of the olm scripts to do the testing as to avoid what would essentially be a duplicate setup. The proper way to do this would likely be to break of the bundle generation as one job, then create a callable github action job to perform the test for a given operator install namespace and watched namespaces. However that seemed like overkill compared to simply extending the existing test. |
There was a problem hiding this comment.
I'd like to propose an alternative way as I don't think we should be expecting OLM specific annotations by default. Feels a bit weird, IMHO. :)
Also setting -Dquarkus.kubernetes.annotations.\"olm.targetNamespaces\"=JOSDK_WATCH_CURRENT sets annotations on all generated resources (svc accounts, services, ...). I think it's not ideal.
We could try this instead:
- Set
quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENTin this properties file. This will keep current behavior where the Operator watches only current namespace by default. This is also what non-OLM installation resources should use. - Modify the OLM scripts to override the watched namespaces via env vars, i.e. set
QUARKUS_OPERATOR_SDK_NAMESPACESto reference theolm.targetNamespaceannotation in the CSV (and nowhere else). Even if it ends up being set to empty string, it should do the trick (observing all namespaces) as the env vars take precedence over properties file.
With this approach we'd make sure the default generated K8s resources (which are namespaced in any case – especially in the case of roles) are setting the Operator to watch current namespace without any additional configuration. At the same time, the OLM specific annotations would be used only in CSV.
WDYT?
There was a problem hiding this comment.
Set quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT in this properties file. This will keep current behavior where the Operator watches only current namespace by default.
I think my confusion here was that quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT seemed to be forcing the behavior in the image. However I just checked again and that's not the case. So it is possible to use that as a default, but that still isn't great - more below.
This is also what non-OLM installation resources should use.
What are these used for? It doesn't appear that quarkus.operator-sdk.namespaces alone will influence the creation of RoleBindings instead of ClusterRoleBindings. So you at least need to include a -Dquarkus.kubernetes.namespace so that the ClusterRoleBinding will be valid. Alternatively quarkus.operator-sdk.controllers.XXX.namespaces can be used, but they definitely do have an effect on the image - so to emit the same kubernetes.yaml with just RoleBindings would take a separate mvn package run specifying quarkus.operator-sdk.controllers.XXX.namespaces for each controller.
Modify the OLM scripts to override the watched namespaces via env vars, i.e. set QUARKUS_OPERATOR_SDK_NAMESPACES to reference the olm.targetNamespace annotation in the CSV (and nowhere else). Even if it ends up being set to empty string, it should do the trick (observing all namespaces) as the env vars take precedence over properties file.
In this pr the env property is already handled via the application.properties. My thought was that eventually it would be nice to move away from using create-olm-bundle.sh - that should cut down on the variance between what we're testing via the local/remote tests vs olm.
Beyond that we don't want the OLM scripts to directly set anything about the namespaces in the CSV - that's a job for OLM itself based upon the InstallMode/operatorgroup.
There was a problem hiding this comment.
I think my confusion here was that quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT seemed to be forcing the behavior in the image. However I just checked again and that's not the case. So it is possible to use that as a default, but that still isn't great - more below.
It is actually forcing the image behaviour but can be easily overridden via env vars.
What are these used for?
For namespaced installations without OLM.
It doesn't appear that quarkus.operator-sdk.namespaces alone will influence the creation of RoleBindings instead of ClusterRoleBindings. So you at least need to include a -Dquarkus.kubernetes.namespace so that the ClusterRoleBinding will be valid.
I'm not sure I fully follow. :) The generated kubernetes.yaml is namespaced installation in any case. The service account is namespaced as well. If I understand correctly, even the ClusterRoleBinding is then specific for that single service account in that namespace. What's the difference than from the current approach with just RoleBindings?
In this pr the env property is already handled via the application.properties.
My point is, I don't think we should be setting OLM specific annotations in application.properties as we don't always use OLM. We support installations without OLM in the upstream.
My thought was that eventually it would be nice to move away from using create-olm-bundle.sh - that should cut down on the variance between what we're testing via the local/remote tests vs olm.
+1
I think we could also use system properties during the operator build for OLM to set these env vars to avoid modifying the generated CSV via script.
Beyond that we don't want the OLM scripts to directly set anything about the namespaces in the CSV - that's a job for OLM itself based upon the InstallMode/operatorgroup.
We would not be. We'd just override the default (that we set via application.properties) to make Operator honor the annotations that are set by OLM.
There was a problem hiding this comment.
It is actually forcing the image behaviour but can be easily overridden via env vars.
By forcing I meant that it could not then be overriden via env vars.
I'm not sure I fully follow. :) The generated kubernetes.yaml is namespaced installation in any case. The service account is namespaced as well. If I understand correctly, even the ClusterRoleBinding is then specific for that single service account in that namespace. What's the difference than from the current approach with just RoleBindings?
Before any of the pending upstream changes if you remove the namespaces from the controller annotations but set quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT it will create ClusterRoleBindings in the kubernetes.yml:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: keycloakrealmimportcontroller-cluster-role-binding
roleRef:
kind: ClusterRole
apiGroup: rbac.authorization.k8s.io
name: keycloakrealmimportcontroller-cluster-role
subjects:
- kind: ServiceAccount
name: keycloak-operator
Note that the subject lacks a namespace, which makes it invalid. You have to also specify quarkus.kubernetes.namespace for the generated yaml to be valid.
There was a problem hiding this comment.
@shawkins I see. I got confused before, sorry. :) We need RoleBindings without any namespace specified then (as it is now). It almost feels like a JOSDK bug to me that quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT makes it generate ClusterRoleBindings. WDYT?
There was a problem hiding this comment.
It is, and that's one reason they've changed to specific build time properties instead, so the quarkus.operator-sdk.namespaces won't affect build time any more. It was also called out by quarkus as an anti-pattern to use the same property for build and run time configuration.
There was a problem hiding this comment.
Ah yes, it was fixed by quarkiverse/quarkus-operator-sdk#654.
Then I'd suggest waiting for that fix to be released before proceeding with this PR.
There was a problem hiding this comment.
Yes quarkiverse/quarkus-operator-sdk#655 and quarkiverse/quarkus-operator-sdk#653 were captured to address JOSDK issues.
Here's a recap of where things are currently:
- removing the namespaces from the controllers causes the kubernetes.yml to use ClusterRoleBindings.
- Using quarkus.operator-sdk.namespaces=JOSDK_WATCH_CURRENT - does not change the generation of ClusterRoleBindings.
- Using quarkus.operator-sdk.controllers.xxx.namespaces also does not change the generated kubernetes.yml - it would also need to be expressed as ${quarkus.operator-sdk.namespaces} otherwise it would prevent you from using the olm annotations to drive the namespaces.
- Using quarkus.operator-sdk..generate-with-watched-namespaces - does affect the kubernetes.yml, but it does appear to force the image to use only what is set - no matter how the property is set.
I had initially used quarkus.operator-sdk.controllers.xxx.namespaces the other direction - leave the annotation namespaces set on the controllers, then allow these properties to override things. The sticking point there however is there isn't a simplistic way to differentate between a missing annotation (non-olm installation) and an empty annotation (all namespaces case), so you have to leave the env field handling to the olm script and not the application.properties.
751f4e7 to
e0e3652
Compare
Partially addresses keycloak#15888
|
So now I'm basically back to the original pr, but with the addition of a config interceptor to handle the case of empty target namespaces for olm meaning watch all - it's not a quarkus issue with the empty property handling, it's a JOSDK issue. Basically defaulting to anything other than watch all makes setting watch all difficult. This is a little bit different that what you were looking at in our meeting @vmuzikar, but even closer to what was already there. Once the upstream fixes are in this will be much cleaner:
|
|
Would it make sense then to wait for the upstream fixes, especially quarkiverse/quarkus-operator-sdk#656? |
I could go either way. The workarounds here now work under the assumption that everything should function as it did before - there's no changes to the manifests, no changes to the mvn properties, etc. Things to review if we move forward before the josdk upstream changes:
Whether we move forward with the whole set of changes or not, the other bits here are:
|
| /** | ||
| * A workaround until the operator sdk supports a generate option that only affects the manifests | ||
| */ | ||
| @Priority(275) | ||
| public class ControllerConfigSourceInterceptor implements ConfigSourceInterceptor { | ||
| private static final long serialVersionUID = 367246512037404779L; | ||
|
|
||
| @Override | ||
| public ConfigValue getValue(final ConfigSourceInterceptorContext context, final String name) { | ||
| ConfigValue configValue = doLocked(() -> context.proceed(name)); | ||
| if (name.equals("quarkus.operator-sdk.namespaces") && Optional.ofNullable(configValue) | ||
| .filter(cv -> cv.getValue() == null || cv.getValue().isEmpty()).isPresent()) { | ||
| configValue = ConfigValue.builder().withName("quarkus.operator-sdk.namespaces") | ||
| .withValue("JOSDK_ALL_NAMESPACES").build(); | ||
| } | ||
| return configValue; | ||
| } | ||
| } No newline at end of file |
There was a problem hiding this comment.
@shawkins If I understand correctly, once quarkiverse/quarkus-operator-sdk#654 is released, we won't need this workaround. Correct? Then I'd wait with this PR, it's targeting Keycloak 23 in any case. WDYT?
|
Once the upstream is fixed, we won't need the config interceptor, the
config properties will be replaced with a single one, and the annotations
will have the namespaces removed. We can certainly hold off on committing
anything, or do you want the parameterization of the olm scripts as a
separate pr?
…On Wed, Jul 26, 2023, 5:17 AM Václav Muzikář ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In
operator/src/main/java/org/keycloak/operator/ControllerConfigSourceInterceptor.java
<#21231 (comment)>:
> +/**
+ * A workaround until the operator sdk supports a generate option that only affects the manifests
+ */
***@***.***(275)
+public class ControllerConfigSourceInterceptor implements ConfigSourceInterceptor {
+ private static final long serialVersionUID = 367246512037404779L;
+
+ @OverRide
+ public ConfigValue getValue(final ConfigSourceInterceptorContext context, final String name) {
+ ConfigValue configValue = doLocked(() -> context.proceed(name));
+ if (name.equals("quarkus.operator-sdk.namespaces") && Optional.ofNullable(configValue)
+ .filter(cv -> cv.getValue() == null || cv.getValue().isEmpty()).isPresent()) {
+ configValue = ConfigValue.builder().withName("quarkus.operator-sdk.namespaces")
+ .withValue("JOSDK_ALL_NAMESPACES").build();
+ }
+ return configValue;
+ }
+}
@shawkins <https://github.com/shawkins> If I understand correctly, once
quarkiverse/quarkus-operator-sdk#654
<quarkiverse/quarkus-operator-sdk#654> is
released, we won't need this workaround. Correct? Then I'd wait with this
PR, it's targeting Keycloak 23 in any case. WDYT?
—
Reply to this email directly, view it on GitHub
<#21231 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAS4NFNWZMCTELV6ZJQ72TDXSDOBLANCNFSM6AAAAAAZUOF4LE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@shawkins Ok, let's hold this PR off. I don't see any issue with doing the OLM scripts parametrization as part of this PR as it's closely related to cluster-wide monitoring. |
|
Sounds good.
…On Wed, Jul 26, 2023, 5:50 AM Václav Muzikář ***@***.***> wrote:
@shawkins <https://github.com/shawkins> Ok, let's hold this PR off.
I don't see any issue with doing the OLM scripts parametrization as part
of this PR as it's closely related to cluster-wide monitoring.
—
Reply to this email directly, view it on GitHub
<#21231 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAS4NFN43JXXRIGUXGXSVELXSDR5JANCNFSM6AAAAAAZUOF4LE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Upstream change has merged: quarkiverse/quarkus-operator-sdk#658 |
|
This is waiting for QOSDK 6.3.0 release to be adopted by Quarkus 3.2. |
The PR (quarkusio/quarkus-platform#936) has been merged and my understanding is that this should therefore be available in 3.2.5. |
|
Replaced by #22821 |
The only thing it is currently asserting is that a keycloak, which is not in the operator's namespace comes up successfully. This can of course be expanded - a test for watched secrets would likely be a good idea.
QUARKUS_OPERATOR_SDK_NAMESPACES, referencing the targetNamespaces annotation, does not appear to function as described in https://github.com/quarkiverse/quarkus-operator-sdk/blob/6885f71fa6fcb8780e873c952c92c2e71ae7e3ca/docs/modules/ROOT/pages/includes/quarkus-operator-sdk.adoc#quarkus-operator-sdk_quarkus.operator-sdk.namespaces - but that could be because it wasn't working until 6.2 quarkiverse/quarkus-operator-sdk@dfd857b
As a workaround that value is being used to set the namespaces for each controller via the application.properties.
What doesn't work well is the all namespaces case. That will currently resolve to an empty string env variable:
QUARKUS_OPERATOR_SDK_NAMESPACES=
As far as I can tell this results in the usage of the default from the annotations, which is watch the current namespace. The workaround is for the operator group to manually set the targetNamespaces to JOSDK_ALL_NAMESPACES. A potentially better alternative is to change the controller annotations to remove the namespaces, so that it defaults instead to watching everything - for usage via OLM that would not be a breaking change.
cc @metacosm
This does have conflict / additional considerations with other open prs that will be resolved once more are committed.
Partially addresses #15888