-
Notifications
You must be signed in to change notification settings - Fork 765
[Feat] add user annotations to k8s objects #6710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feat] add user annotations to k8s objects #6710
Conversation
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
|
Thank you for opening this pull request! 🙌 These tips will help get your PR across the finish line:
|
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6710 +/- ##
==========================================
+ Coverage 56.91% 56.94% +0.03%
==========================================
Files 929 929
Lines 58077 58139 +62
==========================================
+ Hits 33052 33105 +53
- Misses 21984 21993 +9
Partials 3041 3041
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Hey @davidmirror-ops it looks like there's an issue with pulling helm on a bunch of the tests, is this expected? I'm working on updating the code coverage to get things green. |
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
|
@ttitsworth-lila yes, that's still a limitation on the current version of CI. Could you run |
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
|
@davidmirror-ops no problem, done. |
|
We do something similar internally since the principal is not passed from the control plane to the data plane, and thus can't be rendered into the k8s manifests in propeller. I will review this more deeply later. |
@Sovietaced Thanks for the comment. I'd love to understand more about what you mean. Presently I'm testing this in a multi-cluster setup where my above method is able to pass that information. What IDP provider do you use internally? |
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
Not sure what you mean with regards to IDP. What I'm trying to say is that flyte admin (the control plane) has the identity information when the user calls the API to create an execution. None of that information is passed down into the workflow resource and as a result is unusable by the flyte propeller (the data plane). So like you've done here we inject some of that information into the workflow annotations which isn't super elegant, but it works. Arguably the workflow spec should embed some information about the user identity in a more structured way. |
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
Like your suggestion @Sovietaced , but would require a lot more changes across the stack to include in the spec but the current PR can be considered incremental and adds a benefit for having identity info in the dataplane. |
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
|
Thanks for the review @pmahindrakar-oss, I just pushed a change to improve the coverage. I'm having some issues running Hope this is acceptable, and would appreciate some help with landing this PR :) |
The DNS issue looks like possible company VPN / firewall not resolving approved domains |
fixed it here #6726 |
|
Got some conflicts |
Signed-off-by: Tyler Titsworth <ttitsworth@lila.ai>
|
Bito Review Skipped - No Changes Detected |
machichima
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you so much for this! Left a few comments
@fg91 can you share the docs for that? Or just point me to where that is? |
This applies the label on the pod object though and I just realized you would like to have it on the flyteworkflow CR. |
Happy to consolidate to prevent duplication here, I'm just trying to understand how your PR works because I don't currently see anything related to |
No, you don't need to set anything in the pod template 🤔 If I'm not completely mistaken, the only requirement is that flyteadmin needs to have auth enabled but I assume you have that? |
I went and verified that exact key in our values file, and yes we have auth enabled. Obviously we need the auth-flow to work for my changes to function as well. Chart |
On the topic of consolidation, it's not obvious to me the best way to consolidate since the scope of this PR grew to include more than just the identity email. If we do anything, we could just keep both for backwards compatibility and let the community decide how to proceed. Thoughts? |
|
I think this is mostly fine btw, but agreed with @Sovietaced - I feel like there should be a more cohesive story around how identity is injected. And also how it can be used by the various plugins after it's been injected into the wf CRD. Are email/sub the only things that will ever need to be injected? Will anyone ever want to inject an entire token (for further downstream auth by plugins themselves)? |
Not all pieces of an identity should be injected IMO. The user deploying Flyte should have to explicitly choose what IDP labels they will expose to their propeller env(s).
From my perspective, yes. This is now giving contributors a pathway to add new labels for their IDP provider if they have their own use-case.
I had not considered this as a valid method to auth with plugins, it seems like there are security implications with that approach. I am also unaware as to what plugins would benefit from this approach. |
|
@davidmirror-ops It looks like some of the tests are still failing with errors I'm unsure about, how do we want to proceed with this PR? |
|
Bito Review Skipped - No Changes Detected |
I think you need to And I'll look into the PR that @fg91 mentioned, and see if we need to modify this PR to reuse the label. Thank you! |
|
I've tested out the PR that @fg91 mentioned (#4637). I think it's a bit different from what we want to do here. Update: What #4637 did is to add the from flytekit import Secret, task, workflow
@task(
secret_requests=[
Secret(group="testing-secret-1", key="username", mount_requirement=Secret.MountType.ENV_VAR),
Secret(group="testing-secret-2", key="api-token"),
]
)
def my_task_using_secrets():
pass
@workflow
def wf():
my_task_using_secrets()And the pod created will have the label and annotation. The ❯ k describe pod a7qswm5zm79mp6t25jz2-n0-0 -n flytesnacks-development
Name: a7qswm5zm79mp6t25jz2-n0-0
Namespace: flytesnacks-development
Priority: 0
Service Account: default
Node: cdb78fa419be/192.168.215.3
Start Time: Mon, 08 Dec 2025 15:54:30 +0800
Labels: domain=development
execution-id=a7qswm5zm79mp6t25jz2
inject-flyte-secrets=true # <- here
interruptible=false
node-id=n0
project=flytesnacks
shard-key=27
task-name=test-secret-my-task-using-secrets
workflow-name=test-secret-wf
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: false
# and following two flyte.secrets annotations
flyte.secrets/s0: m4zg54lqhiqce4dfon1gs2thfvzwky2smv1c1mjcbjvwk5j1earhk32fojxgc2lfeifg122vnz1f53tfof1ws3tfnvsw34b1ebcu3vs6kzavecq
flyte.secrets/s1: m4zg54lqhiqce4dfon1gs2thfvzwky2smv1c1mrcbjvwk5j1eargc3djfv1g512fnyrau
primary_container_name: a7qswm5zm79mp6t25jz2-n0-0cc @fg91 to confirm if it's correct? In this case I think it's ok to keep this PR as is. WDYT? |
Signed-off-by: Tyler Titsworth <ttitsworth@lila.ai>
|
@machichima I ran Editing the path unraveled the error a little bit more, but I'm just curious if I missed an obvious step. So I instead just ran |
|
@ttitsworth-lila I think you need to |
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
Thank you! Idk why I didn't consider that. |
machichima
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested this on my side and it works well
I think adding those info to the annotation to provide the observability toward who is creating the task LGTM.
Would probably need the final review from an expert to ensure we're following the best conventions for this.
cc @Sovietaced @fg91 @wild-endeavor @EngHabu
Signed-off-by: machichima <nary12321@gmail.com>
|
Congrats on merging your first pull request! 🎉 |
|
Bito Automatic Review Skipped – PR Already Merged |
Why are the changes needed?
Presently when submitting workflows via Flyte, it's hard to know who submitted what. This PR addresses one of two concerns:
Why do I/we care about 2? Let's say you have two teams (A & B), if team A has access to 10 resources and team B has access to 1000 resources and we know who is in each team (exclusively). One can create further admission controls for any user in team A that requests more resources than are allotted to that team. Furthermore, this can be tracked properly with Kueue.
Today when I submit a Flyte workflow I have no way to actually know who submitted what from a k8s perspective because principal information is not passed to the downstream k8s
flyteworkflowobject.What changes were proposed in this pull request?
This PR adds a parameter that enables a new annotation to be injected into each flyteworkflow object that contains the user
principal, which usually carries oauth information. Additionally, it allows customization of the annotation prefix.How was this patch tested?
I added some unit tests, and I also tested in production with the
flyte-corechart on EKS and using Keycloak:Labels
Please add one or more of the following labels to categorize your PR:
This is important to improve the readability of release notes.
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link
Summary by Bito