Skip to content

Conversation

@ttitsworth-lila
Copy link
Contributor

@ttitsworth-lila ttitsworth-lila commented Oct 28, 2025

Why are the changes needed?

Presently when submitting workflows via Flyte, it's hard to know who submitted what. This PR addresses one of two concerns:

  1. I can't filter by users in the Flyte Console
  2. I can't see a user identifier in k8s

Why do I/we care about 2? Let's say you have two teams (A & B), if team A has access to 10 resources and team B has access to 1000 resources and we know who is in each team (exclusively). One can create further admission controls for any user in team A that requests more resources than are allotted to that team. Furthermore, this can be tracked properly with Kueue.

Today when I submit a Flyte workflow I have no way to actually know who submitted what from a k8s perspective because principal information is not passed to the downstream k8s flyteworkflow object.

What changes were proposed in this pull request?

This PR adds a parameter that enables a new annotation to be injected into each flyteworkflow object that contains the user principal, which usually carries oauth information. Additionally, it allows customization of the annotation prefix.

How was this patch tested?

I added some unit tests, and I also tested in production with the flyte-core chart on EKS and using Keycloak:

$ kubectl describe flyteworkflow -n flyte-workspace ax5hwpvbmwnkhdkmq67m | grep flyte.org
Annotations:  flyte.org/user: ttitsworth@lila.ai

Labels

Please add one or more of the following labels to categorize your PR:

  • added: For new features.
  • changed: For changes in existing functionality.
  • deprecated: For soon-to-be-removed features.
  • removed: For features being removed.
  • fixed: For any bug fixed.
  • security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Summary by Bito

  • This pull request introduces a feature for injecting user annotations into Flyte workflows, enhancing visibility and accountability in multi-team environments, while also addressing potential risks related to data integrity and security in Kubernetes environments.
  • It updates the identity annotation prefix to 'flyte.org' and includes customization options for better tracking of user submissions.
  • The changes improve error handling in the execution manager, ensuring better resource management and security.
  • Configuration updates include new identity annotation fields and modifications to deployment configurations, aiding in resource management.
  • Unit tests have been added to ensure the functionality works as intended, validating the new features.
  • Overall, this update introduces new user annotation features, customization options, and enhancements in resource management and security.

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@welcome
Copy link

welcome bot commented Oct 28, 2025

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
  • Sign off your commits (Reference: DCO Guide).

@ttitsworth-lila ttitsworth-lila changed the title create user annotations feature in flyteadmin add user annotations to k8s objects Oct 28, 2025
Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@ttitsworth-lila ttitsworth-lila changed the title add user annotations to k8s objects [Feat] add user annotations to k8s objects Oct 29, 2025
@codecov
Copy link

codecov bot commented Oct 29, 2025

Codecov Report

❌ Patch coverage is 87.87879% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.94%. Comparing base (99f2ddd) to head (31a6c87).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...kg/runtime/interfaces/application_configuration.go 0.00% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6710      +/-   ##
==========================================
+ Coverage   56.91%   56.94%   +0.03%     
==========================================
  Files         929      929              
  Lines       58077    58139      +62     
==========================================
+ Hits        33052    33105      +53     
- Misses      21984    21993       +9     
  Partials     3041     3041              
Flag Coverage Δ
unittests-datacatalog 53.51% <ø> (ø)
unittests-flyteadmin 53.14% <87.87%> (+0.12%) ⬆️
unittests-flytecopilot 43.06% <ø> (ø)
unittests-flytectl 64.02% <ø> (ø)
unittests-flyteidl 75.71% <ø> (ø)
unittests-flyteplugins 60.13% <ø> (ø)
unittests-flytepropeller 53.54% <ø> (ø)
unittests-flytestdlib 63.27% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@davidmirror-ops davidmirror-ops added the added Merged changes that add new functionality label Oct 29, 2025
@ttitsworth-lila
Copy link
Contributor Author

Hey @davidmirror-ops it looks like there's an issue with pulling helm on a bunch of the tests, is this expected?

I'm working on updating the code coverage to get things green.

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@davidmirror-ops
Copy link
Contributor

@ttitsworth-lila yes, that's still a limitation on the current version of CI. Could you run make helm locally and push the changes?

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@ttitsworth-lila
Copy link
Contributor Author

@davidmirror-ops no problem, done.

@Sovietaced
Copy link
Member

Sovietaced commented Nov 3, 2025

We do something similar internally since the principal is not passed from the control plane to the data plane, and thus can't be rendered into the k8s manifests in propeller. I will review this more deeply later.

@ttitsworth-lila
Copy link
Contributor Author

We do something similar internally since the principal is not passed from the control plane to the data plane, and thus can't be rendered into the k8s manifests in propeller. I will review this more deeply later.

@Sovietaced Thanks for the comment. I'd love to understand more about what you mean. Presently I'm testing this in a multi-cluster setup where my above method is able to pass that information. What IDP provider do you use internally?

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@Sovietaced
Copy link
Member

Sovietaced commented Nov 6, 2025

@Sovietaced Thanks for the comment. I'd love to understand more about what you mean. Presently I'm testing this in a multi-cluster setup where my above method is able to pass that information. What IDP provider do you use internally?

Not sure what you mean with regards to IDP. What I'm trying to say is that flyte admin (the control plane) has the identity information when the user calls the API to create an execution. None of that information is passed down into the workflow resource and as a result is unusable by the flyte propeller (the data plane). So like you've done here we inject some of that information into the workflow annotations which isn't super elegant, but it works.

Arguably the workflow spec should embed some information about the user identity in a more structured way.

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@pmahindrakar-oss
Copy link
Contributor

So like you've done here we inject some of that information into the workflow annotations which isn't super elegant, but it works.
Arguably the workflow spec should embed some information about the user identity in a more structured way.

Like your suggestion @Sovietaced , but would require a lot more changes across the stack to include in the spec but the current PR can be considered incremental and adds a benefit for having identity info in the dataplane.

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@ttitsworth-lila
Copy link
Contributor Author

Thanks for the review @pmahindrakar-oss, I just pushed a change to improve the coverage. I'm having some issues running helm make:

Save error occurred:  could not find : chart docker-registry not found in https://helm.twun.io/: looks like "https://helm.twun.io/" is not a valid chart repository or cannot be reached: Get "https://helm.twun.io/index.yaml": dial tcp: lookup helm.twun.io on 10.255.255.254:53: no such host

Hope this is acceptable, and would appreciate some help with landing this PR :)

@Sovietaced
Copy link
Member

Thanks for the review @pmahindrakar-oss, I just pushed a change to improve the coverage. I'm having some issues running helm make:


Save error occurred:  could not find : chart docker-registry not found in https://helm.twun.io/: looks like "https://helm.twun.io/" is not a valid chart repository or cannot be reached: Get "https://helm.twun.io/index.yaml": dial tcp: lookup helm.twun.io on 10.255.255.254:53: no such host

Hope this is acceptable, and would appreciate some help with landing this PR :)

The DNS issue looks like possible company VPN / firewall not resolving approved domains

@pingsutw
Copy link
Member

The DNS issue looks like possible company VPN / firewall not resolving approved domains

fixed it here #6726

@Sovietaced
Copy link
Member

Got some conflicts

Signed-off-by: Tyler Titsworth <ttitsworth@lila.ai>
@flyte-bot
Copy link
Collaborator

Bito Review Skipped - No Changes Detected

Bito didn't review this pull request because we did not detect any changes in the pull request to review.

Copy link
Member

@machichima machichima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for this! Left a few comments

@ttitsworth-lila
Copy link
Contributor Author

There already is a feature to inject the user identity as a label :)

@fg91 can you share the docs for that? Or just point me to where that is?

@fg91
Copy link
Member

fg91 commented Nov 24, 2025

There already is a feature to inject the user identity as a label :)

@fg91 can you share the docs for that? Or just point me to where that is?

#4637

This applies the label on the pod object though and I just realized you would like to have it on the flyteworkflow CR.
Maybe we could reuse the same label though instead of adding new annotation with a different name?

@ttitsworth-lila
Copy link
Contributor Author

There already is a feature to inject the user identity as a label :)

@fg91 can you share the docs for that? Or just point me to where that is?

#4637

This applies the label on the pod object though and I just realized you would like to have it on the flyteworkflow CR. Maybe we could reuse the same label though instead of adding new annotation with a different name?

Happy to consolidate to prevent duplication here, I'm just trying to understand how your PR works because I don't currently see anything related to labels.execution_identity, does this label only appear when you set the execution identity in the podtemplate?

@fg91
Copy link
Member

fg91 commented Nov 24, 2025

There already is a feature to inject the user identity as a label :)

@fg91 can you share the docs for that? Or just point me to where that is?

#4637
This applies the label on the pod object though and I just realized you would like to have it on the flyteworkflow CR. Maybe we could reuse the same label though instead of adding new annotation with a different name?

Happy to consolidate to prevent duplication here, I'm just trying to understand how your PR works because I don't currently see anything related to labels.execution_identity, does this label only appear when you set the execution identity in the podtemplate?

No, you don't need to set anything in the pod template 🤔 If I'm not completely mistaken, the only requirement is that flyteadmin needs to have auth enabled but I assume you have that?

@ttitsworth-lila
Copy link
Contributor Author

There already is a feature to inject the user identity as a label :)

@fg91 can you share the docs for that? Or just point me to where that is?

#4637
This applies the label on the pod object though and I just realized you would like to have it on the flyteworkflow CR. Maybe we could reuse the same label though instead of adding new annotation with a different name?

Happy to consolidate to prevent duplication here, I'm just trying to understand how your PR works because I don't currently see anything related to labels.execution_identity, does this label only appear when you set the execution identity in the podtemplate?

No, you don't need to set anything in the pod template 🤔 If I'm not completely mistaken, the only requirement is that flyteadmin needs to have auth enabled but I assume you have that?

I went and verified that exact key in our values file, and yes we have auth enabled. Obviously we need the auth-flow to work for my changes to function as well. Chart flyte-core-v1.15.3. I'm wondering if we could get a third party like @machichima or @pmahindrakar-oss to help verify if this feature @fg91 introduced is functioning? I can only assume it's working for your team @fg91!

@ttitsworth-lila
Copy link
Contributor Author

There already is a feature to inject the user identity as a label :)

@fg91 can you share the docs for that? Or just point me to where that is?

#4637

This applies the label on the pod object though and I just realized you would like to have it on the flyteworkflow CR. Maybe we could reuse the same label though instead of adding new annotation with a different name?

On the topic of consolidation, it's not obvious to me the best way to consolidate since the scope of this PR grew to include more than just the identity email. If we do anything, we could just keep both for backwards compatibility and let the community decide how to proceed. Thoughts?

@wild-endeavor
Copy link
Contributor

I think this is mostly fine btw, but agreed with @Sovietaced - I feel like there should be a more cohesive story around how identity is injected. And also how it can be used by the various plugins after it's been injected into the wf CRD. Are email/sub the only things that will ever need to be injected? Will anyone ever want to inject an entire token (for further downstream auth by plugins themselves)?

@ttitsworth-lila
Copy link
Contributor Author

I feel like there should be a more cohesive story around how identity is injected.

Not all pieces of an identity should be injected IMO. The user deploying Flyte should have to explicitly choose what IDP labels they will expose to their propeller env(s).

Are email/sub the only things that will ever need to be injected?

From my perspective, yes. This is now giving contributors a pathway to add new labels for their IDP provider if they have their own use-case.

Will anyone ever want to inject an entire token (for further downstream auth by plugins themselves)?

I had not considered this as a valid method to auth with plugins, it seems like there are security implications with that approach. I am also unaware as to what plugins would benefit from this approach.

@ttitsworth-lila
Copy link
Contributor Author

@davidmirror-ops It looks like some of the tests are still failing with errors I'm unsure about, how do we want to proceed with this PR?

@flyte-bot
Copy link
Collaborator

Bito Review Skipped - No Changes Detected

Bito didn't review this pull request because we did not detect any changes in the pull request to review.

@machichima
Copy link
Member

@davidmirror-ops It looks like some of the tests are still failing with errors I'm unsure about, how do we want to proceed with this PR?

I think you need to make helm again and push. Also it seems like there's some lint error, you can run make lint locally to fix.
Seems like tests are passed.
Feel free to ping me after you finished, and I can re-trigger the workflow for you.

And I'll look into the PR that @fg91 mentioned, and see if we need to modify this PR to reuse the label. Thank you!

@machichima
Copy link
Member

machichima commented Dec 8, 2025

I've tested out the PR that @fg91 mentioned (#4637). I think it's a bit different from what we want to do here.

Update:
I can see the execution-identity label on my side, which only shows the subject. I think for showing different info other than subject (e.g. email), the current approach looks better for me.

image

What #4637 did is to add the Secret set in the task to annotations. I tested with following flytekit script:

from flytekit import Secret, task, workflow

@task(
    secret_requests=[
        Secret(group="testing-secret-1", key="username", mount_requirement=Secret.MountType.ENV_VAR),
        Secret(group="testing-secret-2", key="api-token"),
    ]
)
def my_task_using_secrets():
    pass

@workflow
def wf():
    my_task_using_secrets()

And the pod created will have the label and annotation. The flyte.secrets/s0 value is the encoded string for the Secret setting we provided in the task

❯ k describe pod a7qswm5zm79mp6t25jz2-n0-0 -n flytesnacks-development
Name:             a7qswm5zm79mp6t25jz2-n0-0
Namespace:        flytesnacks-development
Priority:         0
Service Account:  default
Node:             cdb78fa419be/192.168.215.3
Start Time:       Mon, 08 Dec 2025 15:54:30 +0800
Labels:           domain=development
                  execution-id=a7qswm5zm79mp6t25jz2
                  inject-flyte-secrets=true   # <- here
                  interruptible=false
                  node-id=n0
                  project=flytesnacks
                  shard-key=27
                  task-name=test-secret-my-task-using-secrets
                  workflow-name=test-secret-wf
Annotations:      cluster-autoscaler.kubernetes.io/safe-to-evict: false
                  # and following two flyte.secrets annotations              
                  flyte.secrets/s0: m4zg54lqhiqce4dfon1gs2thfvzwky2smv1c1mjcbjvwk5j1earhk32fojxgc2lfeifg122vnz1f53tfof1ws3tfnvsw34b1ebcu3vs6kzavecq
                  flyte.secrets/s1: m4zg54lqhiqce4dfon1gs2thfvzwky2smv1c1mrcbjvwk5j1eargc3djfv1g512fnyrau
                  primary_container_name: a7qswm5zm79mp6t25jz2-n0-0

cc @fg91 to confirm if it's correct? In this case I think it's ok to keep this PR as is. WDYT?

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
Signed-off-by: Tyler Titsworth <ttitsworth@lila.ai>
@ttitsworth-lila
Copy link
Contributor Author

ttitsworth-lila commented Dec 8, 2025

@machichima I ran make helm just fine, but make lint didn't work. I did a bit of looking in the docs and files to see if I'm missing anything, but I just did a fresh clone to confirm that for whatever reason make lint fails on the current default branch:

$ make lint
Makefile:130: warning: overriding recipe for target 'go-tidy'
boilerplate/flyte/golang_test_targets/Makefile:59: warning: ignoring old recipe for target 'go-tidy'
make[1]: Entering directory '/home/user/flyte'
make[1]: *** /flytestdlib: No such file or directory.  Stop.
make[1]: Leaving directory '/home/user/flyte'
make: *** [boilerplate/flyte/golang_test_targets/Makefile:10: download_tooling] Error 2

Editing the path unraveled the error a little bit more, but I'm just curious if I missed an obvious step. So I instead just ran golangci-lint directly and found some linting errors in the unrelated cmd directory.

INFO [runner] linters took 29.799047289s with stages: goanalysis_metalinter: 29.796747133s 
cmd/single/console.go:29:6: type consoleFS is unused (unused)
type consoleFS struct {
     ^
cmd/single/console.go:35:20: func consoleFS.Open is unused (unused)
func (f consoleFS) Open(name string) (http.File, error) {
                   ^
cmd/single/without_console_dist.go:11:5: var console is unused (unused)
var console fs.FS
    ^
3 issues:
* unused: 3

@machichima
Copy link
Member

@ttitsworth-lila I think you need to cd flyteadmin then do make lint

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
@ttitsworth-lila
Copy link
Contributor Author

@ttitsworth-lila I think you need to cd flyteadmin then do make lint

Thank you! Idk why I didn't consider that.

machichima
machichima previously approved these changes Dec 10, 2025
Copy link
Member

@machichima machichima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this on my side and it works well

image

I think adding those info to the annotation to provide the observability toward who is creating the task LGTM.

Would probably need the final review from an expert to ensure we're following the best conventions for this.
cc @Sovietaced @fg91 @wild-endeavor @EngHabu

Signed-off-by: ttitsworth-fsp <ttitsworth@lila.ai>
machichima
machichima previously approved these changes Dec 11, 2025
Signed-off-by: machichima <nary12321@gmail.com>
@machichima machichima merged commit be517bd into flyteorg:master Dec 17, 2025
54 checks passed
@welcome
Copy link

welcome bot commented Dec 17, 2025

Congrats on merging your first pull request! 🎉

@flyte-bot
Copy link
Collaborator

Bito Automatic Review Skipped – PR Already Merged

Bito scheduled an automatic review for this pull request, but the review was skipped because this PR was merged before the review could be run.
No action is needed if you didn't intend to review it. To get a review, you can type /review in a comment and save it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

added Merged changes that add new functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants