Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Is it possible to change the dify app execution engine from Celery to another workflow engine, e.g. argo-workflow ? #6935

Open
4 of 5 tasks
Colstuwjx opened this issue Aug 3, 2024 · 5 comments
Open
4 of 5 tasks

Is it possible to change the dify app execution engine from Celery to another workflow engine, e.g. argo-workflow ? #6935

Colstuwjx opened this issue Aug 3, 2024 · 5 comments
Assignees
@takatost
Labels
🌊 feat:workflow Workflow related stuff. stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed

Comments

@Colstuwjx
Copy link

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Hi team,

Thanks for sharing this awesome project! We're evaluating if we can use Dify in our production environment, however, it seems like that we need to support some production requirements in advance:

  1. since it's production environment, it's necessary to have a way to do canary deployment and also traffic control, rate limit, I've raised an individual issue for this, see ref #6893
  2. we need to build a CICD pipeline to evaluate the GenAI app quality by running several golden tests and also manage the released versions by a more robust way, e.g. DSL + GitOps way. I think it's possible for Dify since it supports to export the DSL and also import it, which means we can integrate with a GitOps release pipeline by these two apis
  3. however, since we're using Celery as the workflow execution engine, I'm afraid the isolation between each tenant might not be good enough ? How can we avoid that one bad-design app impact the other GenAI apps running on the same one Dify Cluster ? One of the solution that could address this issue could be using a CloudNative execution engine and use K8s scheduling feature to resolve it. That's why I propose this issue.

Any ideas if the point 3 could be addressed by some ways ?
Thanks!

2. Additional context or comments

I've experienced the last cloud native revolution, and I strongly believe that if one project could be pluggable and involve more community power, it could adopt more end users and gain more contributions from the community. IMO, one of the success factor of Kubernetes project is that it allows to be customized by many ways, the Resource + Reconcile pattern / CNI / CSI / SideCar design allows more community developers to join the game. Dify and the other LLMOps platform could potentially to be the Next Kubernetes in the GenAI indurstry.

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
@dosubot dosubot bot added the 🌊 feat:workflow Workflow related stuff. label Aug 3, 2024
@Colstuwjx
Copy link
Author

BTW, just searched the current best practices of AI apps, e.g. JupyterHub offered a way to integrate with Kubernetes: https://jupyterhub.readthedocs.io/en/stable/explanation/concepts.html#spawner

@crazywoola
Copy link
Member

crazywoola commented Aug 4, 2024 •
edited
Loading

Hello @Colstuwjx

Thank you for your insightful suggestion and interest in the Dify project. We appreciate the enthusiasm and the potential contribution you've expressed.

Regarding your proposal to consider updating or replacing the Celery Worker dependency with another workflow engine such as argo-workflow, we have taken it into serious consideration. We understand the importance of features like canary deployment, traffic control, and rate limiting in a production environment, as well as the benefits of using a CloudNative execution engine with Kubernetes scheduling for better tenant isolation.

However, at this moment, we do not have plans to update or replace our dependency on Celery in the near future. Celery serves as a robust and widely-adopted workflow execution engine that plays a significant role in our current Dify architecture. We will continue to monitor and assess its performance to ensure it meets the needs of our users.

Nevertheless, we truly value your forward-thinking and suggestions for the future development of the project. We believe that with the growing community and contributions like yours, Dify has the potential to become a leading platform in the GenAI industry. We encourage and welcome more community members to get involved in the project, driving its progress and evolution.

Should you have any further ideas or suggestions, or if you are willing to contribute to this feature, please do not hesitate to get in touch to @takatost . We look forward to working with you to build an even stronger and more adaptable Dify platform.

Thank you once again for your contribution and support!

@Colstuwjx
Copy link
Author

Hi @crazywoola , thanks for your kindly reply.

After deep diving into the sourcecode, I suddenly found the Dify workflow execution is actually triggered by a sync api, and also use an original new Thread way rather than Celery workers. The entrance http api is workflow/run and the api controller would execute by this process: AppGenerateService -> WorkflowAppGenerator (new Thread here) -> WorkflowAppRunner -> WorkflowEngineManager.

Related discussion issue: ref #4489.
Thanks.

@Colstuwjx
Copy link
Author

I've got an idea to resolve this issue, by taking advantage of Serverless Engine, e.g. OpenFaas. The details be like:

  1. when someone published an app on Dify platform, it could trigger a GitOps pipeline to generate the serverless function code and its corresponding workflow DSL, the published app would be regarding as an individual microservice
  2. The individual dify app would be running on OpenFaas, e.g. by this way: https://www.openfaas.com/blog/openfaas-flask/
  3. Therefore, Dify doesn't need to reinvent the wheels like API Gateway, CICD pipeline etc, just use the industry leading solutions

Any suggestions ?

@lilyzhoupeijie
Copy link

@Colstuwjx hi,we also want to publish dify workflow to argo ,do you have interest in working together?

@dosubot dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@takatost takatost

Labels
🌊 feat:workflow Workflow related stuff. stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Colstuwjx @takatost @lilyzhoupeijie @crazywoola