Skip to content

Cache loading pipeline from YAML in production #5902

@rohit1729

Description

@rohit1729

I observed that the schedule_all function loads the pipeline every time from YAML. This causes a fair amount of CPU misuse. The option makes sense in development as new changes in yaml can be instantly picked up. But we should cache the results in production as there won't be any yaml changes. New changes will come via redeployment only

We can add a LRU Cache on the function which is loading the file and parsing into yaml. We can let users configure the cache size. We can also keep the max as the number of pipelines in the project

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions