wurzel is the german word for root. So with this framework we provide a combination of the best out of the whole domain of retrieval and beyond.

wurzel is an open-source Python library built to address advanced Extract, Transform, Load (ETL) needs for Retrieval-Augmented Generation (RAG) systems. It is designed to streamline ETL processes while offering essential features like multi-tenancy, cloud-native deployment support, and job scheduling.

The repository includes initial implementations for widely-used frameworks in the RAG ecosystem, such as Qdrant, Milvus, and Hugging Face, providing users with a strong starting point for building scalable and efficient RAG pipelines.

• Advanced ETL Pipelines: Tailored for the specific needs of RAG systems.
• Multi-Tenancy: Easily manage multiple tenants or projects within a single system.
• Cloud-Native Deployment: Designed for seamless integration with Kubernetes, Docker, and other cloud platforms.
• Scheduling Capabilities: Schedule and manage ETL tasks using built-in or external tools.
• Framework Integrations: Pre-built support for popular tools like Qdrant, Milvus, and Hugging Face.
• Type Security: By leveraging capabilities of pydantic and pandera we ensure type security

To get started with wurzel, install the library using pip:

pip install wurzel

Run a step using the CLI:

wurzel run <step_file_path> --inputs ./data --output ./out

To inspect the step requirements:

wurzel inspect wurzel.<step_path>

Run a step using the snippet below:

from wurzel.steps import step
from pathlib import Path
from wurzel.step_executor import BaseStepExecutor

with BaseStepExecutor() as ex:
    ex(step, set(Path("./input")), Path("./output"))

For detailed instructions and examples on how to use wurzel, please refer to our official documentation.

This project has adopted the Contributor Covenant in version 2.1 as our code of conduct. Please see the details in our CODE_OF_CONDUCT.md. All contributors must abide by the code of conduct.

By participating in this project, you agree to abide by its Code of Conduct at all times.

This project follows the REUSE standard for software licensing. Each file contains copyright and license information, and license texts can be found in the ./LICENSES folder. For more information visit https://reuse.software/. You can find a guide for developers at https://telekom.github.io/reuse-template/.