Urgap is a cloud-native framework for file-based data engineering, containing abstraction layers for data and meta data, extensive re-run skipping logic and data versioning. Urgap can be incorporated with any scheduling/pipelining tool making pipeline development independent from business logic and data storage, while offering standardized logging and execution, which makes monitoring and debugging easy.
Urgap gives us the governance constraints required for a decentralized data domain autonomy as Urgap will enforce shared common data IO for storage, a common meta data capturing process in form of an interface thus can be plugged into any existing processes and finally global data lineages.
Watch our introduction talk urgap - unified resource governance and data provenance by Christian Fufezan to get a comprehensive overview of urgap's design and capabilities:
We recommend using a virtual environment for Python projects. This guide uses uv for dependency management.
Basic Installation (local file system access only):
uv syncWith Cloud Storage Support:
uv sync --extra cloudWith All Runtime Extras:
uv sync --extra allAvailable extras include:
server: FastAPI/Flask/uvicorn + MCP server foructl runtabular: pandas/polars/pyarrow and Excel engines for tabular unodesviz: plotly-based reporting and visualizer unodesgithub: GitHub-backed file IOcloud: Azure and Google Cloud storage backendsdatabases: PostgreSQL backenddocs: Sphinx documentation builddev: test tooling (pytest) plus all runtime extrasall: every runtime extra above (notdevordocs)
Install test dependencies:
uv sync --extra devRun the test suite:
uv run pytest testsThe best way to learn urgap is through a complete example. Check out the end-to-end filter CSV pipeline:
- Location: tests/integrationtests/end2end/test_filter_csv_pipeline.py
- What it demonstrates: Complete pipeline setup, node configuration, and execution
- Requirements: Everything needed to run this example is included in the repository
This example can be run entirely on your local machine without any external dependencies.
To run the example:
pytest tests/integrationtests/end2end/test_filter_csv_pipeline.pyPlease use sphinx in the docs folder
Note
Currently CI does not include pushing the documentation to readthedocs, therefore please #. checkout the repo #. pip install -e . #. cd docs #. make html #. open docs/build/index.html