Skip to content

bruin-data/ingestr

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2,225 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Copy data from any source to any destination without any code


ingestr is a command-line app that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.

  • ✨ copy data from your database into any destination
  • ➕ incremental loading: append, merge or delete+insert
  • 🐍 single-command installation

ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.

Installation

We recommend using uv to run ingestr.

pip install uv
uvx ingestr

Alternatively, if you'd like to install it globally:

uv pip install --system ingestr

While installation with vanilla pip is possible, it's an order of magnitude slower.

Quickstart

ingestr ingest \
    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
    --source-table 'public.some_data' \
    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
    --dest-table 'ingestr.some_data'

That's it.

This command:

  • gets the table public.some_data from the Postgres instance.
  • uploads this data to your BigQuery warehouse under the schema ingestr and table some_data.

Documentation

You can see the full documentation here.

Community

Join our Slack community here.

Contributing

Pull requests are welcome. However, please open an issue first to discuss what you would like to change. We maybe able to offer you help and feedback regarding any changes you would like to make.

Note

After cloning ingestr make sure to run make setup to install githooks.

Supported sources & destinations

Source Destination
Databases
BigQuery
ClickHouse
CrateDB
Databricks
IBM Db2 -
DuckDB
DynamoDB -
Elasticsearch
GCP Spanner -
Local CSV file
Microsoft SQL Server
MongoDB
MotherDuck
MySQL
Oracle
Postgres
Redshift
SAP Hana
Snowflake
SQLite
Trino
Platforms
Adjust -
Airtable -
Amazon Kinesis -
Apache Kafka -
AppsFlyer -
App Store -
Asana -
Attio -
Chess.com -
Facebook Ads -
GitHub -
Gorgias -
Google Sheets -
Google Ads -
Google Analytics -
Klaviyo -
LinkedIn Ads -
Notion -
Personio -
Phantombuster -
Pipedrive -
S3 -
Salesforce -
Shopify -
Slack -
Smartsheets -
Solidgate -
Stripe -
TikTok Ads -
Zendesk -

Feel free to create an issue if you'd like to see support for another source or destination.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Some components are licensed under Apache 2.0 - see the NOTICE file for details.

Packages

 
 
 

Contributors

Languages