Small functions (and examples) for interacting with XTDB via requests http.

Sample notebook here, and great xt docs to memorize at https://xtdb.com/docs/.

pip istall py-xtdb

or

poetry add py-xtdb

q_results = query_edn(host="http://localhost:3001",
                      data="""
{:query {:find [?id ?name ?address]
         :keys [id name address]
         :where [[?id :xt/id]
                 [?id :name ?name]
                 [?id :address ?address]]
         :limit 2}}

    """)

print(q_results)

[{'address': '4681 Billy Parkway Suite 747\nNorth James, AR 25849',
  'id': 1,
  'name': 'Mr. David Mills'},
 {'address': '48596 Robert Walks\nWest Angelview, CO 76011',
  'id': 2,
  'name': 'Christopher Gregory'}]

If you’re looking to get query results into pandas fn `DataFrame` reads this sort of thing:

import pandas

print(pandas.DataFrame(q_results))
   id                 name                                            address
0   1      Mr. David Mills  4681 Billy Parkway Suite 747\nNorth James, AR ...
1   2  Christopher Gregory       48596 Robert Walks\nWest Angelview, CO 76011

If you’d like to try out xt and python you can clone this repo, install the deps, start a local xt server, and walk through the jupyter notebooks in /nb/.

Here’s instructions/guidelines in more detail:

First, clone this repo locally and change to said directory:

git clone https://github.com/joefromct/py-xtdb
cd py-xtdb

Now we need to start xtdb in a terminal so the jupyter notebook has something to talk to.

The following command runs utilizing the `deps.edn` file which pulls in xt jars and runs with 2gb of memory.

# from same directory cloned above
clojure -X:xt

You’ll see some metrics flash to the screen occasionally saying what XT is up to. This is all setup with the file xtdb.edn. You can see here that xtdb.edn specifies data as the directory to store our database, and lucene full-text-search module(s).

So in summary, deps.edn pulls the dependencies you need, and xtdb.edn configures xt.

----

Next lets get a jupyterlab environment running.

Open another terminal to this same directory.

Here, install python dependencies:

pip install -f requirements.txt

or use poetry (picks up the pyproject.toml… all same dir.)

poetry install

Now we should hopefully have jupyterlab on our path. Start it up like so:

jupyter-lab nb/demo.ipynb

From here you can step through the jupyter cells as per usual.

Have a look here.

Constructing queries in python strings can be cumbersome.

If you run the xt.py script directly you can pass in a query file and yield results:

If you’re query file has `keys` (returning hashmaps/dicts) it should dump a tabular format using tabulate:

If you have tools like entr available something like below could watch data/query.edn and run the query on each file save. This can sort of feel like a query tool in the relational world. (Clojure likely has better options to interactively visualize data…)

I’m looking to make `xtdb` more accessible to a python-first team, primarily with a focus on data science and/or data ingestion and document loading.

Some shops prefer to process data in python, and ideally they would have a gentle pathway/introduction to xtdb. This example has minimal clojure code, and all dependencies are driven just by deps.edn and xtdb.edn.