Proof of concept vector search for Are.na.

Per their latest investor report (2024 Q3/Q4), the Are.na team is currently focusing on making search much better than what it is now. I'm not sure how it currently works under the hood, but it seems to me that it's doing a fairly primitive text search, only matching keywords rather than the semantic meaning or context behind them.

I wanted to try my hand at implementing vector searching for Are.na, so this is the proof of concept for it.

The project utilizes local LLM models (via Ollama) to extract meaningful information in the form of text from a variety of media (images, writing, PDFs, etc.), though not all formats are supported right now (e.g. embed, mp3s, etc.).

Then, it uses Chroma's default embedding function (all-MiniLM-L6-v2) to generate embeddings for each piece of information before it's added to the database.

Everything runs locally, so you'll need a pretty beefy machine, and be prepared for it to spin up its fans to the max. Don't expect to be able to use your machine for a bit if you're ingesting a large amount of Are.na content. I just let mine sit for a bit and go for a walk or do some writing. Errors are gracefully handled and logged so you can see what was unsuccessfully processed.

Keeping everything in place for this small proof of concept makes sense, I think. Some types are shared between the frontend and backend, hence the shared folder.

In the backend, you should have an .env file with the following fields:

ARENA_SECRET_TOKEN=

which you can obtain from the Are.na dev page.

In the frontend, you should have an .env file with the following fields:

BACKEND_URL=

which should default to localhost:XXXX if developing locally.

Before we can perform vector search on content on Are.na, we must first convert said content into vector embeddings. But even before that, we must be able to represent the content as some string that at least attempts to capture all facets of what they are, both tangible and intangible. Eventually, we'll want to have a database of these vector embeddings that we can perform vector search on.

So let's start from the top and walk through how to build up this database via ingestion of content from Are.na channels. To do so, run yarn dev in the backend folder, which will spin up a Docker instance that houses a chroma database. Then, you can hit the ${BACKEND_URL}/ingestChannel?channelSlug=${CHANNEL_SLUG} endpoint with the slug of the channel you wish to ingest. It's not very optimized or parallelized, but I think it does the job for a small prototype like this.

The ingestion process goes something like this: for each piece of content within the Are.na channel, it identifies the type of content being processed, for example an image or text block. Then, depending on the type of content, a specific strategy is employed to process them into some text representation:

• Block
  • URL
    • Use cheerio to grab HTML and parse before using llama3.2 to generate description, key points, etc.
  • Image
    • Use LLaVa to generate an "alt-text"
  • Text
    • Use llama3.2 to directly generate description, key points, etc.
  • File
    • PDF
      • Use pdf-ts to extract text, follow text parsing strategy
    • mp3s, etc...
      • TBD (WIP)
• Channel
  • Aggregation of its contents (WIP)
• User
  • Aggregation of their channels and blocks (WIP)

The ollama-js library supports structured outputs, which I use to generate a description of each content type, as well as an array of tags that captures concepts outside of what the content literally represents.

Now that we have the text representations of each piece of content within an Are.na channel, we can add them to our chroma database, alongside the source content for reference when searching for it.

1. Write script to fetch channel(s) from Are.na
2. Add blocks from channel(s) to database
3. Parse blocks, adding semantic tags to them
4. Embed them in vector database
5. Query away

• Ingest channel via frontend
• Display channel ingest status
• Click / hover states for search results
• Live update channel ingestion status
• Filtering search results
• Business considerations
• Replicate same.energy network search

From what I can tell, Are.na currently uses GraphQL for their internal backend (and their user-facing APIs use REST instead). Not really sure what database architecture they're using (SQL? NoSQL?).

1. What are the business requirements?
2. Any breaking changes?
3. Any migrations necessary?
4. What is the cost like?
5. How does it impact uptime/downtime?
6. Is it effective?
7. Where does it fail?
8. What privacy and data concerns need to be addressed? ***
9. How will user data be handled when processed by external AI services? ***
10. What is the rollout and iteration plan?

• same.energy

• Vector databases
• Vector search
• What are embeddings?
• t-SNE for visualizing n-dimensions