This project takes a search query like "What is python" and turns it into clean, ranked web content.

Instead of just showing links like Google, it:

query → search engine → web pages → cleaned text → filtering → ranking → final results

Think of it as a mini search pipeline for an AI agent.

Query-Flow is a modular web retrieval system built to feed an AI agent with real web content.

It replaces paid search APIs by using:

• SearXNG (local or public instances)
• Direct web scraping
• Custom ranking system

It is NOT:

• a production search engine
• a polished API service
• a crawler network

It is:

a learning + practical retrieval pipeline for AI assistants

User Query
   ↓
app.py (orchestrator)
   ↓
SearXNG Provider (local → fallback public)
   ↓
fetch_layer (download HTML pages)
   ↓
extraction_layer (extract readable text)
   ↓
filtering_layer (remove junk pages)
   ↓
ranking_layer (score relevance)
   ↓
output_layer (display results)

Before running, ensure you have the following installed:

• Docker & Docker Compose: To run the local SearXNG instance.
• Python 3.10+: The core application language.
• uv (recommended) or pip: For dependency management.
• make: (Optional) For using the provided shortcut commands.

Uses SearXNG:

• Local Docker instance (preferred)
• Public fallback instances (if local fails)

Handles routing automatically.

Downloads raw HTML using httpx.

Problem:

• Some sites block bots (Wikipedia, StackOverflow, Canva, etc.)
• 403 errors are expected, not fatal

Fix used:

• Browser-like headers
• Redirect support

Uses trafilatura to extract readable content from HTML.

Removes:

• navigation
• ads
• cookie banners

Removes:

• empty documents
• very short pages
• low-quality content

Scores documents using:

• keyword matching
• simple IDF weighting
• title vs body weighting
• repetition penalty
• spam detection

Output is a ranked list of documents.

Prints:

• score
• title (best-effort)
• preview text
• keyword matches

Inside config.py:

• MAX_SEARCH_RESULTS → limits crawling
• HTTP_TIMEOUT → request timeout
• ranking weights (title, body, spam, etc.)

make docker_start

or:

docker compose up -d

python app.py

Examples:

• Wikipedia
• StackOverflow
• Canva

Reason:

• bot protection (403 Forbidden)

This is normal.

SearXNG instances differ slightly, so results may vary.

Current ranking is heuristic-based, not ML-based.

So sometimes:

• homepage pages rank high
• noisy pages pass filtering

Requests are sequential → slower performance.

This project was originally built as part of an AI terminal agent experiment.

The goal was:

Replace paid search APIs with a free, controllable retrieval pipeline.

So it can be plugged into:

• AI chatbots
• local agents
• RAG systems
• terminal assistants

Since the current version uses sequential fetching, performance depends on the number of results. If the pipeline feels slow, you can adjust MAX_SEARCH_RESULTS in config.py to a lower value (e.g., 3 or 5) for a much faster response.

Because:

• Search APIs are expensive
• Google APIs are limited
• AI needs external context
• SearXNG is free but raw
• raw web data needs processing

So this pipeline sits in the middle.

• search engine basics
• data pipelines (ETL style)
• web scraping limitations
• ranking heuristics
• fallback system design
• modular architecture

• async fetch (huge speed boost)
• better metadata preservation
• smarter ranking (semantic scoring)
• optional LLM summarization layer
• structured JSON output API