Skip to content
the1skyPublic

About

A high-performance Cascade Intent Router for LLM Agents. Features a dual-layer architecture (Embedding + LLM) to reduce latency by 90% and cut token costs by 80%. Optimized for RAG and Enterprise workflows.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

💎 Prism - High-Performance Intent Router

Prism is a lightweight, dual-layer intent routing middleware designed for LLM Agents. It acts as a "Triage Nurse" before your expensive "Doctor" (LLM), reducing latency by 90% and token costs by 80% for high-frequency queries.

中文文档 | Architecture | Quick Start | Dify Integration


🚀 Why Prism?

Building a production-ready Agent often faces the "Cost vs. Speed" dilemma:

  • Directly using GPT-4/DeepSeek: Too slow (1-3s) and expensive for simple queries like "Check attendance" or "Hi".
  • Keyword Matching: Too dumb to understand variations like "Show me the logs".

Prism solves this by using a Cascade Architecture:

  1. ⚡ Level 1 (Fast Path): Uses local Embeddings (BGE-Small) to intercept simple intents in <20ms.
  2. 🧠 Level 2 (Slow Path): Routes complex/ambiguous queries to LLMs (DeepSeek V3) for reasoning and Parameter Extraction.

🏗 Architecture

Prism employs a "Funnel" strategy to process user queries:

graph TD
    User([👤 User Input]) --> BGE

    subgraph "⚡ Level 1: Fast Path"
        direction TB
        BGE[🔍 BGE Embedding]
        Gate{Confidence > 0.85?}
        BGE --> Gate
    end

    subgraph "🧠 Level 2: Smart Path"
        direction TB
        LLM[🤖 DeepSeek V3]
        Tools[🛠️ Function Calling]
        LLM --> Tools
    end

    Gate -->|Yes| Action_Simple[🚀 Direct Execution]
    Gate -->|No| LLM
    Tools --> Action_Complex[🧩 Complex Execution]

Loading

✨ Key Features

  • Hybrid Routing: Combines the speed of Vector Search with the intelligence of LLMs.
  • Zero-Cost Interception: 80% of daily queries (FAQ, Navigations) are handled locally without consuming LLM tokens.
  • Smart Parameter Extraction: Automatically extracts dates, metrics, and entities (e.g., "Last week's DAU" -> date="2023-10-20", metric="DAU").
  • Dify Ready: Built-in FastAPI wrapper, ready to be added as a Custom Tool in Dify.
  • Private Deployment: Supports Docker deployment; data stays within your VPC.

⚡ Performance Comparison

Scenario Traditional Agent (LLM Only) Prism (Cascade) Improvement
Simple Query


(e.g., "Search Docs") | ~1.2s | ~0.015s | 80x Faster 🚀 | | Complex Query


(e.g., "Show me last week's sales") | ~2.5s | ~2.6s | Same Accuracy | | Cost (1M reqs) | $500 | **$80** | 84% Cheaper 💰 |


🛠 Quick Start

1. Installation

git clone https://github.com/the1sky/prism-router.git
cd prism-router

# Create and activate conda environment
conda create -n prism python=3.13 -y
conda activate prism

# Install dependencies
pip install -r requirements.txt

2. Configuration

Create a .env file or set environment variables:

export DEEPSEEK_API_KEY="sk-your-api-key"

3. Run the Server

# Activate environment (if not already activated)
conda activate prism

# Start the server
python main.py
# Server running at http://0.0.0.0:8000

4. Test It

# Test Fast Path (No Token Cost)
curl -X POST "http://localhost:8000/v1/route" \
     -H "Content-Type: application/json" \
     -d '{"query": "Please check the attendance policy"}'

# Test Slow Path (DeepSeek with Args)
curl -X POST "http://localhost:8000/v1/route" \
     -H "Content-Type: application/json" \
     -d '{"query": "Help me restart the test server"}'

🧩 Dify Integration

Prism is designed to work seamlessly with Dify.

  1. Deploy Prism: Run this project on a server accessible by Dify (e.g., expose via Ngrok or deploy to Render/Vercel).
  2. Add Custom Tool: In Dify, go to Tools -> Custom Tool -> Create.
  3. Import Schema: Use the openapi.json provided at http://your-server/openapi.json.
  4. Use in Workflow:
  • Add the Prism Tool node at the beginning of your Chatflow.
  • Use Conditional Branching (If/Else) based on the output intent.

Pro Tip: See docs/dify_integration.md for a step-by-step screenshot guide.


📂 Project Structure

prism/
├── main.py              # Application entry point
├── api/
│   ├── __init__.py
│   └── server.py        # FastAPI server and route definitions
├── config/
│   ├── kb_data.json     # Knowledge base data (intent examples)
│   └── tools_schema.json # Function calling schemas for LLM
├── core/
│   ├── __init__.py
│   └── router.py        # Core routing logic (Fast/Slow path)
├── requirements.txt
├── Dockerfile
├── README.md            # English documentation
└── README_CN.md         # Chinese documentation

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

📄 License

Distributed under the MIT License. See LICENSE for more information.

About

A high-performance Cascade Intent Router for LLM Agents. Features a dual-layer architecture (Embedding + LLM) to reduce latency by 90% and cut token costs by 80%. Optimized for RAG and Enterprise workflows.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages