A 3-stage deliberation system where multiple LLMs collaboratively answer questions through anonymized peer review and synthesis.
Inspired by karpathy/llm-council.
LLM Council orchestrates multiple large language models to provide more thoughtful, balanced answers. The key innovation is anonymized peer review in Stage 2—models evaluate responses without knowing which model generated them, preventing favoritism and ensuring honest assessment.
User Query
↓
┌─────────────────────────────────────────────────────────┐
│ Stage 0: Prompt Refinement (Optional) │
│ Iteratively clarify intent and revise user prompt │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Stage 1: Individual Responses │
│ All council models answer the query in parallel │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Stage 1.5: Extension Modes (Optional) │
│ - Negative Mode: Identify gaps and blind spots │
│ - Deliberation: Models revise answers based on feedback │
│ - Interactive Response Review │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Stage 2: Anonymized Peer Review │
│ Responses anonymized as "Response A, B, C..." │
│ Each model evaluates and ranks all responses │
│ Aggregate rankings calculated │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Stage 3: Synthesis │
│ Chairman model synthesizes final answer. Options: │
│ - Synthesis Prefs (Optional): User selects mode │
│ - Combines responses, rankings, and negative evaluation │
└─────────────────────────────────────────────────────────┘
↓
Final Answer
- Multi-model consensus: Leverage diverse perspectives from GPT, Claude, Gemini, Grok, and more.
- Anonymized evaluation: Unbiased peer review without model identification.
- Transparent reasoning: View raw evaluations, debug traces, and validate ranking extraction.
- Graceful degradation: Continues with successful responses if some models fail.
- Polling-based sync: Robust state synchronization between frontend and backend.
- Conversation history: Persistent JSON storage with full deliberation records.
- Markdown Export: Export full conversation context and reasoning to Markdown.
- Focus View: Dedicated modal for deep reading of the synthesized answer.
- Negative Mode: Models identify gaps, risks, and blind spots in responses.
- Deliberation: Models revise their answers after seeing peer feedback.
- Interactive Mode: Inline prompts for user guidance during processing.
- Prompt Refinement: Iteratively refine your initial query for better results.
- Python 3.10+
- Node.js 18+
- uv (Python package manager)
- OpenRouter API key
-
Clone the repository:
git clone https://github.com/yourusername/llm-council.git cd llm-council -
Configure environment:
cp .env.sample .env # Edit .env and add your OPENROUTER_API_KEY and API_AUTH_KEY -
Install dependencies:
# Backend uv sync # Frontend cd frontend npm install cd ..
-
Start the application:
./start.sh
Or run separately:
# Terminal 1 - Backend (port 8001) uv run python -m backend.main # Terminal 2 - Frontend (port 5173) cd frontend && npm run dev
All configuration is managed via .env. See .env.sample for full options.
| Variable | Description |
|---|---|
OPENROUTER_API_KEY |
Your OpenRouter API key |
API_AUTH_KEY |
Local API authentication key |
| Variable | Description | Default |
|---|---|---|
COUNCIL_MODELS |
Comma-separated model IDs | GPT-4o, Gemini-1.5, Claude-3.5, etc. |
CHAIRMAN_MODEL |
Synthesis model | google/gemini-pro-1.5 |
TITLE_MODEL |
Title generation model | google/gemini-flash-1.5 |
| Variable | Description | Default |
|---|---|---|
NEGATIVE_MODE_ENABLED |
Enable gap-finding evaluations | false |
DELIBERATION_ENABLED |
Enable revised responses | false |
INTERACTIVE_MODE_ENABLED |
Enable inline user prompts | false |
PROMPT_REFINEMENT_ENABLED |
Enable query refinement stage | false |
llm-council/
├── backend/ # Python FastAPI backend
│ ├── main.py # FastAPI app entry point & routes
│ ├── config.py # Configuration management
│ ├── council.py # Deliberation & Synthesis logic
│ ├── langgraph_flow.py # LangGraph workflow orchestration
│ ├── consultation_manager.py # Async task orchestration
│ └── storage.py # JSON conversation persistence
├── frontend/ # React + Vite + Tailwind
│ ├── src/
│ │ ├── App.jsx # Main orchestration
│ │ ├── store/ # Zustand global state
│ │ ├── api.js # Backend API client
│ │ └── components/ # UI components (Stage1-3, etc.)
│ └── package.json
├── data/ # Conversation storage
│ └── conversations/ # JSON files per conversation
├── AGENTS.md # Development guidelines
├── start.sh # Combined startup script
└── pyproject.toml # Python dependencies
| Endpoint | Method | Description |
|---|---|---|
/api/conversations |
GET | List all conversations |
/api/conversations |
POST | Create new conversation |
/api/conversations/{id} |
GET | Get conversation by ID |
/api/conversations/{id} |
DELETE | Delete conversation |
/api/conversations/{id}/message |
POST | Send message (async) |
/api/conversations/{id}/queue |
GET | Get queued messages |
.../export-markdown |
GET | Export as Markdown |
| Endpoint | Method | Description |
|---|---|---|
/api/consultations/status |
GET | Global system status |
.../consultation/status |
GET | Check processing status |
.../consultation/cancel |
POST | Cancel consultation |
| Endpoint | Method | Description |
|---|---|---|
/api/config |
GET | Get runtime configuration |
/api/interactive-response |
POST | Submit user feedback |
/ |
GET | Health check |
- Backend: FastAPI with async orchestration via
consultation_manager. - Workflow: LangGraph for complex 3-stage deliberation state management.
- Frontend: React 19, Vite 7, Tailwind CSS 4, and Zustand.
- Storage: Lightweight JSON file persistence in
data/conversations/. - Sync: Robust polling mechanism replaces SSE for state consistency.
- AGENTS.md - Development guidelines and project standards.
- .env.sample - Configuration reference and templates.
MIT