• Multi-turn chat with persistent conversation state in PostgreSQL.
• Server-Sent Events streaming from FastAPI to React.
• Groq OpenAI-compatible API using llama-3.3-70b-versatile.
• Langfuse OpenAI SDK wrapper for LLM tracing when Langfuse keys are provided.
• Langfuse evaluation scores, including Groq-powered LLM-as-a-Judge scores for matrix correctness and missing-field behavior.
• A deterministic Strategic Lead Matrix rule engine, so the final qualification is testable and not dependent on LLM guesses.
• Proactive missing-field questions before final qualification.
• Square-footage fallback when annual usage is unknown.
• Docker Compose for frontend, backend, and database.

React + TypeScript UI
        |
        | POST /api/chat/stream  (SSE stream)
        v
FastAPI backend
        |
        |-- LLM extraction via Groq Llama 3.3 70B
        |-- Langfuse traces LLM calls when configured
        |-- Deterministic orchestrator decides missing fields and tier
        v
PostgreSQL
  conversations / messages / leads

The LLM is excellent for extracting facts from free text, but the tier result should be auditable. Therefore:

1. The LLM extracts structured fields from the latest user turn.
2. The backend merges extracted facts into the stored conversation state.
3. A deterministic rule engine checks the Strategic Lead Matrix.
4. The agent asks only for required missing fields.
5. Once enough data is available, the backend stores the final lead record.

If no row matches, the backend returns Lemon (No Priority).

The orchestrator is designed to minimize turns while still collecting the four core matrix fields before finalizing:

• If the user says No Current Provider only, the system records contract_status=no_current_provider but asks for the remaining core fields instead of concluding immediately.
• If the user provides all four core fields in one message, the system can conclude in the same turn.
• If annual usage is missing, the next question asks for annual MWh OR square footage, so the user can answer with whichever they know.
• If the user answers unknown, not sure, no, null, or any for a specific field, that field is stored as Any and is not asked again.
• If a concrete/wildcard combination matches the matrix after all core fields are answered or marked Any, the rule engine returns the highest-priority matching tier.

Edit .env and paste your own Groq key:

GROQ_API_KEY=your_key_here

Optional Langfuse values:

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_BASE_URL=https://cloud.langfuse.com

Security note: real API keys should stay in .env only and should not be committed.

docker compose up --build

Open:

• Frontend: http://localhost:3000
• Backend docs: http://localhost:8000/docs
• Health check: http://localhost:8000/api/health

Request:

{
  "conversation_id": null,
  "message": "Commercial, 75 MWh, month-to-month"
}

Response is Server-Sent Events:

event: start
data: {"conversation_id":"..."}

event: token
data: {"text":"Qualification"}

event: state
data: {"status":"completed","state":{...},"qualification":{...}}

event: done
data: {"conversation_id":"..."}

Returns the current persisted state for a session.

When the following environment variables are set, the Langfuse OpenAI integration traces Groq chat completion calls and sends evaluation scores:

LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...
LANGFUSE_BASE_URL=https://cloud.langfuse.com

The code is in backend/app/services/llm.py, backend/app/services/evaluation.py, and backend/app/api/chat.py. It uses the Langfuse OpenAI wrapper when authentication succeeds, then calls Groq through the OpenAI-compatible base URL. Each turn emits deterministic scores plus LLM-as-a-Judge scores:

• lead_matrix_decision_valid
• field_collection_progress
• asks_for_missing_info
• unknown_as_any_handled
• llm_judge_matrix_correctness
• llm_judge_missing_info_behavior
• llm_judge_unknown_any_handling
• llm_judge_overall_quality

Enable/disable judge scoring with:

LANGFUSE_EVALUATION_ENABLED=true
LANGFUSE_LLM_JUDGE_ENABLED=true

backend/app/services/lead_rules.py      # deterministic matrix + missing-field policy
backend/app/services/orchestrator.py    # multi-turn state manager
backend/app/services/llm.py             # Groq + Langfuse integration and streaming
backend/app/api/chat.py                 # streaming API endpoint
frontend/src/App.tsx                    # responsive chat UI
frontend/src/api.ts                     # SSE parser

For production scale:

1. Run FastAPI behind a load balancer with multiple Uvicorn/Gunicorn workers.
2. Move conversation updates into a transactional service layer with row-level locking for each conversation.
3. Use PgBouncer for PostgreSQL connection pooling.
4. Add Redis for short-lived stream/session cache and idempotency keys.
5. Add queue-based background processing for analytics/evaluation.
6. Use Langfuse datasets and automated evals to regression-test the rule engine and LLM extraction quality.
7. Add rate limits and per-tenant quotas for Groq API calls.
8. Use Prometheus/Grafana for API latency, TTFT, and error monitoring.

1. Chatbot

2. Chatbot

3. Langfuse