Summary

Add a dashboard to the UI that surfaces real-world model prediction quality metrics, override patterns, and category-level accuracy. Currently, the only accuracy metric available is CV accuracy in model_metadata (which overfits on training data), while real-world accuracy on reviewed transactions is significantly lower. There is no visibility into prediction quality from the UI.

Motivation

• CV accuracy is misleading vs. real-world performance
• A meaningful percentage of categorized transactions are overridden by users — but there's no way to see which categories are problematic
• Category confusion patterns are invisible without manual DB queries
• A large portion of transactions remain unreviewed with no visibility into the backlog
• No existing analytics endpoint tracks prediction quality — api/analytics.py only covers budget/spending

Proposed Dashboard Sections

1. Overall Prediction Accuracy

• Real-world accuracy on reviewed transactions
• Override rate
• Accuracy trend over time (per training run)
• Comparison: CV accuracy vs. real-world accuracy

2. Accuracy by Confidence Band

• Breakdown by confidence ranges (e.g., 0.95+, 0.80–0.95, 0.50–0.80, <0.50)
• Show correct/overridden/uncategorized counts per band
• Highlight low-confidence predictions that perform poorly

3. Category Confusion Matrix

• Top N most confused category pairs
• Per-category override rates
• Sortable table view

4. Review Backlog

• Unreviewed transaction count by priority (standard / high / quality_check)
• Average confidence of unreviewed transactions
• Uncategorized transaction count

Implementation Notes

• New API endpoint(s) needed under api/analytics.py or a new api/ml_analytics.py
• Queries compare predicted_category_id vs category_id on reviewed transactions
• Frontend: new dashboard page/tab, likely with charts (bar/line) and tables
• Consider caching computed metrics since they scan the full transactions table