Skip to content

Commit 1bdd9de

Browse files
TimeToBuildBobErikBjaregreptile-apps[bot]
authored
feat(lessons): Add ACE-inspired hybrid lesson matching (#817)
* feat(lessons): add ACE-inspired hybrid lesson matching Implements Phase 5.3 of ACE context optimization task: - Add HybridLessonMatcher with 5-component scoring system - Integrate with existing lesson loading via environment config - Maintain full backward compatibility (fallback to keyword-only) - Add comprehensive tests covering all components Scoring components: - Keyword relevance (25%) - Semantic similarity (40%) - Effectiveness feedback (25% - neutral until Phase 1 metadata) - Recency decay (10% - neutral until Phase 1 metadata) - Tool match bonus (+20%) Two-stage retrieval: 1. Fast candidate filtering (keyword + tool) → top-20 2. Precise hybrid scoring on candidates → top-5 Configuration: - GPTME_LESSONS_USE_HYBRID=true to enable - Falls back gracefully if embeddings unavailable - All weights configurable via HybridConfig Based on Stanford ACE (Agentic Context Engineering) framework achieving 10.6% performance gains and 70-90% token savings. Related: ErikBjare/bob#102, tasks/implement-ace-context-optimization.md Co-authored-by: Bob <bob@superuserlabs.org> * Update gptme/lessons/auto_include.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(lessons): integrate ACE hybrid lesson matching - Add optional ACE integration with GptmeHybridMatcher - Check ACE_AVAILABLE at runtime via try/except import - Use hybrid matching when GPTME_LESSONS_USE_HYBRID=true and ACE available - Fall back to keyword matching if ACE unavailable or hybrid disabled - TODO: Initialize embedder for full hybrid retrieval (Phase 5.3.4) Part of ACE Phase 5.3.3 implementation. Co-authored-by: Bob <bob@superuserlabs.org> * feat(lessons): initialize ACE embedder for hybrid lesson matching Phase 5.3.4: A/B test setup and embedder initialization - Initialize LessonEmbedder with lesson directories from LessonIndex - Use first lesson directory (typically workspace lessons) - Embeddings stored in .gptme/embeddings/lessons/ - Graceful fallback to keyword matching on embedder init failure - Ready for A/B testing hybrid vs keyword retrieval Related: ErikBjare/bob#120 (ACE implementation) Co-authored-by: Bob <bob@superuserlabs.org> * feat(lessons): pass session_id to hybrid matcher for retrieval analytics - Generate session_id from manager.chat_id (unique per gptme session) - Pass session_id to matcher.match() for tracking - Enables Phase 5.4 data collection and effectiveness analysis - Gracefully handles missing chat_id with None fallback Part of ACE Phase 5.4: Dynamic Lesson Retrieval Analytics Co-authored-by: Bob <bob@superuserlabs.org> * fix(lessons): only pass session_id to GptmeHybridMatcher The base LessonMatcher class doesn't accept session_id parameter. Only GptmeHybridMatcher (from ACE package) supports it for tracking. This fixes the test failures where keyword-only fallback was receiving unexpected session_id parameter. Fixes ErikBjare/bob#125 Co-authored-by: Bob <bob@superuserlabs.org> * fix(lessons): move type: ignore comments to import statement lines Mypy checks the 'from X import' line, not the imported symbols. Moving type: ignore comments from symbol lines to import lines resolves typecheck failures in CI. Co-authored-by: Bob <bob@superuserlabs.org> * fix(lessons): use lesson.category instead of lesson.metadata.category Fixes attribute error identified by Greptile review. The Lesson class has category as a direct attribute, not nested under metadata. - Line 127: Changed lesson.metadata.category to lesson.category - Maintains backward compatibility with 'general' fallback - All tests passing (51/51) --------- Co-authored-by: Erik Bjäreholt <erik.bjareholt@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
1 parent 83cbe9f commit 1bdd9de

File tree

4 files changed

+398
-11
lines changed

4 files changed

+398
-11
lines changed

‎gptme/lessons/auto_include.py‎

Lines changed: 39 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,18 +11,31 @@
1111

1212
logger = logging.getLogger(__name__)
1313

14+
# Optional hybrid matching support
15+
try:
16+
from .hybrid_matcher import HybridConfig, HybridLessonMatcher
17+
18+
HYBRID_AVAILABLE = True
19+
except ImportError:
20+
HYBRID_AVAILABLE = False
21+
logger.info("Hybrid matching not available, using keyword-only matching")
22+
1423

1524
def auto_include_lessons(
1625
messages: list["Message"],
1726
max_lessons: int = 5,
1827
enabled: bool = True,
28+
use_hybrid: bool = False,
29+
hybrid_config: "HybridConfig | None" = None,
1930
) -> list["Message"]:
2031
"""Automatically include relevant lessons in message context.
2132
2233
Args:
2334
messages: List of messages
2435
max_lessons: Maximum number of lessons to include
2536
enabled: Whether auto-inclusion is enabled
37+
use_hybrid: Use hybrid matching (semantic + effectiveness)
38+
hybrid_config: Configuration for hybrid matching
2639
2740
Returns:
2841
Updated message list with lessons included
@@ -51,14 +64,26 @@ def auto_include_lessons(
5164
logger.debug("No lessons found in index")
5265
return messages
5366

54-
matcher = LessonMatcher()
67+
# Choose matcher based on configuration
68+
matcher: LessonMatcher
69+
if use_hybrid and HYBRID_AVAILABLE:
70+
logger.debug("Using hybrid lesson matcher")
71+
matcher = HybridLessonMatcher(config=hybrid_config)
72+
else:
73+
if use_hybrid:
74+
logger.warning(
75+
"Hybrid matching requested but not available, falling back to keyword-only"
76+
)
77+
logger.debug("Using keyword-only lesson matcher")
78+
matcher = LessonMatcher()
79+
5580
matches = matcher.match(index.lessons, context)
5681

5782
if not matches:
5883
logger.debug("No matching lessons found")
5984
return messages
6085

61-
# Limit to top N
86+
# Limit to top N (matcher may already limit, but ensure it)
6287
matches = matches[:max_lessons]
6388

6489
# Format lessons for inclusion
@@ -94,12 +119,18 @@ def _format_lessons(matches: list) -> str:
94119

95120
# Add separator between lessons
96121
if i > 1:
97-
parts.append("\n---\n")
122+
parts.append("\n")
98123

99-
# Add lesson with context
124+
# Add lesson header with metadata
100125
parts.append(f"## {lesson.title}\n")
101-
parts.append(f"*Category: {lesson.category}*\n")
102-
parts.append(f"*Matched by: {', '.join(match.matched_by)}*\n\n")
103-
parts.append(lesson.body)
126+
parts.append(f"\n*Path: {lesson.path}*\n")
127+
parts.append(f"\n*Category: {lesson.category or 'general'}*\n")
128+
129+
# Add match info
130+
if match.matched_by:
131+
parts.append(f"\n*Matched by: {', '.join(match.matched_by[:3])}*\n")
132+
133+
# Add lesson content
134+
parts.append(f"\n{lesson.body}\n")
104135

105-
return "\n".join(parts)
136+
return "".join(parts)

‎gptme/lessons/hybrid_matcher.py‎

Lines changed: 243 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
"""Hybrid lesson matching using semantic similarity + effectiveness."""
2+
3+
import logging
4+
from dataclasses import dataclass
5+
from typing import Any
6+
7+
from .matcher import LessonMatcher, MatchContext, MatchResult
8+
from .parser import Lesson
9+
10+
logger = logging.getLogger(__name__)
11+
12+
# Optional embedding support
13+
try:
14+
import numpy as np
15+
from sentence_transformers import ( # type: ignore[import-not-found]
16+
SentenceTransformer,
17+
)
18+
19+
EMBEDDINGS_AVAILABLE = True
20+
except ImportError:
21+
EMBEDDINGS_AVAILABLE = False
22+
logger.info("Embeddings not available, falling back to keyword-only matching")
23+
24+
25+
@dataclass
26+
class HybridConfig:
27+
"""Configuration for hybrid lesson matching."""
28+
29+
# Scoring weights (must sum to ~1.0 for interpretability)
30+
keyword_weight: float = 0.25
31+
semantic_weight: float = 0.40
32+
effectiveness_weight: float = 0.25
33+
recency_weight: float = 0.10
34+
tool_bonus: float = 0.20 # Additional bonus for tool matches
35+
36+
# Retrieval parameters
37+
top_k: int = 20 # Candidate filtering
38+
top_n: int = 5 # Final selection
39+
40+
# Recency decay
41+
recency_decay_days: int = 30 # Half-life for recency score
42+
43+
# Enable/disable components
44+
enable_semantic: bool = True
45+
enable_effectiveness: bool = True
46+
enable_recency: bool = True
47+
48+
49+
class HybridLessonMatcher(LessonMatcher):
50+
"""Hybrid lesson matcher combining keyword, semantic, and metadata signals."""
51+
52+
def __init__(self, config: HybridConfig | None = None):
53+
"""Initialize hybrid matcher.
54+
55+
Args:
56+
config: Hybrid matching configuration
57+
"""
58+
super().__init__()
59+
self.config = config or HybridConfig()
60+
61+
# Initialize embedder if available and enabled
62+
self.embedder = None
63+
if EMBEDDINGS_AVAILABLE and self.config.enable_semantic:
64+
try:
65+
self.embedder = SentenceTransformer("all-MiniLM-L6-v2")
66+
logger.info("Initialized sentence embedder for semantic matching")
67+
except Exception as e:
68+
logger.warning(f"Failed to initialize embedder: {e}")
69+
self.embedder = None
70+
71+
def match(
72+
self, lessons: list[Lesson], context: MatchContext, threshold: float = 0.0
73+
) -> list[MatchResult]:
74+
"""Find matching lessons using hybrid scoring.
75+
76+
If embeddings unavailable, falls back to parent keyword-only matching.
77+
78+
Args:
79+
lessons: List of lessons to match against
80+
context: Context to match (message, tools, etc.)
81+
threshold: Minimum score threshold
82+
83+
Returns:
84+
List of match results, sorted by hybrid score (descending)
85+
"""
86+
# Fallback to keyword-only if semantic matching unavailable
87+
if not self.embedder:
88+
return super().match(lessons, context, threshold)
89+
90+
# Stage 1: Fast candidate filtering (keyword + tool)
91+
candidates = self._get_candidates(lessons, context)
92+
93+
if not candidates:
94+
return []
95+
96+
# Stage 2: Hybrid scoring on candidates
97+
results = self._score_candidates(candidates, context)
98+
99+
# Filter by threshold
100+
results = [r for r in results if r.score >= threshold]
101+
102+
# Sort and limit
103+
results.sort(key=lambda r: r.score, reverse=True)
104+
return results[: self.config.top_n]
105+
106+
def _get_candidates(
107+
self, lessons: list[Lesson], context: MatchContext
108+
) -> list[MatchResult]:
109+
"""Stage 1: Fast candidate filtering using keywords and tools."""
110+
# Use parent's keyword matching
111+
results = super().match(lessons, context, threshold=0.0)
112+
return results[: self.config.top_k]
113+
114+
def _score_candidates(
115+
self, candidates: list[MatchResult], context: MatchContext
116+
) -> list[MatchResult]:
117+
"""Stage 2: Hybrid scoring on filtered candidates."""
118+
if not self.embedder:
119+
return candidates
120+
121+
# Generate query embedding
122+
query_embed = self.embedder.encode(context.message, convert_to_numpy=True)
123+
124+
hybrid_results = []
125+
for match in candidates:
126+
lesson = match.lesson
127+
128+
# Component 1: Keyword score (normalized)
129+
keyword_score = self._keyword_score(lesson, context)
130+
131+
# Component 2: Semantic score
132+
semantic_score = 0.0
133+
if self.config.enable_semantic:
134+
semantic_score = self._semantic_score(query_embed, lesson, context)
135+
136+
# Component 3: Effectiveness score
137+
# NOTE: Neutral until ACE metadata implemented (Phase 1 schema)
138+
effectiveness_score = 0.5
139+
if self.config.enable_effectiveness:
140+
effectiveness_score = self._effectiveness_score(lesson)
141+
142+
# Component 4: Recency score
143+
# NOTE: Assume recent until ACE metadata implemented (Phase 1 schema)
144+
recency_score = 1.0
145+
if self.config.enable_recency:
146+
recency_score = self._recency_score(lesson)
147+
148+
# Component 5: Tool bonus
149+
tool_bonus_score = self._tool_bonus(lesson, context)
150+
151+
# Combine scores
152+
hybrid_score = (
153+
self.config.keyword_weight * keyword_score
154+
+ self.config.semantic_weight * semantic_score
155+
+ self.config.effectiveness_weight * effectiveness_score
156+
+ self.config.recency_weight * recency_score
157+
+ tool_bonus_score
158+
)
159+
160+
# Create new result with hybrid score
161+
hybrid_results.append(
162+
MatchResult(
163+
lesson=lesson,
164+
score=hybrid_score,
165+
matched_by=match.matched_by
166+
+ [
167+
f"hybrid:kw={keyword_score:.2f}",
168+
f"sem={semantic_score:.2f}",
169+
f"eff={effectiveness_score:.2f}",
170+
f"rec={recency_score:.2f}",
171+
],
172+
)
173+
)
174+
175+
return hybrid_results
176+
177+
def _keyword_score(self, lesson: Lesson, context: MatchContext) -> float:
178+
"""Normalized keyword relevance score (0.0-1.0)."""
179+
message_lower = context.message.lower()
180+
matches = sum(
181+
1 for kw in lesson.metadata.keywords if kw.lower() in message_lower
182+
)
183+
total_keywords = (
184+
len(lesson.metadata.keywords) if lesson.metadata.keywords else 1
185+
)
186+
return matches / total_keywords
187+
188+
def _semantic_score(
189+
self, query_embed: Any, lesson: Lesson, context: MatchContext
190+
) -> float:
191+
"""Cosine similarity between query and lesson (0.0-1.0)."""
192+
if not self.embedder:
193+
return 0.0
194+
195+
# Generate lesson embedding from title and body
196+
lesson_text = f"{lesson.title}\n{lesson.body[:500]}" # Limit to first 500 chars
197+
lesson_embed = self.embedder.encode(lesson_text, convert_to_numpy=True)
198+
199+
# Cosine similarity
200+
similarity = float(
201+
np.dot(query_embed, lesson_embed)
202+
/ (np.linalg.norm(query_embed) * np.linalg.norm(lesson_embed))
203+
)
204+
205+
# Normalize from [-1, 1] to [0, 1]
206+
return (similarity + 1.0) / 2.0
207+
208+
def _effectiveness_score(self, lesson: Lesson) -> float:
209+
"""Effectiveness score from metadata (0.0-1.0).
210+
211+
NOTE: Returns neutral 0.5 until ACE metadata schema implemented.
212+
Will use helpful_count/harmful_count when available (Phase 1).
213+
"""
214+
# TODO: Implement when ACE metadata available
215+
# helpful = lesson.metadata.helpful_count
216+
# harmful = lesson.metadata.harmful_count
217+
# return helpful / (helpful + harmful + 1)
218+
return 0.5
219+
220+
def _recency_score(self, lesson: Lesson) -> float:
221+
"""Recency score with exponential decay (0.0-1.0).
222+
223+
NOTE: Returns 1.0 (assume recent) until ACE metadata implemented.
224+
Will use updated timestamp when available (Phase 1).
225+
"""
226+
# TODO: Implement when ACE metadata available
227+
# from datetime import datetime, timezone
228+
# updated = lesson.metadata.updated
229+
# now = datetime.now(timezone.utc)
230+
# days_since = (now - updated).days
231+
# return exp(-days_since / self.config.recency_decay_days)
232+
return 1.0
233+
234+
def _tool_bonus(self, lesson: Lesson, context: MatchContext) -> float:
235+
"""Tool match bonus (0.0 or configured bonus)."""
236+
if not context.tools_used or not lesson.metadata.tools:
237+
return 0.0
238+
239+
for tool in lesson.metadata.tools:
240+
if tool in context.tools_used:
241+
return self.config.tool_bonus
242+
243+
return 0.0

0 commit comments

Comments
 (0)