feat(session-start): rank injected instincts by project/stack relevance - #2466
Conversation
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds project- and stack-based relevance scoring for SessionStart-injected instincts. Integrates scoring behind an environment flag, documents the flag, and adds utility and hook tests for enabled and confidence-only ordering. ChangesInstinct relevance ranking
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant SessionStart
participant RelevanceModule
participant ProjectFilesystem
SessionStart->>RelevanceModule: isRelevanceRankingEnabled()
alt ranking enabled
SessionStart->>RelevanceModule: detectStackKeywords(projectRoot)
RelevanceModule->>ProjectFilesystem: inspect project markers
ProjectFilesystem-->>RelevanceModule: detected stack signals
RelevanceModule-->>SessionStart: stackKeywords
loop each instinct
SessionStart->>RelevanceModule: computeRelevanceBoost(instinct, stackKeywords)
RelevanceModule-->>SessionStart: relevance boost
end
else ranking disabled
SessionStart->>SessionStart: assign zero relevance
end
SessionStart->>SessionStart: sort by confidence plus relevance
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
ECC bundle files are already tracked in this repository. Skipping generation of another bundle PR. |
|
| Filename | Overview |
|---|---|
| README.md | Documents the new ECC_INSTINCT_RELEVANCE_RANKING toggle alongside existing SessionStart instinct settings. |
| scripts/hooks/session-start.js | Integrates relevance boost computation into active instinct ranking while preserving threshold, cap, and opt-out behavior. |
| scripts/lib/instinct-relevance.js | Adds focused helpers for stack detection, token-based instinct matching, boost calculation, and feature-flag parsing. |
| tests/hooks/hooks.test.js | Adds an end-to-end SessionStart test covering relevance-enabled and confidence-only ordering. |
| tests/lib/instinct-relevance.test.js | Adds unit coverage for stack detection markers, token matching, boost math, and environment toggle parsing. |
Sequence Diagram
sequenceDiagram
participant Hook as SessionStart hook
participant Env as Environment
participant Detector as instinct-relevance
participant Store as Instinct directories
participant Output as Additional context
Hook->>Env: Read ECC_INSTINCT_RELEVANCE_RANKING
Hook->>Store: Load project and global instincts
alt relevance ranking enabled
Hook->>Detector: detectStackKeywords(projectRoot)
Detector-->>Hook: stack keyword Set
Hook->>Detector: computeRelevanceBoost(instinct, keywords)
Detector-->>Hook: additive boost
else relevance ranking disabled
Hook->>Hook: Use zero relevance boost
end
Hook->>Hook: Filter by confidence threshold and action
Hook->>Hook: Sort by confidence + relevance, scope, id
Hook->>Output: Inject top capped active instincts
Reviews (3): Last reviewed commit: "test: isolate instinct relevance environ..." | Re-trigger Greptile
|
ECC bundle files are already tracked in this repository. Skipping generation of another bundle PR. |
Instinct selection at SessionStart ranked purely by confidence, so a high-confidence instinct about an unrelated stack could take an injection slot from a lower-confidence instinct that is actually relevant to the current project. Rank by confidence + location/stack relevance instead: project-scoped instincts, and instincts whose domain/trigger matches the detected stack (languages/frameworks via detectProjectType, plus terraform/dbt markers), get a small additive boost. The confidence>=threshold floor and the injection cap are unchanged, and ranking degrades to confidence-only when nothing matches or when ECC_INSTINCT_RELEVANCE_RANKING is set to off. The ranking helpers live in scripts/lib/instinct-relevance.js with unit coverage in tests/lib/, plus an end-to-end ordering test in tests/hooks/. Completes part (b) of affaan-m#2371; part (a) (configurable count + threshold) shipped in affaan-m#2413. Fixes affaan-m#2371
…inct sort Greptile flagged that the secondary `right.confidence` comparison in summarizeActiveInstincts' sort was dead code when relevance ranking is disabled and, when enabled, was reached only on a floating-point tie of the combined score — where it skipped the intended scope-label tiebreaker. Remove it: the primary combined-score comparison already reduces to confidence-only ordering when relevance is off, so behavior there is unchanged; a genuine combined-score tie now falls through to the documented scope-first, then id, order.
465bd0d to
41d37bb
Compare
|
ECC bundle files are already tracked in this repository. Skipping generation of another bundle PR. |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/hooks/session-start.js`:
- Around line 460-469: Update the comparator in the visible sort callback so
equal-confidence items preserve their existing order when both left._relevance
and right._relevance are zero. Return 0 before applying _scopeLabel and id
tie-breakers in that case, while retaining the new tie-breakers when either
relevance boost contributes to the score.
In `@scripts/lib/instinct-relevance.js`:
- Around line 37-45: Update hasFileWithExtension and the other referenced
filesystem-detection catch blocks to emit a sanitized diagnostic to stderr
before returning the existing fallback value. Handle errors explicitly without
exposing sensitive paths or details, while preserving the current fallback
behavior.
In `@tests/hooks/hooks.test.js`:
- Around line 622-648: Update the baseEnv fixture used by the session-start
ranking test to explicitly enable relevance ranking and reset the confidence
threshold and result cap to their expected defaults. Keep these three ECC
instinct configuration variables in baseEnv so both the “on” and opt-out
scenarios are isolated from inherited process.env settings; preserve the
existing off-case override for relevance ranking.
- Around line 603-657: Extract the SessionStart relevance test anchored by
“ranks stack-relevant instincts above higher-confidence unrelated ones (`#2371`)”
from this monolithic test runner into a focused SessionStart test module. Move
or reuse its setup, environment construction, script execution, context parsing,
and cleanup through shared harness helpers, keeping both relevance-on and
relevance-off assertions unchanged; remove the test and any newly duplicated
helpers from the original runner.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 5840c1aa-a467-4fc2-b013-49005c1e87f9
📒 Files selected for processing (4)
README.mdscripts/hooks/session-start.jsscripts/lib/instinct-relevance.jstests/hooks/hooks.test.js
📜 Review details
⏰ Context from checks skipped due to timeout. (27)
- GitHub Check: Test (ubuntu-latest, Node 18.x, yarn)
- GitHub Check: Test (windows-latest, Node 18.x, pnpm)
- GitHub Check: Test (windows-latest, Node 20.x, pnpm)
- GitHub Check: Test (windows-latest, Node 20.x, yarn)
- GitHub Check: Test (macos-latest, Node 20.x, yarn)
- GitHub Check: Test (ubuntu-latest, Node 22.x, yarn)
- GitHub Check: Test (macos-latest, Node 22.x, npm)
- GitHub Check: Test (windows-latest, Node 18.x, npm)
- GitHub Check: Test (macos-latest, Node 18.x, bun)
- GitHub Check: Test (windows-latest, Node 22.x, pnpm)
- GitHub Check: Test (macos-latest, Node 18.x, pnpm)
- GitHub Check: Test (windows-latest, Node 18.x, yarn)
- GitHub Check: Test (ubuntu-latest, Node 18.x, bun)
- GitHub Check: Test (ubuntu-latest, Node 18.x, pnpm)
- GitHub Check: Test (ubuntu-latest, Node 22.x, pnpm)
- GitHub Check: Test (ubuntu-latest, Node 20.x, bun)
- GitHub Check: Test (ubuntu-latest, Node 22.x, bun)
- GitHub Check: Test (macos-latest, Node 18.x, yarn)
- GitHub Check: Test (windows-latest, Node 22.x, yarn)
- GitHub Check: Test (ubuntu-latest, Node 20.x, yarn)
- GitHub Check: Test (windows-latest, Node 22.x, npm)
- GitHub Check: Test (windows-latest, Node 20.x, npm)
- GitHub Check: Test (ubuntu-latest, Node 22.x, npm)
- GitHub Check: Test (ubuntu-latest, Node 20.x, pnpm)
- GitHub Check: Test (ubuntu-latest, Node 20.x, npm)
- GitHub Check: Test (ubuntu-latest, Node 18.x, npm)
- GitHub Check: Coverage
🧰 Additional context used
📓 Path-based instructions (18)
README.md
📄 CodeRabbit inference engine (CLAUDE.md)
When working on README.md files, use the
/readmeskill.
Files:
README.md
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: Use specialized agents proactively for planning, implementation review, testing, security review, build resolution, and domain-specific tasks; run independent operations in parallel.
Write tests before implementation, follow the RED-GREEN-IMPROVE TDD workflow, and maintain at least 80% coverage.
Never compromise security: validate all inputs, prevent injection and XSS, enable CSRF protection, verify authentication and authorization, rate-limit endpoints, and avoid leaking sensitive error details.
Never hardcode secrets; use environment variables or a secret manager, validate required secrets at startup, and rotate exposed secrets immediately.
If a security issue is found, stop, use the security-reviewer agent, fix critical issues, rotate exposed secrets, and search for similar vulnerabilities.
Always create new objects and never mutate existing ones.
Organize code by feature or domain with high cohesion and low coupling; prefer many small files over a few large files, typically 200–400 lines and no more than 800 lines.
Handle errors at every level, show user-friendly messages in UI code, log detailed context server-side, and never silently swallow errors.
Validate all user input at system boundaries using schema-based validation; fail fast with clear messages and never trust external data.
Keep functions under 50 lines, files focused and under 800 lines, avoid nesting deeper than four levels, avoid hardcoded values, and use readable, well-named identifiers.
All required tests include unit tests for functions, utilities, and components; integration tests for APIs and databases; and E2E tests for critical user flows.
Troubleshoot test failures by checking isolation, verifying mocks, and fixing implementation rather than tests unless the tests are incorrect.
Before committing, use Conventional Commits format:<type>: <description>, with types such as feat, fix, refactor, docs, test, chore, perf, and ci.
For pull requests, analyze the full commit history, draf...
Files:
README.mdscripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,go,rb,php,scala,kt}
📄 CodeRabbit inference engine (.cursor/rules/common-coding-style.md)
**/*.{js,ts,jsx,tsx,py,java,cs,go,rb,php,scala,kt}: Always create new objects, never mutate existing ones. Use immutable patterns to prevent hidden side effects and enable safe concurrency
Organize code into many small files (200-400 lines typical, 800 lines max) organized by feature/domain rather than by type
Always handle errors explicitly at every level and never silently swallow errors
Always validate all user input before processing at system boundaries
Use schema-based validation where available
Fail fast with clear error messages when validation fails
Never trust external data (API responses, user input, file content)
Ensure code is readable and well-named
Keep functions small (less than 50 lines)
Keep files focused (less than 800 lines)
Avoid deep nesting (more than 4 levels)
Do not use hardcoded values; use constants or configuration instead
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php,swift,kt,rs,c,cpp,h,hpp}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
No hardcoded secrets (API keys, passwords, tokens) - validate before any commit
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php}: All user inputs must be validated
Enable CSRF protection on all state-changing endpoints
Verify authentication and authorization for all protected endpoints
Implement rate limiting on all endpoints to prevent abuse
Ensure error messages do not leak sensitive data in responses
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php,sql}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
Use parameterized queries to prevent SQL injection
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,html,php,java,cs,rb,go}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
Implement XSS prevention by sanitizing HTML output
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php,swift,kt,rs,c,cpp,h,hpp,properties,yml,yaml,json,env,config}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
NEVER hardcode secrets in source code - ALWAYS use environment variables or a secret manager
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.cursor/rules/typescript-coding-style.md)
**/*.{ts,tsx,js,jsx}: Use spread operator for immutable updates in TypeScript/JavaScript instead of direct mutation
Use async/await with try-catch for error handling in TypeScript/JavaScript
Use Zod for schema-based input validation in TypeScript/JavaScript
No console.log statements in production code; use proper logging libraries instead
**/*.{ts,tsx,js,jsx}: Auto-format JavaScript/TypeScript files using Prettier after edit
Warn aboutconsole.logstatements in edited files
Check all modified files forconsole.logstatements before session ends
**/*.{ts,tsx,js,jsx}: Use the ApiResponse interface pattern with generic type parameter:interface ApiResponse<T> { success: boolean; data?: T; error?: string; meta?: { total: number; page: number; limit: number; } }
Implement custom React hooks following the pattern: export a named function with use prefix, generic type parameters, and proper useEffect cleanup for side effects
**/*.{ts,tsx,js,jsx}: Never hardcode secrets; always use environment variables for sensitive credentials like API keys
Throw an error when required environment variables are not configured to fail fast and ensure security prerequisites are metUse Playwright as the E2E testing framework for critical user flows in TypeScript/JavaScript
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
{package.json,*.config.js,scripts/**/*.js}
📄 CodeRabbit inference engine (CLAUDE.md)
Package manager detection should support npm, pnpm, yarn, and bun, with configuration via CLAUDE_PACKAGE_MANAGER environment variable or project config.
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.js
scripts/**/*.js
📄 CodeRabbit inference engine (CLAUDE.md)
Ensure cross-platform support for Windows, macOS, and Linux via Node.js scripts in the scripts/ directory.
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.js
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{js,ts,jsx,tsx}: Always create new objects and never mutate in place; return new copies instead
Keep files between 200–400 lines typical, with a maximum of 800 lines
Extract helpers when a file exceeds 200 lines
Handle errors explicitly at every level; never swallow errors silently
Validate all user input before processing; use schema-based validation where available
Never trust external data (API responses, file content, query params); always validate
All user inputs must be validated and sanitized
Error messages must be scrubbed of sensitive internals
Use readable, well-named identifiers in all code
Keep functions under 50 lines
Keep files under 800 lines
Avoid nesting deeper than 4 levels
Implement comprehensive error handling in all code
Do not hardcode values; use constants or environment configuration instead
Do not use in-place mutation; always return new objects or state
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,json,env*}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Do not hardcode secrets, API keys, passwords, or tokens
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{js,ts}: Use parameterized queries for all database writes (no string interpolation)
Auth/authz must be checked server-side for every sensitive path
Rate limiting must be applied to all public endpoints
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{jsx,tsx,js,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
HTML output must be sanitized where applicable
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
**/*.{js,ts,env*}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Required environment variables must be validated at startup
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.jstests/hooks/hooks.test.js
{scripts,bin}/**
⚙️ CodeRabbit configuration file
{scripts,bin}/**: Focus on command injection, unsafe subprocess usage, path traversal, SSRF, secret exposure, and missing tests for new CLI behavior.
Files:
scripts/lib/instinct-relevance.jsscripts/hooks/session-start.js
**/*.{test,spec}.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{test,spec}.{js,ts,jsx,tsx}: Write tests before implementation (test-driven development); target 80%+ coverage
Achieve minimum 80% test coverage across all three layers: Unit, Integration, and E2E
Use AAA structure (Arrange / Act / Assert) in tests with descriptive test names that explain behavior under test
Files:
tests/hooks/hooks.test.js
🧠 Learnings (2)
📚 Learning: 2026-06-27T23:49:19.839Z
Learnt from: gaurav0107
Repo: affaan-m/ECC PR: 2373
File: tests/hooks/observe-signal-timeout.test.js:0-0
Timestamp: 2026-06-27T23:49:19.839Z
Learning: In tests under tests/hooks that require a Python runtime to run, the test should fail fast when Python isn’t available (or prerequisites aren’t met). Do not treat a missing Python runtime as test.skip, as an expected/allowed condition, or as a passing state; instead, explicitly fail (e.g., throw/return a rejected promise or use a test runner fail/expect that marks the test as failed) so reviewers can’t accidentally mask environment issues.
Applied to files:
tests/hooks/hooks.test.js
📚 Learning: 2026-07-14T03:26:12.530Z
Learnt from: thejesh23
Repo: affaan-m/ECC PR: 2517
File: tests/hooks/pre-bash-tmux-reminder.test.js:21-25
Timestamp: 2026-07-14T03:26:12.530Z
Learning: In this repository, do not flag `console.log` usage as a guideline violation in hook test files under `tests/hooks/*.test.js`. These tests intentionally use `console.log` for pass/fail output because the repo’s console-based runner (`tests/run-all.js`) is used and there is no Jest/Mocha dependency. Outside this specific hook-test path, follow the normal logging guidelines.
Applied to files:
tests/hooks/hooks.test.js
🪛 ast-grep (0.44.1)
tests/hooks/hooks.test.js
[warning] 609-612: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(
path.join(instinctsDir, 'terraform-first.md'),
'---\nid: terraform-first\nconfidence: 0.75\ndomain: terraform\n---\n## Action\nRun terraform plan before every apply.\n'
)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 613-616: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(
path.join(instinctsDir, 'unrelated-high.md'),
'---\nid: unrelated-high\nconfidence: 0.9\ndomain: python\n---\n## Action\nPin Python dependencies in requirements.txt.\n'
)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 619-619: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(path.join(projectRoot, 'main.tf'), 'resource "null_resource" "x" {}\n')
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 2535-2535: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(hooksPath, 'utf8')
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 3235-3235: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(path.join(skillRoot, 'scripts', 'lib', 'homunculus-dir.sh'), 'utf8')
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 3246-3246: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(path.join(skillRoot, ...rel), 'utf8')
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 3258-3258: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(path.join(__dirname, '..', '..', 'skills', 'continuous-learning-v2', 'agents', 'observer-loop.sh'), 'utf8')
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 4326-4326: Avoid require with non-literal values
Context: require(path.join(scriptsDir, 'post-edit-console-warn.js'))
Note: [CWE-829] Inclusion of Functionality from Untrusted Control Sphere (dynamic require).
(detect-non-literal-require)
🔇 Additional comments (4)
scripts/lib/instinct-relevance.js (1)
17-36: LGTM!Also applies to: 61-87, 96-173
scripts/hooks/session-start.js (1)
27-31: LGTM!Also applies to: 430-458
tests/hooks/hooks.test.js (1)
2533-2557: LGTM!Also applies to: 2707-2710, 3233-3266, 4326-4335
README.md (1)
1368-1374: LGTM!
| .sort((left, right) => { | ||
| if (right.confidence !== left.confidence) return right.confidence - left.confidence; | ||
| // Primary: combined confidence + relevance. When relevance is off every | ||
| // _relevance is 0, so this reduces to the prior confidence-only ordering. | ||
| // Tie-breaks on a genuinely equal combined score: project scope first, | ||
| // then id (deterministic). | ||
| const leftScore = left.confidence + left._relevance; | ||
| const rightScore = right.confidence + right._relevance; | ||
| if (rightScore !== leftScore) return rightScore - leftScore; | ||
| if (left._scopeLabel !== right._scopeLabel) return left._scopeLabel === 'project' ? -1 : 1; | ||
| return String(left.id).localeCompare(String(right.id)); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve prior ordering when relevance contributes nothing.
When ranking is disabled, or no instinct receives a boost, equal-confidence items now reorder by scope and id; that violates the stated confidence-only/preserved-order fallback. Return 0 for equal scores when both relevance boosts are zero, and only apply the new tie-breakers when relevance actually differentiates the ranking.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/hooks/session-start.js` around lines 460 - 469, Update the comparator
in the visible sort callback so equal-confidence items preserve their existing
order when both left._relevance and right._relevance are zero. Return 0 before
applying _scopeLabel and id tie-breakers in that case, while retaining the new
tie-breakers when either relevance boost contributes to the score.
| function hasFileWithExtension(root, extensions) { | ||
| try { | ||
| return fs.readdirSync(root, { withFileTypes: true }).some( | ||
| (entry) => entry.isFile() && extensions.includes(path.extname(entry.name)) | ||
| ); | ||
| } catch { | ||
| return false; | ||
| } | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Do not silently suppress stack-detection failures.
Filesystem/project-detection errors become “no stack detected,” silently reverting ranking behavior. Emit a sanitized stderr diagnostic before falling back so permission or malformed-project failures are diagnosable.
As per coding guidelines, “Always handle errors explicitly at every level and never silently swallow errors.”
Also applies to: 53-59, 88-95
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/lib/instinct-relevance.js` around lines 37 - 45, Update
hasFileWithExtension and the other referenced filesystem-detection catch blocks
to emit a sanitized diagnostic to stderr before returning the existing fallback
value. Handle errors explicitly without exposing sensitive paths or details,
while preserving the current fallback behavior.
Source: Coding guidelines
| if ( | ||
| await asyncTest('ranks stack-relevant instincts above higher-confidence unrelated ones (#2371)', async () => { | ||
| const isoHome = fs.mkdtempSync(path.join(os.tmpdir(), 'ecc-instinct-relevance-')); | ||
| const homunculusDir = path.join(isoHome, 'homunculus'); | ||
| const instinctsDir = path.join(homunculusDir, 'instincts', 'personal'); | ||
| fs.mkdirSync(instinctsDir, { recursive: true }); | ||
| // A stack-matching 0.75 instinct and an unrelated higher-confidence 0.9. | ||
| fs.writeFileSync( | ||
| path.join(instinctsDir, 'terraform-first.md'), | ||
| '---\nid: terraform-first\nconfidence: 0.75\ndomain: terraform\n---\n## Action\nRun terraform plan before every apply.\n' | ||
| ); | ||
| fs.writeFileSync( | ||
| path.join(instinctsDir, 'unrelated-high.md'), | ||
| '---\nid: unrelated-high\nconfidence: 0.9\ndomain: python\n---\n## Action\nPin Python dependencies in requirements.txt.\n' | ||
| ); | ||
| // A project root that detects as terraform via a *.tf marker. | ||
| const projectRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'ecc-tf-project-')); | ||
| fs.writeFileSync(path.join(projectRoot, 'main.tf'), 'resource "null_resource" "x" {}\n'); | ||
|
|
||
| const baseEnv = { | ||
| HOME: isoHome, | ||
| USERPROFILE: isoHome, | ||
| CLV2_HOMUNCULUS_DIR: homunculusDir, | ||
| CLAUDE_PROJECT_DIR: projectRoot, | ||
| }; | ||
|
|
||
| try { | ||
| const on = await runScript(path.join(scriptsDir, 'session-start.js'), '', baseEnv); | ||
| assert.strictEqual(on.code, 0); | ||
| const ctxOn = getSessionStartAdditionalContext(on.stdout); | ||
| const tfOn = ctxOn.indexOf('Run terraform plan before every apply.'); | ||
| const pyOn = ctxOn.indexOf('Pin Python dependencies in requirements.txt.'); | ||
| assert.ok(tfOn !== -1 && pyOn !== -1, `both instincts should inject, ctx: ${ctxOn}`); | ||
| assert.ok(tfOn < pyOn, `stack-matching 0.75 should rank above unrelated 0.9 when relevance is on, ctx: ${ctxOn}`); | ||
|
|
||
| // Opting out restores pure confidence ordering (0.9 before 0.75). | ||
| const off = await runScript(path.join(scriptsDir, 'session-start.js'), '', { | ||
| ...baseEnv, | ||
| ECC_INSTINCT_RELEVANCE_RANKING: 'off', | ||
| }); | ||
| assert.strictEqual(off.code, 0); | ||
| const ctxOff = getSessionStartAdditionalContext(off.stdout); | ||
| const tfOff = ctxOff.indexOf('Run terraform plan before every apply.'); | ||
| const pyOff = ctxOff.indexOf('Pin Python dependencies in requirements.txt.'); | ||
| assert.ok(tfOff !== -1 && pyOff !== -1, `both instincts should still inject, ctx: ${ctxOff}`); | ||
| assert.ok(pyOff < tfOff, `with ranking off, higher-confidence 0.9 should rank first, ctx: ${ctxOff}`); | ||
| } finally { | ||
| fs.rmSync(isoHome, { recursive: true, force: true }); | ||
| fs.rmSync(projectRoot, { recursive: true, force: true }); | ||
| } | ||
| }) | ||
| ) | ||
| passed++; | ||
| else failed++; | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
Move this SessionStart test into a focused test module.
This adds to a 6,000+ line test runner. Extract the SessionStart relevance coverage and shared harness into focused files rather than extending this monolith.
As per coding guidelines, “Keep files under 800 lines” and “Extract helpers when a file exceeds 200 lines.”
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 609-612: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(
path.join(instinctsDir, 'terraform-first.md'),
'---\nid: terraform-first\nconfidence: 0.75\ndomain: terraform\n---\n## Action\nRun terraform plan before every apply.\n'
)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 613-616: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(
path.join(instinctsDir, 'unrelated-high.md'),
'---\nid: unrelated-high\nconfidence: 0.9\ndomain: python\n---\n## Action\nPin Python dependencies in requirements.txt.\n'
)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 619-619: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(path.join(projectRoot, 'main.tf'), 'resource "null_resource" "x" {}\n')
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/hooks/hooks.test.js` around lines 603 - 657, Extract the SessionStart
relevance test anchored by “ranks stack-relevant instincts above
higher-confidence unrelated ones (`#2371`)” from this monolithic test runner into
a focused SessionStart test module. Move or reuse its setup, environment
construction, script execution, context parsing, and cleanup through shared
harness helpers, keeping both relevance-on and relevance-off assertions
unchanged; remove the test and any newly duplicated helpers from the original
runner.
Source: Coding guidelines
|
Rebased onto current The conflict was in The diff against Verified locally: |
|
ECC bundle files are already tracked in this repository. Skipping generation of another bundle PR. |
|
Thank you so much for the careful follow-through here, and apologies for the review delay. The project/stack-aware ranking is the remaining useful half of #2371: it keeps relevant learned instincts from being crowded out while preserving the existing threshold, cap, and opt-out. I refreshed the branch onto current main and isolated the integration fixture from inherited environment overrides in 92994ef. The focused relevance suite passes 18/18 and the hook suite passes 249/249. Required hosted CI is rerunning on that exact head; we will merge once it is green through the serial main-branch gate. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/hooks/hooks.test.js`:
- Around line 627-629: The isolated hook fixtures must override inherited
environment settings used by runScript. In tests/hooks/hooks.test.js:627-629,
add explicit values for ECC_SESSION_START_CONTEXT, ECC_SESSION_START_MAX_CHARS,
and ECC_AGENT_DATA_HOME alongside the existing environment fixtures; in
tests/hooks/hooks.test.js:4759-4762, set ECC_AGENT_DATA_HOME to the temporary
.claude directory used by that fixture.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 01615585-d8c1-43e9-aa14-4c862275684a
📒 Files selected for processing (2)
README.mdtests/hooks/hooks.test.js
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (15)
**/*.{js,ts,jsx,tsx,py,java,cs,go,rb,php,scala,kt}
📄 CodeRabbit inference engine (.cursor/rules/common-coding-style.md)
**/*.{js,ts,jsx,tsx,py,java,cs,go,rb,php,scala,kt}: Always create new objects, never mutate existing ones. Use immutable patterns to prevent hidden side effects and enable safe concurrency
Organize code into many small files (200-400 lines typical, 800 lines max) organized by feature/domain rather than by type
Always handle errors explicitly at every level and never silently swallow errors
Always validate all user input before processing at system boundaries
Use schema-based validation where available
Fail fast with clear error messages when validation fails
Never trust external data (API responses, user input, file content)
Ensure code is readable and well-named
Keep functions small (less than 50 lines)
Keep files focused (less than 800 lines)
Avoid deep nesting (more than 4 levels)
Do not use hardcoded values; use constants or configuration instead
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php,swift,kt,rs,c,cpp,h,hpp}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
No hardcoded secrets (API keys, passwords, tokens) - validate before any commit
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php}: All user inputs must be validated
Enable CSRF protection on all state-changing endpoints
Verify authentication and authorization for all protected endpoints
Implement rate limiting on all endpoints to prevent abuse
Ensure error messages do not leak sensitive data in responses
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php,sql}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
Use parameterized queries to prevent SQL injection
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,html,php,java,cs,rb,go}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
Implement XSS prevention by sanitizing HTML output
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,py,java,cs,rb,go,php,swift,kt,rs,c,cpp,h,hpp,properties,yml,yaml,json,env,config}
📄 CodeRabbit inference engine (.cursor/rules/common-security.md)
NEVER hardcode secrets in source code - ALWAYS use environment variables or a secret manager
Files:
tests/hooks/hooks.test.js
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.cursor/rules/typescript-coding-style.md)
**/*.{ts,tsx,js,jsx}: Use spread operator for immutable updates in TypeScript/JavaScript instead of direct mutation
Use async/await with try-catch for error handling in TypeScript/JavaScript
Use Zod for schema-based input validation in TypeScript/JavaScript
No console.log statements in production code; use proper logging libraries instead
**/*.{ts,tsx,js,jsx}: Auto-format JavaScript/TypeScript files using Prettier after edit
Warn aboutconsole.logstatements in edited files
Check all modified files forconsole.logstatements before session ends
**/*.{ts,tsx,js,jsx}: Use the ApiResponse interface pattern with generic type parameter:interface ApiResponse<T> { success: boolean; data?: T; error?: string; meta?: { total: number; page: number; limit: number; } }
Implement custom React hooks following the pattern: export a named function with use prefix, generic type parameters, and proper useEffect cleanup for side effects
**/*.{ts,tsx,js,jsx}: Never hardcode secrets; always use environment variables for sensitive credentials like API keys
Throw an error when required environment variables are not configured to fail fast and ensure security prerequisites are metUse Playwright as the E2E testing framework for critical user flows in TypeScript/JavaScript
Files:
tests/hooks/hooks.test.js
**/*.{test,spec}.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{test,spec}.{js,ts,jsx,tsx}: Write tests before implementation (test-driven development); target 80%+ coverage
Achieve minimum 80% test coverage across all three layers: Unit, Integration, and E2E
Use AAA structure (Arrange / Act / Assert) in tests with descriptive test names that explain behavior under test
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{js,ts,jsx,tsx}: Always create new objects and never mutate in place; return new copies instead
Keep files between 200–400 lines typical, with a maximum of 800 lines
Extract helpers when a file exceeds 200 lines
Handle errors explicitly at every level; never swallow errors silently
Validate all user input before processing; use schema-based validation where available
Never trust external data (API responses, file content, query params); always validate
All user inputs must be validated and sanitized
Error messages must be scrubbed of sensitive internals
Use readable, well-named identifiers in all code
Keep functions under 50 lines
Keep files under 800 lines
Avoid nesting deeper than 4 levels
Implement comprehensive error handling in all code
Do not hardcode values; use constants or environment configuration instead
Do not use in-place mutation; always return new objects or state
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,jsx,tsx,json,env*}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Do not hardcode secrets, API keys, passwords, or tokens
Files:
tests/hooks/hooks.test.js
**/*.{js,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{js,ts}: Use parameterized queries for all database writes (no string interpolation)
Auth/authz must be checked server-side for every sensitive path
Rate limiting must be applied to all public endpoints
Files:
tests/hooks/hooks.test.js
**/*.{jsx,tsx,js,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
HTML output must be sanitized where applicable
Files:
tests/hooks/hooks.test.js
**/*.{js,ts,env*}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Required environment variables must be validated at startup
Files:
tests/hooks/hooks.test.js
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: Delegate complex features to a planner agent, architectural decisions to an architect agent, modified code to a code-reviewer agent, and security-sensitive work to a security-reviewer agent; use parallel agents for independent operations.
Never compromise security; validate all inputs and prevent hardcoded secrets, injection, XSS, CSRF, authentication or authorization failures, sensitive error leakage, and missing rate limits.
Never hardcode secrets; use environment variables or a secret manager, validate required secrets at startup, and rotate exposed secrets immediately.
Always create new objects and never mutate existing ones.
Plan complex features before implementation, identifying dependencies, risks, and phases.
Prefer many small, focused files; keep functions under 50 lines, files under 800 lines where practical, avoid nesting deeper than four levels, and use readable, well-named identifiers.
Handle errors at every level, provide user-friendly UI messages, log detailed server-side context, and never silently swallow errors.
Validate all user input at system boundaries using schema-based validation; fail fast with clear messages and never trust external data.
Required tests include unit tests, integration tests for APIs and database operations, and end-to-end tests for critical user flows.
Follow the mandatory TDD cycle: write a failing test, implement the minimum passing solution, then refactor and verify coverage.
Use a consistent API response envelope containing a success indicator, data payload, error message, and pagination metadata.
Encapsulate data access behind a repository interface with operations such asfindAll,findById,create,update, anddelete; business logic must depend on the abstraction rather than storage details.
Files:
tests/hooks/hooks.test.jsREADME.md
README.md
📄 CodeRabbit inference engine (CLAUDE.md)
When working on README.md files, use the
/readmeskill.
Files:
README.md
🧠 Learnings (2)
📚 Learning: 2026-06-27T23:49:19.839Z
Learnt from: gaurav0107
Repo: affaan-m/ECC PR: 2373
File: tests/hooks/observe-signal-timeout.test.js:0-0
Timestamp: 2026-06-27T23:49:19.839Z
Learning: In tests under tests/hooks that require a Python runtime to run, the test should fail fast when Python isn’t available (or prerequisites aren’t met). Do not treat a missing Python runtime as test.skip, as an expected/allowed condition, or as a passing state; instead, explicitly fail (e.g., throw/return a rejected promise or use a test runner fail/expect that marks the test as failed) so reviewers can’t accidentally mask environment issues.
Applied to files:
tests/hooks/hooks.test.js
📚 Learning: 2026-07-14T03:26:12.530Z
Learnt from: thejesh23
Repo: affaan-m/ECC PR: 2517
File: tests/hooks/pre-bash-tmux-reminder.test.js:21-25
Timestamp: 2026-07-14T03:26:12.530Z
Learning: In this repository, do not flag `console.log` usage as a guideline violation in hook test files under `tests/hooks/*.test.js`. These tests intentionally use `console.log` for pass/fail output because the repo’s console-based runner (`tests/run-all.js`) is used and there is no Jest/Mocha dependency. Outside this specific hook-test path, follow the normal logging guidelines.
Applied to files:
tests/hooks/hooks.test.js
🪛 ast-grep (0.45.0)
tests/hooks/hooks.test.js
[warning] 3824-3824: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(otherTmpFile, 'some other data\n')
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 4751-4751: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(olderSession, olderContent)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
[warning] 4755-4755: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.writeFileSync(newerSession, buildSessionStartFixture('', { title: '# Newer Session' }))
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🔇 Additional comments (2)
tests/hooks/hooks.test.js (1)
1308-1310: LGTM!Also applies to: 3824-3824, 6272-6274
README.md (1)
1437-1443: LGTM!
| ECC_INSTINCT_RELEVANCE_RANKING: 'on', | ||
| ECC_INSTINCT_CONFIDENCE_THRESHOLD: '0.7', | ||
| ECC_MAX_INJECTED_INSTINCTS: '6', |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Prevent inherited environment settings from invalidating isolated hook tests.
runScript spreads process.env into each child process. Isolated fixtures must override the environment variables that control SessionStart output and session-data paths.
tests/hooks/hooks.test.js#L627-L629: set explicit values forECC_SESSION_START_CONTEXT,ECC_SESSION_START_MAX_CHARS, andECC_AGENT_DATA_HOME.tests/hooks/hooks.test.js#L4759-L4762: setECC_AGENT_DATA_HOMEto the temporary.claudedirectory.
📍 Affects 1 file
tests/hooks/hooks.test.js#L627-L629(this comment)tests/hooks/hooks.test.js#L4759-L4762
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/hooks/hooks.test.js` around lines 627 - 629, The isolated hook fixtures
must override inherited environment settings used by runScript. In
tests/hooks/hooks.test.js:627-629, add explicit values for
ECC_SESSION_START_CONTEXT, ECC_SESSION_START_MAX_CHARS, and ECC_AGENT_DATA_HOME
alongside the existing environment fixtures; in
tests/hooks/hooks.test.js:4759-4762, set ECC_AGENT_DATA_HOME to the temporary
.claude directory used by that fixture.
What Changed
Completes part (b) of #2371.
scripts/hooks/session-start.jsselected whichinstincts to inject at
SessionStartby confidence alone, so ahigh-confidence instinct about an unrelated stack could take an injection slot
from a lower-confidence instinct that is actually relevant to the current
project.
scripts/lib/instinct-relevance.js: pure, unit-tested helpers —detectStackKeywords()(reusesdetectProjectType, plus the*.tf/*.tfvars→ terraform and
dbt_project.yml→ dbt markers the issue names),instinctMatchesStack()(whole-token match overdomain/trigger/stack),computeRelevanceBoost(), andisRelevanceRankingEnabled().summarizeActiveInstincts()now ranks by confidence + relevance:project-scoped instincts and instincts whose domain/trigger match the detected
stack get a small additive boost. The
confidence >= thresholdfloor and theinjection cap are untouched.
ECC_INSTINCT_RELEVANCE_RANKING(default on;off/false/0/norestores confidence-only ranking). When nothing matches, ordering is identical
to before.
Why This Change
Part (a) (#2413) made the injection count + confidence threshold configurable
but left selection confidence-only. As an instinct library grows, an unrelated
high-confidence instinct (e.g. a terraform instinct in a Node repo) can crowd
out a relevant one. At
SessionStartthere is no user task yet, so relevance islocation/stack relevance — project scope + detected stack — which is exactly
what this ranks on. Task-relevance and guardrail-pinning are deliberately left
to the future-work items the issue calls out.
Testing Done
node tests/run-all.js— 2972/2972 pass (clean env,CLAUDE_PLUGIN_ROOTunset)tests/lib/instinct-relevance.test.js— 18 unit tests (stack detection incl.*.tf/dbt, token matching incl.go!=good, boost math, toggle parsing)tests/hooks/hooks.test.jsend-to-end test: a stack-matching 0.75 instinct ranks above an unrelated 0.9 in a*.tfproject;ECC_INSTINCT_RELEVANCE_RANKING=offrestores confidence ordernpx eslint scripts/lib/instinct-relevance.js scripts/hooks/session-start.js tests/**— cleannpx markdownlint README.md— cleanvalidate-hooks.js,validate-no-personal-paths.js,check-unicode-safety.js— all passType of Change
Security & Quality Checklist
go!=good)Documentation
README.md— documentedECC_INSTINCT_RELEVANCE_RANKINGalongside the existing instinct knobsFixes #2371