Add Request Logging to MDEMG Service#9
Conversation
…ment var Added two new configuration options for request logging: - LOG_FORMAT: "text" (default) or "json" for structured logging - LOG_SKIP_HEALTH: boolean to skip logging /healthz and /readyz endpoints Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…rapper Add HTTP request logging middleware implementation: - responseWriter wrapper to capture status codes - Request ID generation using crypto/rand - LogConfig struct with Format and SkipHealth options - LoggingMiddleware function supporting text and JSON formats - Health endpoint exclusion option (/healthz, /readyz) - Duration timing with time.Since() Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Updated server.go Routes() method to wrap the ServeMux with LoggingMiddleware. Creates LogConfig from s.cfg.LogFormat and s.cfg.LogSkipHealth, then returns the wrapped handler instead of the bare mux. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…n tests Added comprehensive table-driven tests for middleware functionality: - TestIsHealthEndpoint: tests health endpoint detection - TestGenerateRequestID: tests unique request ID generation - TestResponseWriter_WriteHeader: tests status code capture - TestResponseWriter_Write: tests implicit 200 OK behavior - TestLoggingMiddleware: tests full middleware flow - TestLoggingMiddleware_RequestIDUniqueness: tests ID uniqueness - TestLoggingMiddleware_PassesRequestThrough: tests request passthrough - TestLogConfig: tests config struct - TestLogEntry: tests JSON marshaling Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fix pre-existing bug in cache_test.go where test expected incorrect LRU eviction behavior. The test creates a cache of size 2, and when text1 is re-added after being evicted, text2 becomes the LRU and is also evicted. The test incorrectly expected text2 to still be in cache. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughIntroduces HTTP request logging middleware for a Go service with configurable text/json formats and optional health endpoint filtering. Adds request ID generation, integrates middleware into the server router, and updates configuration to support logging parameters. Includes comprehensive test coverage and adjusts a cache test expectation. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant LoggingMiddleware
participant HTTPHandler
participant ResponseWriter
participant Logger
Client->>LoggingMiddleware: HTTP Request
LoggingMiddleware->>LoggingMiddleware: Generate Request ID
LoggingMiddleware->>LoggingMiddleware: Start Timer
LoggingMiddleware->>ResponseWriter: Wrap Original ResponseWriter
LoggingMiddleware->>HTTPHandler: Pass Request + Wrapped ResponseWriter
HTTPHandler->>ResponseWriter: Write Status Code
HTTPHandler->>ResponseWriter: Write Headers & Body
HTTPHandler-->>LoggingMiddleware: Handler Complete
LoggingMiddleware->>LoggingMiddleware: Calculate Duration
LoggingMiddleware->>Logger: Format Log Entry (Text/JSON)
LoggingMiddleware->>Logger: Write Log
LoggingMiddleware-->>Client: Response
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
- Update status: 14 PRs merged (#1-12) - Add PRs 9-12 to task table (Tasks 012-015) - Document new features: - Request Logging Middleware (PR #9) - Configurable Scoring Hyperparameters (PR #10) - Memory Stats Endpoint (PR #11) - Memory Archive/Delete Endpoints (PR #12) - Add new environment variables (SCORING_*, LOG_*) - Update Key Files Reference with new files - Mark Tasks 016, 017 as in-progress Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Gap Interview System (Task #8): - Add GapInterviewer for generating interview prompts from capability gaps - Create type-specific prompts for data_source, reasoning, query_pattern gaps - Add RunWeeklyInterview() APE job with configurable scheduling - Add API endpoints: GET/POST /v1/system/gap-interviews - Add prompt answer/skip tracking with Neo4j persistence - Add V0010 migration for InterviewPrompt schema - Wire StartWeeklyGapInterviews() background job into server CMS Integration Tests (Task #9): - Add integration_test.go with Neo4j test fixtures - Test visibility filtering (private/team/global) - Test Context Cooler graduation and decay - Test Jiminy rationale generation - Test REFERS_TO cross-module linking - Test surprise detection for corrections - Test end-to-end conversation flow Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Close three feedback loop gaps in the Jiminy guidance system to create a self-improving guidance pipeline where chronically ignored guidance decays automatically and consistently followed guidance strengthens. Gap 1 — Dispatcher gap (3 new executors): - review_guidance_effectiveness: diagnostic executor that queries per-constraint effectiveness and categorizes items as high/low/insufficient - adjust_guidance_confidence: mutating executor that boosts items with effectiveness >= 0.7, decays items < 0.1 (min 5 surfaces), archives stale - archive_ineffective_constraints: delegates to ArchiveStaleConstraints Gap 2 — Confidence scope gap: - ConfidenceUpdater now fires for ALL guidance types (constraints, corrections, patterns, decisions), not just GuidanceConstraint - Archive guard added: only constraint nodes get archived when confidence drops below threshold — corrections/learnings/patterns retain status Gap 3 — SignalLearner isolation: - Jiminy Guide() now records Hebbian emissions for each surfaced item - RecordOutcome() records responses for followed/partial-compliance outcomes - SignalLearnerProvider interface avoids circular import between jiminy/ape New reflection pattern #15 (guidance_confidence_drift): - Triggers adjust_guidance_confidence when GuidanceHealth is between 0 and 0.7 - Complements pattern #9 (diagnostic at 0.5) with corrective action at 0.7 Architecture: - GuidanceCalibrationProvider interface in ape/types_rsic.go - rsicGuidanceCalibrationAdapter in api/rsic_adapters.go bridges ape→jiminy - 5 new RSIC_GUIDANCE_* config variables with sensible defaults - Wired in server.go: SetGuidanceCalibrator + SetSignalLearner Tests: 10 new tests across 4 test files Docs: feature doc, UATS spec, cli-reference, .env.example, AGENT_HANDOFF Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lag-off) + Phase 13 Epic 6 V0017 audit-writer fix + Phase 11+ feature-doc backfill (narrow close)
Narrow close per operator approval after Epic 0+1+2 produced design questions
that warrant dedicated follow-up sprints. Note 05 deferred to Phase 14.2;
Note 06 default flip deferred to Phase 14.1.
What landed
-----------
* Phase 13 Epic 6 V0017 audit-writer fix (in-flight discovery)
- tsdb/retrieval_audit_writer.go (new, ~165 LOC; buffered + 30s flush via CopyFrom)
- retrievalAuditAdapter in api/server.go (cycle-safe translation)
- V0017 was empty since Phase 13 because SetRetrievalAuditWriter had no
callers; now writes per retrieve when RETRIEVAL_AUDIT_ENABLED=true.
- Live verification: 279 audit rows accumulated in 4h since fix landed.
* Note 06 sparse activation gate (flag-off)
- retrieval/gate.go (~190 LOC) + 9 Tier 1 unit tests, all green
- Wired post-aggregation, pre-rerank in service.go
- 4 config knobs (SPARSE_*); default off, percentile 0.95, min 3, max 20
- Per-request override via ?sparse=true|false and ?sparse_percentile=N
- debug.sparse_gate_* + debug.below_threshold_* (when JiminyEnabled)
- 3 Prometheus histograms
* TSDB V0019 sparse_gate_metrics
- migrations/019_sparse_gate_metrics.sql (hypertable, 7-day chunks)
- tsdb/sparse_gate_writer.go (~165 LOC)
- sparseGateRecorderAdapter in api/server.go (always wired so per-request
overrides record even when default off)
- TSDB_REQUIRED_SCHEMA_VERSION 18 -> 19
* Epic 0 forensic doc — phase_14_score_distribution_analysis.md
- Defaults derived from llm_interactions.retrieval_scores (99k+50k score
points across consulting.classify + retrieval.rerank_cross)
- Heavy-tail confirmed (p98/p50 ~ 4-5x); within-call clamp dominates
percentile choice in dominant K=20-50 regime
- Note 05 catalog redesign needed for whk-wms (0 distinct symbols, 0
distinct roles) — flagged for Phase 14.2
* A/B verdicts captured
- 16q quick at MIN=3 / p95,p98,p99: all FAIL (q69 boundary)
- 16q quick at MIN=10 / p95: PASS (mean +0.019, 0 regressions, 3 improvements)
- 120q full at MIN=10 / p95: FAIL per-question (mean parity 0.413=0.413,
7 boundary regressions across 4 categories, 3 of 7 in
architecture_structure)
- Per sprint plan §10 risk #1: ship flag-off; Phase 14.1 will retune.
* Phase 11+ feature-doc backfill (operator request 2026-05-04)
- new: docs/features/{mlx-watchdog,uvts-validation,column-voting-retrieval,
local-llm-runtime,sparse-retrieval}.md
- extended: docs/features/service-resilience.md (Phase 11.6.x additions)
- Standing rule saved as memory feedback_per_feature_docs_required.md
* Follow-up sprint stubs scoped
- sprint_plan_phase_14_1_adaptive_per_category_gate.md (~3 days, ~$15)
- sprint_plan_phase_14_2_note_05_sparse_fingerprints.md (~7 days, ~$25)
Decision-fork outcomes
----------------------
| Fork | Provisional | Outcome |
|---|---|---|
| #2 percentile default | 0.98 | 0.95 (Epic 0 data) |
| #5 catalog bit policy | static 64/64/64/64 | adaptive (deferred Phase 14.2) |
| #8 gate ordering | pre-rerank | pre-rerank (confirmed) |
| #9 default flip | per-Note conditional | flag-off (Phase 14.1 will flip) |
OpenAI spend (actual): ~$13. Well under sprint $25-50 budget.
Tests + lint
------------
* go test -race ./internal/{retrieval,config,metrics,tsdb}: all green
* golangci-lint run on affected packages: 0 issues
* Live smoke: /healthz green, retrieve returns 20 (gate off), 279 V0017
audit rows in 4h (Phase 13 Epic 6 fix verified in production)
Memory observations
-------------------
* rw0mzergwcqct8abpw0dli9x — Phase 14 Epic 8 doc-backfill scope
* sc4iwy3of9ndn5kowja1i14i — Epic 0 forensic + audit-writer gap
* omr2rs5jppqrvee2k0l1xtd1 — Epic 1 gate code complete
* re4k7rpd3hjt5a52l8qwx8fp — Epic 2 verdict + Phase 14.1 scope
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This feature adds HTTP request logging middleware to the MDEMG Go service. Currently, the service uses basic
log.Printfstatements for startup messages and warnings but lacks comprehensive request-level logging for observability. Request logging will capture method, path, status code, duration, and optional request IDs for each HTTP request, enabling debugging, performance monitoring, and audit capabilities.Summary by CodeRabbit
Release Notes
New Features
LOG_FORMATandLOG_SKIP_HEALTH.Tests
✏️ Tip: You can customize this high-level summary in your review settings.