TT-18010: share MCP protocol context and analytics O#6 by andrei-tyk · Pull Request #8678 · TykTechnologies/tyk

andrei-tyk · 2026-09-06T20:40:21Z

Parses the bounded request envelope once and restores the body with its original closer. Shared typed protocol context retains raw declarations, effective version, detection source and validation outcome for policy, filtering, errors and analytics. Adds effective_protocol_version, declared_protocol_version and protocol_version_source through Gateway, Pump and Dashboard. Session IDs do not imply negotiation; legacy initialize and the 2025-03-26 fallback remain supported. Discovery/subscription classifications add no subscription runtime.

Removed the dedicated MCP v2 workflow and the added MCP review documents from the affected PR branches. Dashboard pins the corresponding cleaned Gateway commits. Verified all 15 incremental PR diffs, branch ancestry and unchanged runtime/test sources; Go module tidying changed only the Gateway pin/checksums. The full-suite results below refer to the explicitly recorded pre-cleanup commits; those suites were not rerun for this file cleanup.

Based on Gateway #8677. Counterpart: Dashboard PR. Prerequisite: Pump #1098. Fixtures: cache #6.

Review head: 1134ce76e08f47fb4ce5962a1f0b2ea00e1022e2. Matching Gateway: 1134ce76e08f47fb4ce5962a1f0b2ea00e1022e2. Pump pin: 977a5ddcda4a523337b4d8e8dd36a0df03c81aef.

Module versions: github.com/modelcontextprotocol/go-sdk v1.6.0, github.com/TykTechnologies/tyk-pump v1.17.0-rc1.0.20260906204536-977a5ddcda4a.

Incremental integration: 6 passed, 0 failed, 0 skipped using python scripts/run_mcp_v2.py tests/mcp/mcp_ga_protocol_analytics_test.py --junitxml=reports/TT-18010.xml from Dashboard tests/api. The runner uses pytest_local.ini, explicit environment overrides and -n 0; missing ticket fixtures fail setup. Affected Go packages were run at each layer with GOWORK=off and -count=1.

Incremental affected-Go result: 596 passing test/subtest events, 0 failures, 0 skips.

This Gateway layer’s full CI reports: 7470 functional, 2561 shared-race and 67 scoped-race passing test/subtest events; zero failures. Existing skips: 75 functional, 2 shared-race, 0 scoped-race.

Historical validation before the workflow/document cleanup used committed pins: Dashboard 5176927f20055da63c601d951d8561cb6e58f1b5, Gateway 47b7aae7b3c679b3b28bacb4f5b3b2686481ed6c, Pump 977a5ddcda4a523337b4d8e8dd36a0df03c81aef, mock 97e603291b7c0e1bc80c5b9effec947f05d439b1. No local module replacements are used.

Full serial Python MCP suite: 495 passed, 0 failed, 2 existing skips both locally (1,326.29 seconds) and in CI (1,372.38 seconds). All 85 new ticket cases pass without skips. The two existing skips are empty OAuth PRM placeholders (test_resource_metadata_on_scope_check_challenge, test_no_resource_metadata_when_prm_disabled). The existing analytics usage tests now select their own API records; the complete suite exercises that correction.
Gateway CI: 7,511 passing test/subtest events, 0 failures, 75 existing skips. Command: GOWORK=off CI=true go test -json -p 1 -parallel 1 -cover -tags 'goplugin dev' -timeout 15m -count=1 ./gateway ./internal/mcp/... ./internal/jsonrpc/... ./internal/httpctx ./internal/httputil ./apidef/... ./config ./user. Build the matching plugin first with go build -cover -tags 'goplugin dev' -buildmode=plugin -o test/goplugins/goplugins.so ./test/goplugins.
Shared-package race checks: 2,587 passing test/subtest events, 0 failures, 2 existing skips using GOWORK=off go test -json -p 1 -race -count=1 ./internal/mcp/... ./internal/jsonrpc/... ./internal/httpctx ./internal/httputil ./apidef/... ./config ./user. The two skips are existing OAS filled-golden cases. Scoped Gateway MCP/SSE/request-state race checks: 80 passing test/subtest events, 0 failures/skips (exact selection in the historical validation run). Additional local ./ctx race coverage passes (10 test/subtest events). Production and development Gateway builds pass without an SDK upgrade. Go counts include subtests; existing CI-gated/flaky/placeholder skips are retained, not counted as passes.
Pump compatibility: 153 passing test/subtest events, zero failures/skips across changed serializers and storage mappings. Independent mock full race suite: 16 passing events, zero failures/skips. Dashboard Origin/configuration and protocol JSON compatibility Go checks pass (5 events).

Reproduce the four-repository run with python tests/api/scripts/run_mcp_v2_stack.py --gateway-root /path/to/gateway --pump-root /path/to/pump --mock-root /path/to/mock --artifacts /path/to/reports --license-file /path/to/existing/license from Dashboard; use the supporting-service/environment configuration in tests/api/MCP_V2.md on the final Dashboard layer. The license is supplied only at runtime.

Durable evidence: Gateway validation run, artifact mcp-v2-gateway-validation; four-repository integration run, artifact mcp-v2-integration. These retain JUnit, raw test events, logs and exact source/module records for 30 days. Integration CI status: all build/test/report stages passed; 495 active MCP tests passed.

Historical CI findings were investigated individually. Dependency guards pass. Stock draft/stack aggregators report failure because their test/build jobs are skipped by existing workflow rules. Pump unit tests and lint pass; its managed drift check flags unchanged .github/workflows/zizmor.yml. No managed-workflow edit or override label was applied. Automated Visor findings remain on earlier layers: response/log parsing optimization suggestions and localhost fixture warnings; these are recorded as open review findings, not misreported as successful checks. Dashboard's branch-suggestion npm installation failure passed on rerun.

The broader full-package race experiment exposed pre-existing Streams/RPC/shutdown global-state races; these reproduced on the fetched Gateway base. Other unchanged versioning/policy race paths were reported but did not reproduce in that single baseline retry. The original failed run remains available. Required changed-state/SSE race coverage is isolated and passes; this does not claim the whole legacy Gateway package is race-free. An unchanged JWT global-log test also intermittently captured an unrelated uptime-shutdown message in the error-layer run; the retry passed all functional/race/build stages, and all six Gateway layer workflows passed before their removal.

Gateway’s embedded MCP SDK remains v1.6.0; the independently built mock uses v1.7.0. All builds use GOWORK=off and published modules have exact commit-based pins with no local filesystem replacements. The parent epic retains the embedded SDK upgrade, modern synthetic runtime, source-header projection and end-to-end cancellation. OAuth issuer changes and stream idle deadlines are outside this stack. A released Pump dependency is required before production release readiness; no release tag was created.

Ticket: https://tyktech.atlassian.net/browse/TT-18010

Ticket Details

TT-18010


Status	Ready for Testing
Summary	Detect and record protocol version

Generated at: 2026-09-17 08:38:42

probelabs · 2026-09-06T20:42:13Z

This pull request refactors the Model Context Protocol (MCP) request handling in the Tyk Gateway to improve performance and enrich analytics. The core change is to parse the MCP request body only once at the ingress point and share the result—including the parsed envelope, raw body, and protocol version details—via a new request-scoped ProtocolContext. This eliminates redundant parsing in downstream middleware.

Additionally, the PR enhances observability by adding effective_protocol_version, declared_protocol_version, and protocol_version_source to MCP analytics records. A new "completion analytics" mechanism is introduced for paired MCP proxies (where a public API proxies to an internal one). This system uses httpsnoop to observe the internal response without buffering, ensuring that a complete and accurate analytics record is generated for the original public-facing request upon its completion.

Files Changed Analysis

The changes are concentrated in the gateway package, with new supporting code in internal/mcp and internal/httpctx.

New Context & Model (internal/mcp/protocol_context.go, internal/httpctx/mcp.go, ctx/ctx.go): These files introduce the ProtocolContext struct, which acts as the single source of truth for a parsed MCP request. They also define the context key and accessors to manage this object throughout the request lifecycle.
Ingress Logic Refactor (gateway/mw_jsonrpc.go): The JSONRPCMiddleware is significantly refactored to be the sole owner of the initial MCP request parsing. It now performs a size-limited read, populates the ProtocolContext, and restores the request body with its original io.Closer to ensure it remains readable by downstream components.
Completion Analytics (gateway/mcp_completion_analytics.go, gateway/api_loader.go): This new system handles analytics for paired MCP proxies. mcp_completion_analytics.go contains the httpsnoop-based observer logic, while api_loader.go integrates it into the DummyProxyHandler to activate it for the specific paired proxy scenario.
Analytics Enrichment (gateway/handler_success.go): The success handler is updated to consume the shared ProtocolContext and populate the new protocol version fields in the final analytics record.
Middleware Simplification (gateway/mw_jsonrpc_rest_as_mcp_policy.go): This middleware is streamlined by removing its own parsing logic; it now consumes the shared ProtocolContext.
Dependency Management (go.mod, go.sum): The tyk-pump dependency is updated to a version that can process the new analytics fields, and felixge/httpsnoop is added as a new dependency.

Architecture & Impact Assessment

What this PR accomplishes: It enhances MCP performance by eliminating redundant request body parsing, enriches observability with detailed protocol version analytics, and ensures correct analytics attribution for complex paired MCP proxy setups.
Key technical changes introduced:
1. A new request-scoped mcp.ProtocolContext is established as the single source of truth for parsed MCP requests.
2. The JSONRPCMiddleware is refactored to own the initial request body read and context creation, centralizing the process.
3. A response-snooping pattern (mcpCompletionObserver) is introduced using httpsnoop to trigger analytics for a parent proxy based on the completion of an internal, proxied request, without delaying the response to the client.
Affected system components: The changes directly affect the Tyk Gateway's MCP request pipeline. There is a critical downstream dependency on Tyk Pump, which must be updated to a compatible version to handle the new analytics fields. Systems that consume this data, such as the Tyk Dashboard, will also be affected, requiring coordinated releases.

Paired MCP Proxy Analytics Flow

sequenceDiagram
    participant Client
    participant PublicProxy as Public MCP Proxy
    participant Observer as Completion Observer
    participant InternalAdapter as Internal Adapter
    participant Analytics

    Client->>PublicProxy: POST /public-mcp/
    activate PublicProxy
    PublicProxy->>Observer: newMCPCompletionAnalyticsOwner(w)
    note right of PublicProxy: Wraps ResponseWriter with snoop
    PublicProxy->>InternalAdapter: Proxies request internally
    activate InternalAdapter
    InternalAdapter-->>PublicProxy: Writes response body
    deactivate InternalAdapter
    Observer-->>Observer: Captures status & body snippet
    PublicProxy-->>Client: Streams response immediately
    deactivate PublicProxy
    Observer->>Observer: owner.Complete() called via defer
    Observer->>Analytics: Records hit for PublicProxy

Scope Discovery & Context Expansion

The impact of this PR extends beyond the Gateway repository. The explicit update of the tyk-pump dependency and the mention of a counterpart Dashboard PR confirm this is a cross-stack feature that establishes a new data contract between the Gateway and the analytics pipeline. This necessitates synchronized deployments of Gateway, Pump, and Dashboard to function correctly.

The base branch, TT-18030-v2-mcp-origin-validation, indicates these changes are part of a larger initiative to enhance MCP v2, building upon a foundation of security and validation by adding a crucial layer of protocol interpretation and observability.

Metadata

Review Effort: 4 / 5
Primary Label: feature

Powered by Visor from Probelabs

Last updated: 2026-09-16T19:08:37.326Z | Triggered by: pr_updated | Commit: 11fde7c

💡 TIP: You can chat with Visor using /visor ask <your question>

probelabs · 2026-09-06T20:42:46Z

✅ Security Check Passed

No security issues found – changes LGTM.

✅ Architecture Check Passed

No architecture issues found – changes LGTM.

✅ Security Check Passed

No security issues found – changes LGTM.

\n\n

✅ Architecture Check Passed

No architecture issues found – changes LGTM.

\n\n

Performance Issues (1)

Severity	Location	Issue
🟡 Warning	`internal/mcp/protocol_context.go:40`	The `NewProtocolContext` function creates a copy of the `rawBody` byte slice. This slice is passed from `gateway.JSONRPCMiddleware`, where it is read from the request and not modified thereafter. This copy results in an unnecessary memory allocation for every MCP request, which can be up to 1MB (`mcpIngressReadLimit`). For paired MCP requests, this is one of three potential copies of the request body, contributing to memory pressure. 💡 Suggestion Remove the redundant copy and store the `rawBody` slice by reference. The data in the slice is effectively immutable at this stage of the request lifecycle, so sharing the reference is safe and avoids the allocation. 🔧 Suggested Fix `RawBody: rawBody,`

Quality Issues (3)

Severity	Location	Issue
🟡 Warning	`gateway/mcp_completion_analytics_test.go:153-154`	The test `TestSuccessHandlerObservesNativeMCPErrorCode` uses hard-coded numeric literals for `int64` boundaries (`-9223372036854775808` and `9223372036854775807`). While the test name provides context, using constants like `math.MinInt64` and `math.MaxInt64` from the `math` package would improve readability and maintainability. 💡 Suggestion Replace the hard-coded integer boundary values with their corresponding constants from the `math` package to make the intent clearer and the code more robust against typos. The values need to be embedded in the JSON string, so using `fmt.Sprintf` or string concatenation is necessary.
🟡 Warning	`gateway/mcp_completion_analytics_test.go:210-212`	The test `TestJSONRPCCompletionErrorCode` uses hard-coded numeric literals (`2147483648`, `-9223372036854775808`, `-2147483649`) to represent values just outside `int32` boundaries and at the `int64` boundary. These should be replaced with expressions using constants from the `math` package (e.g., `math.MaxInt32 + 1`, `math.MinInt64`, `math.MinInt32 - 1`) to improve clarity and explicitly state the testing intent. 💡 Suggestion Define the expected codes using constants from the `math` package to make the boundary conditions being tested more explicit. For example, use `code: math.MaxInt32 + 1` instead of `code: 2147483648`.
🟡 Warning	`gateway/mcp_completion_analytics_test.go:392`	The concurrency test `TestDummyProxyHandlerKeepsConcurrentMCPCompletionAttributionRequestLocal` uses the hard-coded numeric literal `2147483648` as a base for generating unique error codes. This value, which is `math.MaxInt32 + 1`, should be defined as a constant to improve readability and make the code's intent clearer. 💡 Suggestion Introduce a named constant for the base error code, such as `const baseErrorCode = int64(math.MaxInt32) + 1`. Use this constant in both the request generation (`fmt.Sprintf("%d", baseErrorCode+int64(i))`) and the assertion (`assert.Equal(t, baseErrorCode+int64(i), ...)`).

Powered by Visor from Probelabs

Last updated: 2026-09-16T19:08:15.162Z | Triggered by: pr_updated | Commit: 11fde7c

💡 TIP: You can chat with Visor using /visor ask <your question>

github-actions · 2026-09-13T09:29:38Z

🎯 Recommended Merge Targets

Based on JIRA ticket TT-18010: Detect and record protocol version

Fix Version: Tyk 5.16.0

⚠️ Warning: Expected release branches not found in repository

Required:

master - No matching release branches found. Fix will be included in future releases.

📋 Workflow

Merge this PR to master first

edsonmichaque

LGTM

sentinelone-cnapp-eu1 · 2026-09-14T14:00:06Z