Nobody should be on-call.
Polylane finds issues in your app and fixes them for you before they become incidents.
Get started for free
No credit card required.
Loading video…
Detect
Catch the issues nobody wrote a monitor for
Stop waiting for users to report bugs. Polylane watches your logs, metrics and traces across every service, learns what normal looks like, and raises an issue the moment something drifts.
Start detecting Topology Share
Search your cloud resources...
Filter Galaxy Flow Table
Critical Critical to your architecture.
Issue hotspot: 4 issues in the last 7 days
Change hotspot: 12 changes in the last 7 days
Cloudflare · coreplane · earth · Compute
Critical Critical to your architecture.
Change hotspot: 7 changes in the last 7 days
Cloudflare · coreplane · earth · Databases
Standard Important to your architecture.
Issue hotspot: 2 issues in the last 7 days
AWS · coreplane-prod · us-east-1 · Compute
Critical Critical to your architecture.
Issue hotspot: 1 issue in the last 7 days
Change hotspot: 3 changes in the last 7 days
PlanetScale · coreplane · us-east · Databases
Troubleshoot
Get the root cause before you open your laptop
Polylane investigates each issue across your code, your deploys and your telemetry, and shows you what broke, why, and the evidence behind it.
Start troubleshooting Threads Critical latency degradation in checkout-edge worker Share
What I did
I queued an autofix that restores the Hyperdrive pool hd-prod in the coreplane/checkout-edge repo with:
max_connectionsrestored from 5 to 50 on the pool binding- a regression test that fails when checkout queues on the pool
wrangler.jsoncconfigured to deploy ascheckout-edge
This gives Boris a PR with deployable code. Once merged, running wrangler deploy will replace the degraded worker with the fixed version.
The root cause report has been updated with the occurrence #2 evidence: Root Cause: checkout-edge Latency Degradation (Recurring).
Wall Time
checkout-edge worker · last hour
Wall Time P99
28,381ms
Wall Time P95
2,696ms
Wall Time P50
118ms
010k20k30k
18:0018:2519:00
Worked for 4m 12s
AI can make mistakes. Please double check cited sources.
Dig deeper...
Fix
Wake up to a fix ready to merge
Polylane writes the fix and opens a pull request. It goes through your normal review and CI, and nobody gets paged.
Start fixingRestore Hyperdrive pool size in checkout-edge
View PR description 
View on GitHub Dismiss Merge
Files changedInvestigation
Unified Side by side src/hyperdrive/pool.ts
@@ -14,21 +14,29 @@ export class HyperdrivePool {
14 14 #inFlight = 0;
15 15
16 - constructor(private readonly binding: Hyperdrive) {
17 - this.#ceiling = DEFAULT_CEILING;
16 + constructor(
17 + private readonly binding: Hyperdrive,
18 + private readonly ceiling = poolCeiling(binding),
19 + ) {
20 + if (this.ceiling < MIN_CEILING) {
21 + throw new PoolMisconfigured(this.ceiling, MIN_CEILING);
22 + }
18 23 }
19 24
20 25 async acquire(signal: AbortSignal): Promise<Connection> {
21 - while (this.#inFlight >= this.#ceiling) {
22 - await this.#queue.wait();
26 + if (this.#inFlight >= this.ceiling) {
27 + queueDepth.record(this.#queue.size, { binding: this.binding.id });
28 + if (this.#queue.size > this.ceiling * QUEUE_ALARM_RATIO) {
29 + logger.warn("hyperdrive pool saturated", {
30 + ceiling: this.ceiling,
31 + waiting: this.#queue.size,
32 + });
33 + }
34 + await this.#queue.wait(signal);
23 35 }
24 36 this.#inFlight += 1;
25 37 return this.binding.connect();
26 38 }
@@ -41,9 +53,13 @@ export class HyperdrivePool {
41 53 release(connection: Connection): void {
42 54 this.#inFlight -= 1;
43 - this.#queue.next();
55 + queueDepth.record(this.#queue.size, { binding: this.binding.id });
56 + this.#queue.next();
44 57 }
45 58
46 - get saturated(): boolean {
47 - return this.#inFlight >= this.#ceiling;
59 + get saturated(): boolean {
60 + return this.#inFlight >= this.ceiling;
48 61 }
49 62 }
@@ -58,14 +71,18 @@ function poolCeiling(binding: Hyperdrive) {
58 - const configured = binding.maxConnections;
59 - if (!configured) return DEFAULT_CEILING;
60 - return configured;
71 + const configured = binding.maxConnections ?? DEFAULT_CEILING;
72 + // The binding's ceiling is the contract: a worker that opens more
73 + // connections than the pool allows queues here instead of at the origin,
74 + // which is invisible in origin latency. Fail loudly at boot instead.
75 + return Math.max(configured, MIN_CEILING);
61 76 }
62 77
63 - export const DEFAULT_CEILING = 5;
64 - export const QUEUE_ALARM_RATIO = 4;
78 + export const DEFAULT_CEILING = 50;
79 + export const MIN_CEILING = 25;
80 + export const QUEUE_ALARM_RATIO = 0.5;
65 81
66 - const queueDepth = noopCounter();
67 - const logger = console;
82 + const queueDepth = meter.createHistogram("hyperdrive.queue_depth");
68 83
Ask about this pull request