Looking at this a bit more now. Here's an example from today, two things happened:

websocket request to /xrpc/com.atproto.sync.subscribeRepos that succeeded, served for 1h+10s, then returned HTTP 101 continue
another websocket request to /xrpc/com.atproto.sync.subscribeRepos that did nothing, hung for 1h, then returned HTTP 504

After that, the Bluesky relay kept making subscribeRepos requests that did the same thing as in snarfed/arroba#2.

I suspect we're not handling something in the initial HTTP 101 code path, and that's killing the serving thread or something similar.

2024-03-06 06:51:48.496 - 07:51:58.540
GET 101 /xrpc/com.atproto.sync.subscribeRepos?cursor=1864

New websocket client for com.atproto.sync.subscribeRepos
com.atproto.sync.subscribeRepos: {'cursor': 1864} None
Running method
subscribeRepos: fetching existing commits from seq 1864
Sending to com.atproto.sync.subscribeRepos websocket client: {'op': 1, 't': '#commit'} {'repo': 'did:plc:sp6aj7mvt7dw26xp6wbvbv6i', 'ops': [{'action': 'update', 'path': 'app.bsky.actor.profile/self', 'cid': CID('base32', 1, 'dag-cbor', '12202e0b1c21d9f0ebeffda410546889e3f4d22d91d2baef7996c9e920cb4c1f2340')}], 'commit': CID('base32', 1, 'dag-cbor', '1220b1ccee20ec7e5ef7aa55752834791e7214bdc7e9ed91ea26351bf72700fb6188'), 'blocks': b":\xa2eroots\x81\xd8*X%\x00\x01q\x12 \xb1\xcc\xee \xec~^\xf7\xaaUu(4y\x1er\x14\xbd\xc7\xe9\xed\x91\xea&5\x1b\xf7'\x00\xfba\x88gversion\x01\x88\x02\x01q\x...
subscribeRepos: serving new commits

2024-03-06 07:51:58.945 - 08:51:58.946
GET 504 /xrpc/com.atproto.sync.subscribeRepos?cursor=1864

Event subscription handler hangs after a while #8

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions