Skip to content

双向流,连接不存在,但 err = stream.Send 返回不了错误,err 为nil,检测不到连接断了 #1865

Description

@fashionwu

Describe the bug

使用kitex grpc 双向流,连接已经通过netstat 确认不存在,但 err = stream.Send 返回不了错误,err 为nil,检测不到连接断了。

To Reproduce

Steps to reproduce the behavior:
很难复现,偶尔会出现。

Expected behavior

当双向流的连接断了后,预期err = stream.Send() , err 应该不为nil

**Kitex version: **

v0.14.1

Activity

  1. DMwangnima commented on Sep 29, 2025

    @DMwangnima
    Contributor

    此时 Recv 有报错么?具体报错是什么呢?

  2. fashionwu commented on Oct 11, 2025

    @fashionwu
    Author

    因为send 没有发出去,所有recv,一直没有报错。

  3. fashionwu commented on Oct 11, 2025

    @fashionwu
    Author
    //在 Send() 前做健康检查
    if err = stream.Context().Err(); err != nil {
    }
    

    不知道这样,可以不可以解决这个问题

  4. DMwangnima commented on Oct 13, 2025

    @DMwangnima
    Contributor

    这样不能解决问题,如果真的是连接断了但 Kitex 没有感知到,那么这个 Stream 在 Kitex 视角也是没有结束的。
    通信链路上,除了 Kitex-Client 与 Kitex-Server,是否还有其它组件?(例如 envoy)

  5. fashionwu commented on Oct 13, 2025

    @fashionwu
    Author

    没有其他组件,那该怎么解决呢?我这边使用kitex 双向流。send 和recv 是分别在两个不同的协程里面操作的

  6. DMwangnima commented on Oct 14, 2025

    @DMwangnima
    Contributor

    辛苦提供一个最小的可复现 client/server?我感觉大概率是代码问题。另外是在 linux 还是 windows 环境跑的?

  7. fashionwu commented on Oct 14, 2025

    @fashionwu
    Author

    很难复现,在linux 环境跑的

  8. DMwangnima commented on Oct 28, 2025

    @DMwangnima
    Contributor

    贴下代码吧,不清楚所谓的连接断了是什么意思,对端 Recv 有报错么?

  9. fashionwu commented on Dec 3, 2025

    @fashionwu
    Author

    就是对端服务进程挂掉了或不存在了,client 和server 的tcp 连接都不存在了,err = stream.Send 返回不了错误,err 为nil,检测不到连接断了

  10. DMwangnima commented on Dec 17, 2025

    @DMwangnima
    Contributor

    我猜测 stream.Send 发生在 tcp 连接断开前,使得这次 stream.Send 成功了(本质只是写入缓冲区)。如果一直调用 stream.Send 直到报错,肯定是能感知到 tcp 连接断开的。

  11. fashionwu commented on Mar 13, 2026

    @fashionwu
    Author

    我现在还遇到类似的情况。就是连接还在,但是之前都是空闲的,只有keep alive 的心跳包。突然有流量后stream.Send 不会发出网络包,最后会卡死在如下堆栈:
    goroutine 770 [select, 605 minutes]:
    github.com/cloudwego/kitex/pkg/remote/trans/nphttp2/grpc.(*writeQuota).get(0xc001e91420, 0x8e)
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/pkg/remote/trans/nphttp2/grpc/flowcontrol.go:61 +0x112
    github.com/cloudwego/kitex/pkg/remote/trans/nphttp2/grpc.(*http2Client).Write(0xc001e18140, 0xc001e18280, {0xc020baf440, 0x5, 0x10}, {0xc01b973200, 0x89, 0x89}, 0xc020baf42c)
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/pkg/remote/trans/nphttp2/grpc/http2_client.go:752 +0x2c5
    github.com/cloudwego/kitex/pkg/remote/trans/nphttp2.(*clientConn).WriteFrame(0xc017a42600, {0xc020baf440, 0x5, 0x10}, {0xc01b973200, 0x89, 0x89})
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/pkg/remote/trans/nphttp2/client_conn.go:155 +0xfb
    github.com/cloudwego/kitex/pkg/remote/trans/nphttp2.(*buffer).Flush(0xc01d0d4540)
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/pkg/remote/trans/nphttp2/buffer.go:100 +0xaa
    github.com/cloudwego/kitex/pkg/remote/trans/nphttp2.(*cliTransHandler).Write(0xc017f681f8, {0x35e52e0, 0xc0102e86f0}, {0x35efd50, 0xc017a42600}, {0x35fa520, 0xc010131a70})
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/pkg/remote/trans/nphttp2/client_handler.go:62 +0x234
    github.com/cloudwego/kitex/pkg/remote/trans/nphttp2.(*clientStream).SendMsg(0xc017e525a0, {0x35e52e0, 0xc0102e86f0}, {0x3158220, 0xc011ad0f80})
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/pkg/remote/trans/nphttp2/stream.go:261 +0x31d
    github.com/cloudwego/kitex/pkg/remote/trans/nphttp2.(*grpcClientStream).SendMsg(0xc01cd1c5c0, {0x3158220, 0xc011ad0f80})
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/pkg/remote/trans/nphttp2/stream.go:215 +0x69
    github.com/cloudwego/kitex/internal/stream.(*StreamingConfig).BuildSendInvokeChain.func1({0x35f19b0, 0xc01cd1c5c0}, {0x3158220, 0xc011ad0f80})
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/internal/stream/stream_option.go:64 +0x44
    github.com/cloudwego/kitex/client.(*grpcStream).SendMsg(0xc01871aa80, {0x3158220, 0xc011ad0f80})
    /go/pkg/mod/github.com/cloudwego/kitex@v0.14.1/client/stream.go:346 +0x65

    请问这个该如何处理?

  12. fashionwu commented on Mar 20, 2026

    @fashionwu
    Author

    出现上面的情况,是对端虽然链接还在,但接收协程退出了。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions