Skip to content

ER/T terminator create reports "invalid edge router for session" with no router-side error and slow recovery #4193

Description

@plorenz

When an edge router tunneler gains bind access to a service before the edge router policy and
service edge router policy that also govern hosting are in place, the terminator create is
rejected and the failure is close to undiagnosable.

Symptoms

The controller logs:

{"error":"invalid edge router for session","file":".../create_tunnel_terminator_v2.go:86",
 "level":"error","msg":"responded with error","routerId":"...","terminatorId":"..."}

No terminator appears for the service, and the router logs nothing at all.

Problems

  1. The error message is wrong. There is no session in the ER/T hosting path; it was removed
    when ER/T was migrated to the router data model. InvalidEdgeRouterForSessionError is reused
    from the session-based path, so the message misdirects anyone diagnosing it toward session
    state. The same message is used on the sessionless create_circuit_v3 path.

  2. The error is not actionable. IsAccessToEdgeRouterAllowed checks two independent policy
    links (identity to edge router via edge router policy, service to edge router via service
    edge router policy) and collapses them to a single bool, so the denial can't say which
    policy is missing.

  3. The router logs nothing. HandleCreateTerminatorResponse discards response.Msg and
    response.ErrorCode on failure. From the router's side hosting silently doesn't work, and
    the only trace anywhere is the single controller log line.

  4. Recovery is slow. On a non-busy failure the terminator stays in Establishing and is not
    requeued, so the next attempt waits on the retry scan, which requires lastAttempt older
    than 2 minutes against a 1 minute ticker. Config applied in quick succession (a script or a
    paste of several ziti edge create commands) can lose the race and then appear broken for
    2-3 minutes.

Reproduction

Create a service, add a bind service policy, then grant the router the hosting role attribute
before creating the edge router policy and service edge router policy. The router begins
hosting as soon as the bind policy grants access and immediately fires the create.

Notes

Requiring the service edge router policy on the hosting side is intentional. The underlying
gap is that the ER/T has no view of edge router or service edge router policies, so it can't
know it is not yet in a valid hosting situation. That is expected to be resolved separately by
bringing the data into the router data model. This issue covers making the failure diagnosable
and narrowing the race in the meantime.

Also found

Controller error codes never reached SDK clients on the dial paths. The router read only the
message body from the controller's error reply and dropped ErrorCodeHeader, and
sendStateClosedReply looked for the code on the SDK's request rather than the controller's
response. As a result edge.ConnRefusalError always classified dial refusals as
CauseUnknown, making the access-denied, invalid-session, and invalid-service classifications
unreachable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions