When an edge router tunneler gains bind access to a service before the edge router policy and
service edge router policy that also govern hosting are in place, the terminator create is
rejected and the failure is close to undiagnosable.
Symptoms
The controller logs:
{"error":"invalid edge router for session","file":".../create_tunnel_terminator_v2.go:86",
"level":"error","msg":"responded with error","routerId":"...","terminatorId":"..."}
No terminator appears for the service, and the router logs nothing at all.
Problems
-
The error message is wrong. There is no session in the ER/T hosting path; it was removed
when ER/T was migrated to the router data model. InvalidEdgeRouterForSessionError is reused
from the session-based path, so the message misdirects anyone diagnosing it toward session
state. The same message is used on the sessionless create_circuit_v3 path.
-
The error is not actionable. IsAccessToEdgeRouterAllowed checks two independent policy
links (identity to edge router via edge router policy, service to edge router via service
edge router policy) and collapses them to a single bool, so the denial can't say which
policy is missing.
-
The router logs nothing. HandleCreateTerminatorResponse discards response.Msg and
response.ErrorCode on failure. From the router's side hosting silently doesn't work, and
the only trace anywhere is the single controller log line.
-
Recovery is slow. On a non-busy failure the terminator stays in Establishing and is not
requeued, so the next attempt waits on the retry scan, which requires lastAttempt older
than 2 minutes against a 1 minute ticker. Config applied in quick succession (a script or a
paste of several ziti edge create commands) can lose the race and then appear broken for
2-3 minutes.
Reproduction
Create a service, add a bind service policy, then grant the router the hosting role attribute
before creating the edge router policy and service edge router policy. The router begins
hosting as soon as the bind policy grants access and immediately fires the create.
Notes
Requiring the service edge router policy on the hosting side is intentional. The underlying
gap is that the ER/T has no view of edge router or service edge router policies, so it can't
know it is not yet in a valid hosting situation. That is expected to be resolved separately by
bringing the data into the router data model. This issue covers making the failure diagnosable
and narrowing the race in the meantime.
Also found
Controller error codes never reached SDK clients on the dial paths. The router read only the
message body from the controller's error reply and dropped ErrorCodeHeader, and
sendStateClosedReply looked for the code on the SDK's request rather than the controller's
response. As a result edge.ConnRefusalError always classified dial refusals as
CauseUnknown, making the access-denied, invalid-session, and invalid-service classifications
unreachable.
When an edge router tunneler gains bind access to a service before the edge router policy and
service edge router policy that also govern hosting are in place, the terminator create is
rejected and the failure is close to undiagnosable.
Symptoms
The controller logs:
No terminator appears for the service, and the router logs nothing at all.
Problems
The error message is wrong. There is no session in the ER/T hosting path; it was removed
when ER/T was migrated to the router data model.
InvalidEdgeRouterForSessionErroris reusedfrom the session-based path, so the message misdirects anyone diagnosing it toward session
state. The same message is used on the sessionless
create_circuit_v3path.The error is not actionable.
IsAccessToEdgeRouterAllowedchecks two independent policylinks (identity to edge router via edge router policy, service to edge router via service
edge router policy) and collapses them to a single bool, so the denial can't say which
policy is missing.
The router logs nothing.
HandleCreateTerminatorResponsediscardsresponse.Msgandresponse.ErrorCodeon failure. From the router's side hosting silently doesn't work, andthe only trace anywhere is the single controller log line.
Recovery is slow. On a non-busy failure the terminator stays in
Establishingand is notrequeued, so the next attempt waits on the retry scan, which requires
lastAttemptolderthan 2 minutes against a 1 minute ticker. Config applied in quick succession (a script or a
paste of several
ziti edge createcommands) can lose the race and then appear broken for2-3 minutes.
Reproduction
Create a service, add a bind service policy, then grant the router the hosting role attribute
before creating the edge router policy and service edge router policy. The router begins
hosting as soon as the bind policy grants access and immediately fires the create.
Notes
Requiring the service edge router policy on the hosting side is intentional. The underlying
gap is that the ER/T has no view of edge router or service edge router policies, so it can't
know it is not yet in a valid hosting situation. That is expected to be resolved separately by
bringing the data into the router data model. This issue covers making the failure diagnosable
and narrowing the race in the meantime.
Also found
Controller error codes never reached SDK clients on the dial paths. The router read only the
message body from the controller's error reply and dropped
ErrorCodeHeader, andsendStateClosedReplylooked for the code on the SDK's request rather than the controller'sresponse. As a result
edge.ConnRefusalErroralways classified dial refusals asCauseUnknown, making the access-denied, invalid-session, and invalid-service classificationsunreachable.