Repository navigation
Add the GCP management network for Postgres - #6623
Open
furkansahin wants to merge 5 commits into
Open
furkansahin wants to merge 5 commits into
furkansahin wants to merge 5 commits into
Conversation
furkansahin
force-pushed
the
gcp-postgres-mgmt-network
branch
from
October 7, 2026 14:43
0da71d1 to
f398f1e
Compare
Member
|
AFAIR, GCP doesn't allow detaching the NIC from running VM. Did they change that behavior? |
Member
Author
|
Regular NICs don't support detaching. You're right, but since 2025, they have Dynamic NICs as child resources to regular NICs which you can attach/detach: https://docs.cloud.google.com/vpc/docs/multiple-interfaces-concepts#dynamic-nics |
furkansahin
added this pull request to stack #6637
October 8, 2026 10:19
furkansahin
force-pushed
the
gcp-postgres-mgmt-network
branch
from
October 8, 2026 13:48
f398f1e to
0e2a04b
Compare
GCP Postgres VMs will get a management NIC for control-plane SSH, so that a primary can be fenced by deleting its user NIC through the API, as AWS does with detach_nic. Each location gets one management subnet in the Postgres service project, named ubicloud-pg-mgmt. Subnet names are unique per project and location, so the name identifies it and needs no new column. The management subnet always gets a dedicated VPC, so it shares no network, tag key or firewall policy with another subnet. It gets no Private Google Access, so a fenced server cannot reach Google APIs through its management NIC. It gets no subnet tag key and no allow rules: VM firewalls do not apply there, and the management VPC's own rules apply to every NIC in it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The management VPC holds only the management NICs of GCP Postgres VMs. Its firewall policy gets no tag-targeted rules and no VPC-wide RFC1918 deny rules. Instead, two untagged rules allow tcp/22 from the IPv4 and from the IPv6 entries of control_plane_outbound_cidrs, one rule per address family like the VPC-wide deny rules, and two rules deny all IPv4 and IPv6 egress. The firewall is stateful, so SSH replies still pass. The egress deny keeps a fenced server from sending anything through its management NIC: no WAL to the timeline bucket, no replication, no client traffic, also when the guest gets a default route on nic0 back from DHCP. It also keeps the management NICs of two servers from reaching each other. An update_firewall_rules on the management VPC re-applies these rules instead of pushing VpcUpdateFirewallRules, so a change of control_plane_outbound_cidrs reaches the policy. The SSH rule of an address family with no CIDR is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prog::Vnet::Gcp::SubnetNexus.assemble_management creates the management subnet of a GCP location in the Postgres service project, at 100.64.0.0/20, with its strand. The strand then creates the dedicated VPC, its rules and the GCE subnet through the paths of the previous two commits. It does not use Prog::Vnet::SubnetNexus.assemble, which always attaches a firewall to the new subnet: Vm#private_subnet_firewalls joins through every NIC, so that firewall would apply to the VM of every management NIC. The range is outside RFC 1918 because every customer subnet is in RFC 1918, and GCE rejects an instance whose NICs have overlapping subnet ranges, also when the subnets are in different VPCs. The same range serves every location: each location has its own management VPC, and the VPCs are not peered. The insert ignores a conflict on the (project, location, name) unique index, and the strand is created in the same transaction, so a second call, also a concurrent one, returns the existing strand. The operator runs it once per location; the E2E setup will run it once per run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
random_private_ipv4 splits a subnet into at most 2**8 blocks and gives each NIC one block, a /28 in a /20. So the /20 management subnet would hold only 256 NICs, one per Postgres server of a location. The new PrivateSubnet#random_host_ipv4 picks a single /32 anywhere in net4 outside the addresses the provider reserves, which gives the management subnet 4,092 addresses. It checks a candidate with a lookup on the unique (private_subnet_id, private_ipv4) index of nic instead of loading every NIC of the subnet. NicNexus.assemble uses it for NICs in the GCP management subnet. It locks the subnet row first, so two assembles never pick the same address; the unique index backs that up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vm::Nexus.assemble passes management_nic_use_eip: false for a management NIC that the control plane reaches over IPv6, and the AWS NIC prog then allocates no EIP. The GCP NIC prog ignored use_eip and reserved a static external IPv4 for every NIC. Such an address would never be attached, so it would cost the unattached rate and count against the regional address quota for nothing. The GCP NIC prog now hops to wait without a static IPv4 when the frame's use_eip is false. A frame without the key keeps the old behavior, as on AWS. Today only AWS and metal callers pass use_eip: false (GitHub runners), so no GCP NIC changes until GCP management NICs exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
furkansahin
force-pushed
the
gcp-postgres-mgmt-network
branch
from
October 9, 2026 08:55
0e2a04b to
1cbbdab
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second PR of the GCP Postgres dual-NIC fencing series. Stacked on #6620, so its diff shows only this PR's commits. It adds the management network that the management NICs of GCP Postgres VMs will live in. Nothing calls it yet, so it is dark: no management subnet exists until an operator runs the bootstrap.
What it adds
Prog::Vnet::Gcp::SubnetNexus.assemble_management(location_id)creates subnetubicloud-pg-mgmtin the Postgres service project at100.64.0.0/20, plus its strand. It is idempotent. The name identifies the subnet (unique per project and location), so there is no new column.control_plane_outbound_cidrs(one rule per address family), all egress denied (IPv4 and IPv6). No tag-targeted rules and no RFC1918 deny rules.update_firewall_ruleson this VPC re-applies these rules instead of syncing VM firewalls.PrivateSubnet#random_host_ipv4, instead of one of the 256 /28 blocks thatrandom_private_ipv4gives.use_eipis false. This is for the IPv6 management mode. Today only AWS and metal callers pass it.Why these choices
100.64.0.0/20is outside RFC 1918: every customer subnet is RFC 1918, and GCE rejects an instance whose NICs have overlapping subnet ranges, also across VPCs.nic0back from DHCP.nic0. This was measured in GCE dual-NIC fence: firewall tag rules on a Dynamic NIC, test results and required changes #6619.Tested live
I ran a local control plane (respirate and monitor) from this branch against a GCP dev project, in
gcp-us-east4:assemble_managementcreated one subnet; a second call returned the same strand. The subnet and its VPC reachedwaitin about 90 s. In GCP: subnet100.64.0.0/20with Private Google Access off, no tag key, and a policy with exactly the four rules above (no RFC1918 deny rules).storage.googleapis.com, theprivate.googleapis.comVIP and the other VM are blocked.control_plane_outbound_cidrsand anupdate_firewall_rules, the IPv6 SSH rule was removed and SSH over IPv6 timed out. Restoring both brought it back.bin/e2e --provider gcp --test-cases postgres_standardpassed on this branch in 13.8 min, and left no GCP resources behind. One strand retried once on the existingVnet::Gcp::NicNexus#startrace (ps.gcp_vpc.nameon nil before the subnet links its VPC). That line is unchanged here.incr_destroyon the management subnet removed the subnet, the firewall policy and the network in about 70 s.🤖 Generated with Claude Code