Skip to content

Add the GCP management network for Postgres - #6623

Open
furkansahin wants to merge 5 commits into
mainfrom
gcp-postgres-mgmt-network
Open

furkansahin wants to merge 5 commits into
mainfrom
gcp-postgres-mgmt-network

Conversation

@furkansahin

@furkansahin furkansahin commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Second PR of the GCP Postgres dual-NIC fencing series. Stacked on #6620, so its diff shows only this PR's commits. It adds the management network that the management NICs of GCP Postgres VMs will live in. Nothing calls it yet, so it is dark: no management subnet exists until an operator runs the bootstrap.

What it adds

  • Management subnet per location. Prog::Vnet::Gcp::SubnetNexus.assemble_management(location_id) creates subnet ubicloud-pg-mgmt in the Postgres service project at 100.64.0.0/20, plus its strand. It is idempotent. The name identifies the subnet (unique per project and location), so there is no new column.
  • Own VPC, no tags, no Google access. The management subnet always gets a dedicated VPC. Its GCE subnet has Private Google Access off. It gets no subnet tag key and no allow rules.
  • Firewall rules of the management VPC. tcp/22 in from control_plane_outbound_cidrs (one rule per address family), all egress denied (IPv4 and IPv6). No tag-targeted rules and no RFC1918 deny rules. update_firewall_rules on this VPC re-applies these rules instead of syncing VM firewalls.
  • Addresses. NICs in the management subnet get a single /32 from the whole range (4,092 addresses) through PrivateSubnet#random_host_ipv4, instead of one of the 256 /28 blocks that random_private_ipv4 gives.
  • No static IPv4 when use_eip is false. This is for the IPv6 management mode. Today only AWS and metal callers pass it.

Why these choices

  • 100.64.0.0/20 is outside RFC 1918: every customer subnet is RFC 1918, and GCE rejects an instance whose NICs have overlapping subnet ranges, also across VPCs.
  • The egress deny keeps a fenced server from sending anything through its management NIC: no WAL to the timeline bucket, no replication, no client traffic, also when the guest gets a default route on nic0 back from DHCP.
  • A dedicated VPC means the management NICs share no network, tag key or firewall policy with customers. Customer tag rules reach only NICs in the tag key's network, so they never apply to nic0. This was measured in GCE dual-NIC fence: firewall tag rules on a Dynamic NIC, test results and required changes #6619.

Tested live

I ran a local control plane (respirate and monitor) from this branch against a GCP dev project, in gcp-us-east4:

  1. Bootstrap. assemble_management created one subnet; a second call returned the same strand. The subnet and its VPC reached wait in about 90 s. In GCP: subnet 100.64.0.0/20 with Private Google Access off, no tag key, and a policy with exactly the four rules above (no RFC1918 deny rules).
  2. Behaviour. I put two test VMs in the management subnet:
    • SSH from the control plane address works over IPv4 and IPv6. tcp/5432 with a listener is blocked.
    • From the guest, internet (IPv4 and IPv6), storage.googleapis.com, the private.googleapis.com VIP and the other VM are blocked.
    • The metadata server, DNS and NTP keep working under the egress deny.
  3. Re-apply. After a restart with IPv4-only control_plane_outbound_cidrs and an update_firewall_rules, the IPv6 SSH rule was removed and SSH over IPv6 timed out. Restoring both brought it back.
  4. Regression. bin/e2e --provider gcp --test-cases postgres_standard passed on this branch in 13.8 min, and left no GCP resources behind. One strand retried once on the existing Vnet::Gcp::NicNexus#start race (ps.gcp_vpc.name on nil before the subnet links its VPC). That line is unchanged here.
  5. Teardown. incr_destroy on the management subnet removed the subnet, the firewall policy and the network in about 70 s.

🤖 Generated with Claude Code

@furkansahin
furkansahin force-pushed the gcp-postgres-mgmt-network branch from 0da71d1 to f398f1e Compare October 7, 2026 14:43
@furkansahin
furkansahin changed the base branch from main to gcp-postgres-dual-nic-prep October 7, 2026 14:43
@byucesoy

byucesoy commented Oct 7, 2026

Copy link
Copy Markdown
Member

AFAIR, GCP doesn't allow detaching the NIC from running VM. Did they change that behavior?

@furkansahin

Copy link
Copy Markdown
Member Author

Regular NICs don't support detaching. You're right, but since 2025, they have Dynamic NICs as child resources to regular NICs which you can attach/detach: https://docs.cloud.google.com/vpc/docs/multiple-interfaces-concepts#dynamic-nics

@furkansahin
furkansahin added this pull request to stack #6637 October 8, 2026 10:19
@furkansahin
furkansahin force-pushed the gcp-postgres-mgmt-network branch from f398f1e to 0e2a04b Compare October 8, 2026 13:48
Base automatically changed from gcp-postgres-dual-nic-prep to main October 9, 2026 08:55
furkansahin and others added 5 commits October 9, 2026 10:55
GCP Postgres VMs will get a management NIC for control-plane SSH, so
that a primary can be fenced by deleting its user NIC through the API,
as AWS does with detach_nic. Each location gets one management subnet
in the Postgres service project, named ubicloud-pg-mgmt. Subnet names
are unique per project and location, so the name identifies it and
needs no new column.

The management subnet always gets a dedicated VPC, so it shares no
network, tag key or firewall policy with another subnet. It gets no
Private Google Access, so a fenced server cannot reach Google APIs
through its management NIC. It gets no subnet tag key and no allow
rules: VM firewalls do not apply there, and the management VPC's own
rules apply to every NIC in it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The management VPC holds only the management NICs of GCP Postgres VMs.
Its firewall policy gets no tag-targeted rules and no VPC-wide RFC1918
deny rules. Instead, two untagged rules allow tcp/22 from the IPv4 and
from the IPv6 entries of control_plane_outbound_cidrs, one rule per
address family like the VPC-wide deny rules, and two rules deny all
IPv4 and IPv6 egress.

The firewall is stateful, so SSH replies still pass. The egress deny
keeps a fenced server from sending anything through its management NIC:
no WAL to the timeline bucket, no replication, no client traffic, also
when the guest gets a default route on nic0 back from DHCP. It also
keeps the management NICs of two servers from reaching each other.

An update_firewall_rules on the management VPC re-applies these rules
instead of pushing VpcUpdateFirewallRules, so a change of
control_plane_outbound_cidrs reaches the policy. The SSH rule of an
address family with no CIDR is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prog::Vnet::Gcp::SubnetNexus.assemble_management creates the management
subnet of a GCP location in the Postgres service project, at
100.64.0.0/20, with its strand. The strand then creates the dedicated
VPC, its rules and the GCE subnet through the paths of the previous two
commits. It does not use Prog::Vnet::SubnetNexus.assemble, which always
attaches a firewall to the new subnet: Vm#private_subnet_firewalls joins
through every NIC, so that firewall would apply to the VM of every
management NIC.

The range is outside RFC 1918 because every customer subnet is in RFC
1918, and GCE rejects an instance whose NICs have overlapping subnet
ranges, also when the subnets are in different VPCs. The same range
serves every location: each location has its own management VPC, and
the VPCs are not peered.

The insert ignores a conflict on the (project, location, name) unique
index, and the strand is created in the same transaction, so a second
call, also a concurrent one, returns the existing strand. The operator
runs it once per location; the E2E setup will run it once per run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
random_private_ipv4 splits a subnet into at most 2**8 blocks and gives
each NIC one block, a /28 in a /20. So the /20 management subnet would
hold only 256 NICs, one per Postgres server of a location. The new
PrivateSubnet#random_host_ipv4 picks a single /32 anywhere in net4
outside the addresses the provider reserves, which gives the management
subnet 4,092 addresses. It checks a candidate with a lookup on the
unique (private_subnet_id, private_ipv4) index of nic instead of loading
every NIC of the subnet.

NicNexus.assemble uses it for NICs in the GCP management subnet. It
locks the subnet row first, so two assembles never pick the same
address; the unique index backs that up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vm::Nexus.assemble passes management_nic_use_eip: false for a
management NIC that the control plane reaches over IPv6, and the AWS
NIC prog then allocates no EIP. The GCP NIC prog ignored use_eip and
reserved a static external IPv4 for every NIC. Such an address would
never be attached, so it would cost the unattached rate and count
against the regional address quota for nothing.

The GCP NIC prog now hops to wait without a static IPv4 when the
frame's use_eip is false. A frame without the key keeps the old
behavior, as on AWS. Today only AWS and metal callers pass use_eip:
false (GitHub runners), so no GCP NIC changes until GCP management NICs
exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants