AWS Network
Edi Wibowo
Contents
Virtual Private Cloud 2
VPC Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Types of route entries in Route Table . . . . . . . . . . . . . . . 2
Route Priority . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
OS level routing . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
VPC Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Identify Stateful and Stateless Packet Filtering . . . . . . . . . . 7
Stateful and Stateless Security in VPC . . . . . . . . . . . . . . . 7
Understand connection tracking in Security Group . . . . . . . . 8
Common Customer Issues . . . . . . . . . . . . . . . . . . . . . . 9
Direct Connect 12
Direct Connect Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Supported Features . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Non Supported Features . . . . . . . . . . . . . . . . . . . . . . . 12
Non Supported Traffic Flow . . . . . . . . . . . . . . . . . . . . . 13
Multi Account Direct Connect GW . . . . . . . . . . . . . . . . . 14
Transit Gateway Association . . . . . . . . . . . . . . . . . . . . 16
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
VLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
VLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
ARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Q-in-Q tunneling . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Direct Connect Billing . . . . . . . . . . . . . . . . . . . . . . . . 19
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1
Virtual Private Cloud
VPC Routing
Types of route entries in Route Table
Internet Gateway
In order to make a subnet public within a VPC, a route targeting an Internet
Gateway (IGW) must be added. A separate route entry is required in this route
table for IPv6 and IPv4, respectively.
Note: AWS customers sometimes forget to add the IPv6 route after enabling
IPv6 in the VPC, so it is important to check that this has been done, when
customers experience difficulty.
Table 1: Route Table of Public Subnet
Destination Target
0.0.0.0/0 igw-xyz
::/0 igw-xyz
NAT Gateway
NAT Gateways are only used with IPv4 traffic. When you require traffic sourcing
from instances within a private subnet to be able to connect to the Internet,
you can create a NAT gateway in a public subnet. Once you have done this,
add an IPv4 default route, targeting the NAT gateway, as seen in the private
subnet route table. AWS customers would need to use NAT in order to launch
instances into private subnets that need to reach public API endpoints (or any
public resource).
NAT gateways are not supported for IPv6 traffic—use an outbound-only (egress-
only) internet gateway instead which will be discussed later in this chapter.
2
Table 2: Route Table of Private Subnet
Destination Target
0.0.0.0/0 nat-xyz
Table 3: Route Table of Public Subnet
Destination Target
0.0.0.0/0 igw-xyz
Virtual Private Gateway
AWS Virtual Private Gateway (VGW) is used to enable instances within a VPC
to communicate with a remote (on-premises) network over a VPN or Direct
Connect service. You can either add static routes to the remote network as
the destination and VGW as the target in the route table; or enable Route
Propagation to automatically populate the remote network into the route table
in VPC.
To enable route propagation using the console:
1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/
2. In the navigation pane, choose Route Tables, and then select the route
table.
3. Choose Actions, Edit route propagation.
4. Select the Propagate check box next to the virtual private gateway,
and then choose Save.
In the route table below, we have manually added a static route to the remote
(on-premises) network with the target vgw to enable communication between
EC2 instances and the remote network (e.g. 192.168.0.0/16).
Destination Target
192.168.0.0/16 vgw-xyz
VPC Peering Connection
VPC peering provides networking connection between two VPCs over their
private IPv4 addresses in the same or different regions; or IPv6 in the same
region. In order to enable routing of traffic between the two VPCs you must add
a route in the VPC route table with a destination of a whole or part of peered
VPC CIDR and with the VPC peering connection (pcx-xxx) as the target.
To create a VPC peering connection, first create a request to peer with another
3
VPC. You can request a VPC peering connection with another VPC in the same
AWS account, or with a VPC in a different AWS account.
To activate the request, the owner of the responder VPC must accept the request.
Example: In VPC-A route table, the following route entry is manually added:
destination of 192.168.0.0/16 which is VPC-B CIDR and target VPC peering
connection pcx-0590e7be089b0fdae.
Table 5: VPC-A Route Table
Destination Target
192.168.0.0/16 pcx-0590e7be089b0fdae
Likewise: In VPC-B route table, the following route entry is manually added:
destination of 10.0.0.0/16 which is VPC-A CIDR and target VPC peering
connection pcx-0590e7be089b0fdae.
Table 6: VPC-B Route Table
Destination Target
10.0.0.0/16 pcx-0590e7be089b0fdae
Gateway VPC Endpoint
Gateway VPC endpoint enables you to create a private connection between your
VPC and specific AWS services such as S3 and DynamoDB. When you create
a gateway VPC endpoint, you need to specify which route tables in your VPC
that will use the endpoint. A route is then automatically added to each of the
route tables with a destination that specifies the prefix list ID of the service
(pl-xxx) and a target with the VPC endpoint ID (vpce-xxx). The prefix list ID
represents the range of public IP addresses used by the service.
To create and set up a gateway endpoint, follow these general steps:
1. Specify the VPC in which to create the endpoint, and the service to which
you’re connecting. A service is identified by a prefix list—the name and
ID of a service for a Region. A prefix list ID uses the form pl-xyz and a
prefix list name uses the form "com.amazonaws.region.service‘".
Use the prefix list name (service name) to create an endpoint.
2. Attach an endpoint IAM policy to your endpoint that allows access to some
or all of the service to which you’re connecting. For more information, see
Using VPC endpoint policies.
3. Specify one or more route tables in which to create routes to the service.
Route tables control the routing of traffic between your VPC and the other
service. Each subnet that’s associated with one of these route tables has
4
access to the endpoint, and traffic from instances in these subnets to the
service is then routed through the endpoint.
Table 7: Private Subnet Route Table
Destination Target
pl-xyz (“com.amazonaws.region.service”) vpce-xyz
Prefix List ID of the Service = pl-xyz
Target with the Endpoint ID = vpce-xyz
Elastic Network Interface
You can add a route to elastic network interface (eni) if you want to route traffic
out through an instance in your VPC. Use cases for this include: NAT instance,
third party firewall, etc.
Table 8: Private Subnet Route Table
Destination Target
0.0.0.0/0 eni-xyz
Egress-Only Internet Gateway
When using IPv6, there is no concept of a private IP address that is translated to
a public IP address because all IPv6 addresses are publicly routable. To provide
similar functionality to a private subnet, when using IPv6, a customer would
use an egress only Internet gateway. This allows only connections originating
from VPC toward the Internet to pass through it. To configure routing for an
egress-only Internet gateway, add a route for the private subnet that routes IPv6
Internet traffic ::/0 with target as the eigw-id.
Table 9: Private Subnet Route Table
Destination Target
::/0 eigw-id
An egress-only Internet gateway is a horizontally scaled, redundant, and
highly available VPC component that allows outbound communication over
IPv6 from instances in your VPC to the Internet, and prevents the Internet
from initiating an IPv6 connection with your instances.
5
Transit Gateway
AWS transit gateway (TGW) is a service that enables customers to connect
their AWS VPCs and on-premises networks to a single gateway. To ensure that
resources in a VPC can use a TGW to route to other VPCs or on-premises
networks, you need to create a route in the route table having a target as the
TGW ID. TGW would have to be associated with the related subnets in VPCs
within the same region.
Route Priority
We use the most specific route in your route table that matches the traffic to
determine how to route the traffic (longest prefix match).
However, when the lengths of the prefixes in the route table are the same, here
is the order of priority:
1. Local routes
2. Static routes
3. Propagated routes
In this example, your route table has a static route to an internet gateway
(which you have added manually), and a propagated route to a virtual private
gateway. Both routes have a destination of 172.31.0.0/24. In this case, all
traffic destined for 172.31.0.0/24 is routed to the internet gateway — it is a
static route and therefore takes priority over the propagated route.
Table 10: Same prefix lengths in a route table
Destination Target
172.31.0.0/24 vgw-xyz (propagated)
172.31.0.0/24 igw-xyz (static)
If the destinations for the static and propagated routes are the same, the static
route takes priority.
OS level routing
Customers may use software based appliances like Cisco’s CSR, Juniper’s vSRX
or Sopho’s firewall to make OS level routing changes. Customer may configure
these devices to be their own VPN headends, through which all traffic egresses.
Customers may also attach multiple ENI’s to an instance and can change the
OS level to prevent asymmetric routing with such a configuration. OS level
routing, should be discouraged, unless there is a valid reason, such as one of the
above. OS level routing changes can result in unmaintainable configuration and
introduces errors that are very difficult to troubleshoot as routing behaviour
becomes non-deterministic.
6
References https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html#route-
tables-priority
VPC Security
Identify Stateful and Stateless Packet Filtering
Stateless Firewall
Stateless firewall treats each network frame or packet individually. They do not
keep track of the packet context such as the nature of the traffic. Therefore, such
a firewall has no way of knowing if the packet is part of an existing connection;
or is trying to establish a new connection, or is just a rogue packet. In analogy,
it is like a security door in a data center that you have to badge in and out every
time you pass through. Every time you badge the door the security system has
to ensure you have access.
In summary, stateless firewall:
• treats each network frame or packet individually
• does not keep track of packet context
• like a security door you have to badge in and out
Stateful Firewall
Stateful packet inspection means that packets are evaluated based on their
relation to previous packets. Therefore, this method involves tracking the
operating state and characteristics of network connections traversing it. The
firewall is configured to distinguish legitimate packets for different types of
connections. Only packets matching a known active connection are allowed to
pass the firewall. In analogy, it is like in a theme park where you get a stamp
when you enter the park. Once you have gained entry you are allowed to come
and go as you please.
In summary, stateful firewall:
• packets are evaluated based on their relation to previous packets
• configured to distinguish legitimate packets for different types of connec-
tions
• only packets matching a known active connection are allowed to pass
• like a theme park gate that allows you to come and go after your hand has
been stamped.
Stateful and Stateless Security in VPC
In AWS VPC, in terms of key security features, we have Security Groups (SG)
and Network Access Control List (NACL). Both act as a firewall in VPC. They
filter packets using different approaches.
7
Stateless security is implemented through the use of NACLs. On the other hand,
stateful security is implemented using SGs.
NACLs and SGs do not filter certain types of traffic that is considered service
traffic such as traffic to and from:
• link-local addresses (169.254.0.0/16)
• AWS-reserved IPv4 addresses: AmazonProvidedDNS, DHCP, EC2 instance
metadata, Key Management Server (KMS-license management for Windows
instances)
Both SG and NACL filter based on destination port (or port range). Neither
can filter based on source port. Moreover, both have implicit deny rule.
Table 11: Differences between SG and NACL
Security Group Network Access Control List
stateful: return traffic is stateless: return traffic must be explicitly
automatically allowed allowed by rules
ENI level filtering Subnet level filtering
allow rules allow and deny rules
unordered ordered lists (starting with the lowest)
supports SG ID referencing does not support NACL ID referencing
combined limit (1000) hard limit of 40 (per direction)
Figure 1: VPC Security
Understand connection tracking in Security Group
In VPC, Security Groups use connection tracking to track information about
traffic to and from the instance. Ultimately, this allows the stateful behaviour of
security groups. Each connection state has its own timeout values.
Connection tracking is a very important concept in VPC. Connection tracking
logic is usually apply on the hypervisor host very soon after the packet hits the
network interface and is an extremely efficient process done in the kernel. It
relies on memory structures for every connection, meaning that there are limits
to the number of connections a system can track.
8
If a security group rule permits TCP or UDP flows for all traffic (0.0.0.0/0) and
there is a corresponding rule in the other direction that permits all response
traffic (0.0.0.0/0) for all ports (0-65535), the that flow of traffic is not tracked.
An exception is ICMP traffic, it is always tracked.
In this example, TCP traffic on port 80 from and to the instance is not tracked
because both the inbound and outbound rules allow all traffic (0.0.0.0/0)
Table 12: SG Inbound Rule
Protocol Type Destination Port Source IP
TCP 80 0.0.0.0/0
Table 13: SG Outbound Rule
Protocol Type Destination Port Destination IP
TCP All 0.0.0.0/0
Figure: TCP Finite State Machine (from TCP Guide by Charles M. Kozierok)
Common Customer Issues
Connection Tracking Limits Hit Unable to establish new connections (in-
bound and/or outbound)
Intermittent DNS resolution failures (in custom DNS server use case) and/or
connectivity tests failure
Solutions If customers are hitting the limits, we can suggest security group
rule changes. Sometimes, no connection tracking may be used in conjunction
with NACLs to restrict traffic to/from certain CIDR IP(s). Scaling out to reduce
the number of connections per instance, putting EC2 behind load balancer or
change the instance type to increase the total number of connections allowed. If
customer is using multiple ENIs on EC2, ask if they are using all elastic network
interfaces (ENIs). If not, they may consider removing some which should increase
connection limits per ENI.
References
Stateful Firewall https://en.wikipedia.org/wiki/Stateful_firewall
Comparison of SG and NACL https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Security.html#VPC_Se
Controlling access to instances in a subnet https://docs.aws.amazon.com/vpc/latest/userguide/vpc-
network-acls.html#nacl-examples
9
Figure 2: TCP-FSM
10
Connection Tracking https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-
security-groups.html#security-group-connection-tracking
TCP Guide http://www.tcpipguide.com/buybook.htm?free
11
Direct Connect
Direct Connect Gateway
Supported Features
Global Resource
Private and Transit Virtual Interfaces (not Public)
Allows access to VPCs across all regions and accoutns
One BGP session to multiple VPCs
Non Supported Features
VPCs to which you connect through a Direct Connect gateway cannot have
overlapping CIDR blocks.
Virtual Private Gateway (VGW) that you associate with a Direct Connect
gateway must be attached to a VPC.
You cannot associate a VGW with more than one Direct Connect gateway and
you cannot attach a private virtual interface to more than one Direct Connect
gateway.
You cannot use a Direct Connect gateway in a non-China region to a VPC in
the China regions (Beijing and Ningxia)
12
Non Supported Traffic Flow
VPC to VPC
Direct communication between VPCs that are associated with a Direct Connect
gateway is not supported.
This includes traffic from one VPC to another by using a hairpin through
on-premises network through a single virtual interface.
Figure 3: DXGW-onprem-hairpin
Direct Connect VIF to VPN
Direct communication between the virtual interfaces that are attached to a
single Direct Connect gateway and a VPN connection on a Virtual Private
Gateway (VGW) that is associated with the same Direct Connect gateway is
not supported.
13
Figure 4: DXGW-VPC_VPN
On-premises to On-premises (CloudHub)
Direct communication between the virtual interfaces that are attached to a single
Direct Connect gateway is not supported.
Multi Account Direct Connect GW
You can associate a Direct Connect gateway with a virtual private gateway that
is in a different AWS account. The owner of the virtual private gateway creates
an association proposal and the owner of the Direct Connect gateway must
accept the association proposal. An association proposal can contain prefixes
that will be allowed from the virtual private gateway. The owner of the Direct
Connect gateway can optionally override any requested prefixes in the association
proposal.
Note:
• Customer can associate up to 10 VPCs from multiple accounts with a
Direct Connect gateway.
• There are no additional charges to use multi-account Direct Connect
gateway
• Multi-account support for Direct Connect gateway is available in all com-
mercial AWS regions, except for AWS China (Beijing nd Ningxia) regions.
14
Figure 5: DXGW_onprem-onprem
Figure 6: DXGW-multi_accounts
15
Transit Gateway Association
The below diagram illustrates how the Direct Connect gateway enables you to
create a transit virtual interface to access all of your VPCs through a Transit
Gateway.
Figure 7: DXGW-TGW
The solution involves the following components:
• Transit gateway (TGW) that have VPC attachments.
• Direct Connect gateway (DXGW)
• Association between the DXGW and TGW
• Transit virtual interface that is attached to the Direct Connect gateway.
You can attach up to three TGWs in the same or different regions and accounts
to a single DXGW.
Direct Connect Gateway Requirements for a TGW
• You cannot attach a DXGW to a TGW when DXGW is already associated
with a VGW or is attached to a Private VIF.
• DX GW supports communication between attached transit VIF and asso-
ciated TGW only.
• Use unique ASNs for TGW, DXGW and on-premises router.
Multi Account Support for TGW with DXGW You can associate an
existing Direct Connect gateway or a new Direct Connect gateway with a Transit
gateway that is in a different account. The owner of the transit gateway creates
an association proposal and the owner of the Direct Connect gateway must accept
the association proposal. The association proposal must contain prefixes that will
be allowed from the transit gateway. The owner of the Direct Connect gateway
can optionally override any requested prefixes in the association proposal.
16
Table 14: Allowed Prefixes
VGW - DXGW TGW - DXGW
Optional Mandatory
Prefixes can be exact match of the Prefixes can be a subnet, exact match
VPC CIDR or a supernet or a supernet of the VPC CIDR
Example, if VPC CIDR 10.0.0.0/16, Example, if VPC CIDR is
Customer can either have 10.0.0.0/16 10.0.0.0/16, customer can either have
or 10.0.0.0/8 in the allowed prefix list. 10.0.0.0/24, 10.0.0.0/16 or 10.0.0.0/8
AWS advertises only 10.0.0.0/16 to in the allowed prefix list. AWS
customer advertises the prefixes entered in the
list.
No limit Maximum of 20 prefixes per TGW
association to DXGW
References
How to create Direct Connect Gateways https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-
connect-gateways-intro.html
Associating a VGW across accounts https://docs.aws.amazon.com/directconnect/latest/UserGuide/multi-
account-associate-vgw.html
DXGW and TGW association https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-
connect-transit-gateways.html
Associating a TGW across accounts https://docs.aws.amazon.com/directconnect/latest/UserGuide/multi-
account-associate-tgw.html
Allowed Prefix List https://docs.aws.amazon.com/directconnect/latest/UserGuide/allowed-
to-prefixes.html
VLAN
VLAN
Virtual LAN is logical grouping of devices by using 802.1q tags in the same
broadcast domain. It creates the appearance of functionality of network traffic
that is physically on a single network. It acts as if it is split between separate
networks.
VLANs work by applying tags to network frames and handling theses tags in
networking systems - creating the appearance and functionality of network traffic
that is physically on a single network, but acts as if it is split between separate
networks.
VLAN tag is a 12-bit field in the Ethernet header that provides support for up
to 4096 VLANs per switching domain.
17
Each switch uses the tag to keep each VLAN traffic separate from other VLANs,
forwarding it only where the VLAN is configured. When the frame reaches the
destination switch port, the VLAN tag is removed before the frame is to be
transmitted to the destination device.
Switch port types:
• Access port can have only one VLAN configured on the interface.
• Trunk port can have two or more VLANs configured.
• Native VLAN is used to carry untagged traffic when egressing the port.
Note: AWS Direct Connect routers do not have a native VLAN configured, so
all traffic must be tagged with 802.1q.
Customers can have multiple private, public or transit virtual interfaces (VIFs)
over the same Direct Connect connection such as physical connection.
VLAN identifiers or tags are used to logically distinguish and identify network
traffic associated with individual VIFs over the same physical Direct Connect
connection.
ARP
Address Resolution Protocol (ARP) is used to map IP network addresses to
hardware addresses used by Data Link protocol.
To reduce the number of address resolution requests, a device normally caches
resolved addresses for a period of time. On AWS Direct Connect router, the
default cache value is 300 seconds.
An ARP request is a broadcast and an ARP response is a unicast.
Q-in-Q tunneling
Q-in-Q tunnelling is used by service providers to segregate and identify each of
its custoemers network traffic, which might be using the same 802.1q tag values
when sent over shared service provider network.
If a customer has a Direct Connect connection using a APN partner, the APN
partner might be using Q-in-Q tunnelling to distinguish each customer’s traffic
when sent over its own shared network.
Please ensure that when the service provider is forwarding traffic to AWS Direct
Connect router, they should strip off the S-VLAN tag and only have C-VLAN
tag attached to it. AWS will drop any frame that has more than one VLAN tag.
Customer configuration:
• Encapsulation method is 802.1q.
• Switch port is configured in trunk mode (not access mode) if customer is
using switch virtual interface (SVI).
18
• VLAN ID used for the VIF is not configured as native VLAN.
• IP address and mask configured on the sub-interface is correct.
Direct Connect Billing
Scenario 1:
An EC2 instance in Account X sends 1 GB of data to a public prefix that was
advertised to AWS over a Public VIF owned by Account X.
Outcome 1:
Account X is charged Direct Connect Data transfer out rate for the 1 GB of
data - subject to the source AWS region and Direct Connect location.
Scenario 2:
An EC2 instance in Account X sends 1 GB of data to a public prefix that was
advertised to AWS over a public VIF owned by Account Y.
Outcome 2:
Account Y incurs no charges. Account X is charged the standard EC2 data
transfer out to Internet cost.
Scenario 3:
An EC2 instance in Account X sends 1 GB of data to a public prefix that was
advertised to AWS over a public VIF owned by Account Y. Both Account X and
Account Y are consolidated under the same payer account.
Outcome 3:
Account X is charged Direct Connect Data transfer rate for the 1 GB of data -
subject to the source AWS region and Direct Connect location. The charge is
aggregated to the master payer account.
References
L2 Troubleshooting https://docs.aws.amazon.com/directconnect/latest/UserGuide/Troubleshooting.html#ts-
layer-2
Direct Connect Pricing https://aws.amazon.com/directconnect/pricing/
19
