Link Balancer is a program to help you manage multiple internet connections, or in-general multiple connections between any two points (for example you can use Link Balancer to balance a dual-path VPN between two offices) and apply policy based routing.

Link Balancer has 4 key functions:

1. It can check all your gateways, if they are alive.

You can even write your own checks for your gateways. Link Balancer can continuously monitor them and react to failures.

2. It can set multiple default gateways that will be used in a weighted round-robin fashion.

Link Balancer can configure your routing tables to achieve optimal balancing of all your upstream providers, or balance multiple VPN links.

3. It can manage your routing tables.

Link Balancer will copy routing tables and apply all changes incrementally, after comparing routing tables line by line.

4. It can apply policy based routing rules.

Link Balancer can configure policy based routing rules, allowing you to configure routes based both on source and destination IPs, on marks, tos, etc.

For example, the following can all be handled by Link Balancer:

• 2+ internet provider links are balanced concurrently.
• Balancing can be combined in any fashion. For example LAN1 is using providers A and B, LAN2 is using providers B and C, host 1 is using only B, host 2 is using A, B and C and some internet servers are always being routed through A.
• Balancing is implemented using weights, so that each provider is getting a percentage of the connections. The links do not need to be of the same speed.
• In cooperation with FireHOL, even individual services may have special routing (for example SMTP goes through DSL A, HTTP through DSL B, anything else through DSL C).

Linux supports multiple routing tables. When you enter ip route show you actually mean ip route show table main. Table main is the default routing table of the machine. You may create up to 252 routing tables. Each routing table is independent of the others, having its own routes, including a default-gateway.

Linux supports policy based routing. Run ip rule show to list it. Policy based routing allows matching traffic (using source ips, destination ips, input interface, marks and tos) and for the traffic matched, defining the routing table that will serve it. This means you can say ip rule add src 1.1.1.1 dst 2.2.2.2 lookup ABC to have all packets from 1.1.1.1 to 2.2.2.2 (and their replies) use the routes in routing table ABC.

Linux supports multiple default-gateways on each routing table. For example the command ip route add default nexthop via 1.1.1.1 weight 100 nexthop via 2.2.2.2 weight 50 nexthop via 3.3.3.3 weight 200 defines 3 default gateways (1.1.1.1, 2.2.2.2 and 3.3.3.3) that will be balanced according to the weights given. This balancing is done per connection, not per packet. Each gateway will receive complete connections, not random packets from multiple connections. This means each of these gateways may apply connection tracking on the packets it receives.

Of course you can use all these features without Link Balancer. If you do, you will face these problems:

When you have a balanced default gateway (multiple gateways as a default gateway), like the one defined above, if the interface of one gateway goes down, the whole default gateway (all of them) will be removed. You will need to add the remaining gateways back in the routing table by hand.

Link Balancer solves this by allowing you to call it from (for example) /etc/ppp/ip-down to re-apply the remaining default gateways back. It can also run continuously (once every 300 secs for example) to check gateways and update the routing tables with the remaining gateways.

Furthermore, it allows you to configure fallback gateways, i.e. gateways that will be used only when none of the primary ones is available.

When you use routing tables to apply policy based routing, you may need to copy the routes from the main table to the new routing table and update them when new routes become available. This is needed because you may have direct routes to LANs or VPNs that you want to be respected by traffic sent to the new routing table.

Link Balancer will automatically copy and update tables for you. You just define that routing table A should be a copy of table B with a different default-gateway. On each run Link Balancer will compare the routing tables and even perform incremental updates.

Link Balancer will also do a full dependency check on the tables, so that if you say A comes from B which comes from C which comes from D, it will properly process first D, then C, then B, then A. It will also report an error if you define an endless loop (a dependency graph without end).

When an interface goes down all its routes are vanished from all routing tables. But the policy based routing rules you will have, will continue to send traffic to the routing table that has no routes any more.

Link Balancer will automatically either use a fallback gateway or copy the default-gateway of the origin table to the new table, so that traffic will continue to be served by the routing table that all its gateways went down. Of course, when the interface is restored, Link Balancer will restore the proper default gateway for this interface.

Link Balancer configuration is /etc/firehol/link-balancer.conf. As with FireHOL and FireQOS, Link Balancer configuration is a BASH script file, allowing to write whatever BASH accepts in its scripts.

First you need to define all your gateways.

It is required each gateway to have each own interface. If you have multiple gateways on the same interface, it would be impossible to know which one of them is talking to your linux host. This means you will not be able to pick the right gateway when sending packets back.

If you do have multiple gateways on the same interface, you have the following options:

1. If they are on the same ethernet, try to create one vlan for each of them.
2. If you cannot use vlans, you may create IP in IP tunnels with iproute. Check this: http://www.lartc.org/lartc.html#LARTC.TUNNEL
3. Use different Ethernet devices. If they are DSL modems you can find very cheap USB Ethernet ports.

Keep in mind that if you have multi PPPoE connections on the same Ethernet, this should not be a problem, since each PPPoE should have its own PPP device.

The template for defining gateways is:

gateway GATEWAY_NAME dev DEVICE [gw "IPs"] [src "IPs"] [check METHOD ["IPs"] ]

where:

• DEVICE is the interface via which the gateway is accessible
• gw IPs is autodetected if not given. You can give multiple IPs (space separated and enclosed in quotes), but keep in mind this assumes all the IPs are exactly the same gateway.
• src IPs is autodetected if not given. This is a list IPs your host is using to access the gateways. This is not NAT or anything related. It is used to figure out how we can check if the gateway is a live (i.e ping from this IP to the gateway IP).
• available METHODs are ping traceroute and alwayson (default is ping)
• check IPs are the IPs to be checked (pinged) to figure out if the gateway is alive. There are cases where in order to be sure the gateway is alive, you need to ping something behind the gateway. In this case specify these IPs. If not given, the gateway IPs will be used.

In general:

• for point to point PPP interfaces everything is autodetected
• for point to lan PPP interfaces, you have to give gw IP and src IP
• for eth devices you have to specify the gateway IP

Hint: If you use PPP devices, specify in the peers file of pppd the unit number you want it to use. This way, you will know that, for example, your office VPN is always at ppp14. Another way is to rename the ppp device in /etc/ppp/ip-up, so that instead of having 'ppp14', you will have the interface as 'work1'.

Examples:

# wan providers
gateway dsl1 dev ppp11
gateway dsl2 dev eth3 gw 2.2.2.2
gateway dsl3 dev eth4 gw 3.3.3.3

# dual path office VPN
gateway work1 dev ppp14 check 172.16.1.2
gateway work2 dev tun0 check 172.16.1.4

Then define all the routing tables.

Tables can be defined using numbers from 1 to 252. Do not use tables 0, 253, 254 and 255. They are system tables.

You can give names to tables by editing /etc/iproute2/rt_tables If you give names, you can use them in Link Balancer.

The template for defining tables is:

table NAME [from ORIGIN_TABLE_NAME] [nodefault]

• NAME and ORIGIN_TABLE_NAME can either be IDs or names.
• The default ORIGIN_TABLE_NAME is main.
• nodefault means in case the table is left without a default gateway, do not copy the default gateway of the ORIGIN_TABLE_NAME.

Then for each table, define the default gateways you want: The template is:

default via GATEWAY_NAME [weight 0-250]

• GATEWAY_NAME is the name of the gateway as given in the gateways section.
• default weight is 100. Weights can be given as 0 to 255.
• weight 0 is special. It means don't use this unless there is no other gateway available. You can also define fallbacks as fallback via GATEWAY_NAME.

You may add as many default gateways per table, as you need. They will be balanced.

You should define one table per gateway and one for load-balancing the gateways. You need one table per gateway, so that if you receive requests from the gateways, you will respond to these requests via the same path.

Examples:

# one table per WAN gateway
table dsl1
	default via dsl1

table dsl2
	default via dsl2

table dsl3
	default via dsl3

# this is the main system routing table
table main
	default via dsl1 weight 250
	default via dsl2 weight 150
	default via dsl3 weight 50


# one table per WORK gateway
table work1 from WORK # we copy WORK here, not 'main'
	default via work1

table work2 from WORK # we copy WORK here, not 'main'
	default via work2

# this is based on 'main'
# but we don't need its default gateway
table WORK nodefault
	default via work1 weight 100
	default via work2 weight 50

You start policy based routing using this:

policy

The first thing you need to do in policy based routing is to handle responses back to the gateways. We do this using iptables connection marking.

First you should give every gateway a different connmark. You do this in firehol.conf. At the top of your firehol.conf add these:

# at the top of /etc/firehol/firehol.conf

connmark 0x1 interface ppp11  # device of dsl1
connmark 0x2 interface eth3   # device of dsl2
connmark 0x3 interface eth4   # device of dsl3
connmark 0x11 interface ppp14 # device of work1
connmark 0x12 interface tun0  # device of work2

As you see, you define the mark id and the input interface you expect to receive traffic from this gateway. These commands will mark all the incoming connections with the proper mark. These are connection marks, not just marks. The difference is that connection marks will automatically mark with the same mark every packet of the same connection.

The result is that all packets that originated from dsl1 (ppp11) and their replies will now have mark 0x1.

Next in link-balancer.conf add these next to the policy section:

# at the policy section of /etc/firehol/link-balancer.conf

policy
	connmark 0x1 table dsl1
	connmark 0x2 table dsl2
	connmark 0x3 table dsl3
	connmark 0x11 table work1
	connmark 0x12 table work2

The above says that all packets that have a the mark 0x1 should be served by the table dsl1. Similarly for the rest of the marks. You have to have this above all other policy based routing rules.

Next, you may need to have a few daemon binded on specific interfaces or IPs, and based on this alone, to have them use the proper table.

You can use this syntax:

rules src gw-src-ips GATEWAY_NAME table TABLE_NAME

gw-src-ips is a keyword that resolves to src IPs of the gateway.

The above says that all the source IPs of gateway GATEWAY_NAME should be routed via TABLE_NAME.

Examples:

rules src gw-src-ips dsl1 table dsl1
rules src gw-src-ips dsl2 table dsl2
rules src gw-src-ips dsl3 table dsl3
rules src gw-src-ips work1 table work1
rules src gw-src-ips work2 table work2

That's all you need!

The above should give you a working load-balanced environment for both your WAN and your WORK.

Now you can define advanced policy based routing.

You may add more policy based routing rules. Normally, any other rule you may add should be placed below the connmarks and above the src gw-src-ips rules.

The template for defining policy routing rules is this:

rules [not] [from|src IPs] [to|dst IPs] [mark X] [tos X] table TABLE_NAME

• not will match packets not match matched by this rule. Unlike firehol, not is applied to the entire rule.
• from or src matches the source IP of packets
• to or dst matches the destination IP of packets
• mark matches the iptables on the packets. Please used hex in the form 0xNNN.
• tos matches the type of service on the packets. Possible values are: mincost, reliability, throughput, lowdelay.

examples:

# clients at 192.168.1.0 use only dsl1
rules src 192.168.1.0/24 table dsl1

# web server at 1.2.3.4 is routed only via dsl2
rules dst 1.2.3.4 table dsl2

# even complex ones:
rules src 192.168.2.0/24 dst 10.0.0.0/24 mark 0x98 table work2

You can use LinkBalancer to find all the IP address space of your upstream providers: Run:

link-balancer asips ONE_IP_OF_YOUR_PROVIDER

LinkBalancer will query RIPE to find all the IPs of your provider. Note that you need whois and jq installed for this to work.

So you can add in link-balancer.conf:

rules dst IPS_OF_PROVIDER_1 table dsl1
rules dst IPS_OF_PROVIDER_2 table dsl2
rules dst IPS_OF_PROVIDER_3 table dsl3

You can also have all the IPs in separate files. Run:

link-balancer asips ONE_IP_OF_YOUR_PROVIDER_1 >/etc/firehol/PROVIDER1_IPS

Then you can add in link-balancer.conf:

rules dst loadfile PROVIDER1_IPS table dsl1

This is the final link-balancer configuration for balancing 3 upstream providers for your main routing table, and 2 VPNs for your WORK dual-path routing.

# -----------------------------------------

# wan providers
gateway dsl1 dev ppp11
gateway dsl2 dev eth3 gw 2.2.2.2
gateway dsl3 dev eth4 gw 3.3.3.3

# dual path office VPN
gateway work1 dev ppp14 check 172.16.1.2
gateway work2 dev tun0 check 172.16.1.4


# -----------------------------------------

# one table per WAN gateway
table dsl1
	default via dsl1

table dsl2
	default via dsl2

table dsl3
	default via dsl3

# this is the main system routing table
table main
	default via dsl1 weight 250
	default via dsl2 weight 150
	default via dsl3 weight 50


# one table per WORK gateway
table work1 from WORK # we copy WORK here, not 'main'
	default via work1

table work2 from WORK # we copy WORK here, not 'main'
	default via work2

# this is based on 'main'
# but we don't need its default gateway
table WORK nodefault
	default via work1 weight 100
	default via work2 weight 50

# -----------------------------------------

policy
	# handle replies
	connmark 0x1 table dsl1
	connmark 0x2 table dsl2
	connmark 0x3 table dsl3
	connmark 0x11 table work1
	connmark 0x12 table work2

	# add advanced policy based routing rules here

	# handle local daemons
	rules src gw-src-ips dsl1 table dsl1
	rules src gw-src-ips dsl2 table dsl2
	rules src gw-src-ips dsl3 table dsl3
	rules src gw-src-ips work1 table work1
	rules src gw-src-ips work2 table work2

You can run Link Balancer without making any changes to the system by issuing:

link-balancer -t

-t stand for TEST.

This will do everything except altering the system. It will show you though which commands is going to run.

There are cases where you need run Link Balancer from ppp scripts to enable new interfaces that just came up, or re-act to interfaces that just went down.

By default Link Balancer will check all the gateways by pinging them. This can be time consuming. You can run Link Balancer with option -f like this:

link-balancer -f

-f stands for FAST. In this mode, Link Balancer assumes the check for all gateways is alwayson. So if the interface of a gateway is up, the gateway of this interface is assumed to be alive too.

Link Balancer provides two hooks for running custom actions when something is changed.

You can define two function in /etc/firehol/link-balancer.conf like this:

updated_rules() {
	return 0
}

updated_routes() {
	local table="${1}"
	local def="${2}" # if this is 1, the default route has been updated

	return 0
}

updated_rules is called every time the policy based rules are updated.

updated_routes is called every time a table is updated. The first parameter is the table name, the second is either 0 or 1. If it is 1, the default gateway of the table has been updated.

Link Balancer allows you to write your own checks for determining if a gateway is alive or not.

The format is like this:

check_mycheck() {
	DEBUGCMD "${FUNCNAME}" "${#FUNCNAME[*]}" "${@}"
	
	local dev="$1"
	local src="$2"
	local dst="$3"
	
	if [ "${LB_IPV}" = "6" ]
	then
		echo >&2 "Checking IPv6..."
		
		# do your IPv6 check here

		# return 0 for success, or 1 for failure
	else
		echo >&2 "Checking IPv4..."
		
		# do your IPv4 check here
		
		# return 0 for success, or 1 for failure
	fi
}

Using the above you may use something like this:

gateway mygw dev eth0 check mycheck 1.2.3.4

If there are multiple check IPs, Link Balancer will call this function once for every IP.

You can call Link Balancer like this:

link-balancer boot [seconds]

This will create a screen session and run Link Balancer in it. Link Balancer will run every seconds (by default 300 = 5 mins).

You can enter this screen yourself by running screen -r link-balancer. To exit the screen without stopping link-balancer, hit Control-a, then d.