Stupid Bridge Tricks #1 - How to add more bandwidth

Kelley Reynolds — Fri, 16 Jun 2006 15:15:00 -0400

Ever been in a situation where you had one, maybe two T1 lines and it just wasn’t quite enough bandwidth? Ever thought it would it be really nice if you could just get a cheap DSL or cable account (or two) to offload some of the unexciting web and mail traffic and not have to go through the hassle of restructuring the network or even rebooting a running machine? Through the magic of OpenBSD and PF, it’s pretty easy.

Tools

For this exercise, we will be using OpenBSD 3.7 and PF. You will also need a machine with a minimum of 3 NICs (2 for the slicing and 1 per extra connection you are adding)

Why OpenBSD 3.7? That’s really old man. Yah, I know, but that is what is already installed and in production for this example. As far as I know, everything is the same with 3.9 (the current release).
What is it with you and PF? PF is ridiculously powerful and flexible .. you have to use it for a while to really understand how magical it is compared to most other tools, but that’s part of what these articles are for.
What about (ipf|iptables|whatever)? This might be possible in those systems, I don’t know. I’ve never had to explore them as PF accomplished everything I’ve needed it to thus far.

Proof of Concept

Phase One – Slicing into the existing network

The goal here is to make a transparent bridge that slides in between your existing network and the internet. In our case, this was between a ProCurve 4000m and a Cisco 2600 series. For physical cabling you will want to be careful .. you may need a crossover cable depending on the NICs you’ve chosen and your particular equipment. Bridge setup is pretty straightforward and right out of the OpenBSD FAQ but I’ll reproduce the steps here just for fun.

1. The interface going to the Cisco is fxp0 and we want it to have an IP so we can get to it remotely so we execute:

echo "inet 216.123.123.123 255.255.252.0 NONE" > /etc/hostname.fxp0

(You will obviously want to change this to a valid IP/netmask on your local network.)

2. The interface going to the ProCurve is em1 and it needs to be up to be part of the bridge so we execute:

echo "up media auto" > /etc/hostname.em1

(Again, you should pick whichever interface is going to the other device, not the one in the example)

3. Set up the bridge device for the two by editing the file /etc/bridgename.bridge0 with the contents:

add em1
add fxp0
up

(The order of the first two don’t matter, but up must be last as the startup scripts execute each line in turn.)

4. Enable forwarding of packets by uncommenting the following line in /etc/sysctl.conf:

net.inet.ip.forwarding=1

(You should look through some of these just for fun, you can encrypt swap pages and everything)

5. At this point, you should reboot the machine (to make sure that the settings come up properly on reboot) and test it between a workstation and the switch to make sure that the bridging is in fact working as advertised (outgoing interface now goes to the switch and the bridged interface goes to the workstation). Once you are certain that it is functioning properly, the additional internet connection can be added.

Phase Two – Adding another internet connection

To add another connection, we basically repeat some of the steps above but for the new interface (in our case a DSL modem and em0)

1. Set the interface in /etc/rc.conf

echo "inet 192.168.1.64 255.255.0.0 NONE" > /etc/hostname.em0

(You will need to alter this line depending on what type of configuration comes from your extra connection. In our case it’s a broken Speedstream modem that can’t really issue DHCP so we pick would would’ve been discovered as a DHCP address as a static address.)

2. Reboot and try to ping the new device by IP. If you can’t, don’t bother moving on until you can otherwise you will simply destroy your network.

Phase Three – Configuring PF to NAT sometimes

This step can be a little tricky because not all protocols can really be sliced out this way. FTP is a great example of this type of annoyance, but many other common ones such as SSH, HTTP, SMTP, IMAP have no issues at all. For the sake of clarity, I will just copy an entire example pf.conf and explain it after the fact.

#### Configuration for basic functions
# Specify our interfaces
ext_if1="fxp0" 
ext_if2="em0" 
ext_gw2="192.168.0.1" 
int_if="em1" 

# This contains localnet traffic that does not go out a T (shouldn't be state tracked (for this application anyway))
table <localnet> persist { 216.28.123.0/24 216.28.124.0/24 216.28.125.0/24 216.29.126.0/24 }

# Make this a # sign to disable the dsl routing
dsl_enable = "#" 

# A list of all ports to send through the dsl connection
dslports="{ 22 80 110 143 443 }" 
# Table of ips to use the dsl modem for outgoing traffic
dslnet = "{ 216.28.123.34/32 216.28.123.175/32 216.28.123.176/32 216.28.123.99/32 216.28.123.10/32 }" 

# Network address translations
$dsl_enable nat on $ext_if2 from $dslnet to any port $dslports -> ($ext_if2)

# Pass quick all of our local traffic without further ado
pass quick on { $ext_if1 $int_if } from <localnet> to <localnet>

# Run our load-balancer here
# pass our dsl ports first
$dsl_enable pass in quick on $int_if route-to ($ext_if2 $ext_gw2) proto { tcp udp } from $dslnet to !<localnet>  port $dslports modulate state label "DSL-1 $srcaddr:$dstport" 
$dsl_enable pass out quick on $ext_if1 route-to ($ext_if2 $ext_gw2) proto { tcp udp } from $ext_if2 to any port $dslports modulate state label "DSL-2 $srcaddr:$dstport" 

# Modulate state on all connections
pass on $ext_if1 modulate state

You can of course add filtering and optimizing rules here but for the purposes of this article, we’ll stick to the salient points.

1. The definitions at the top define which interfaces and IPs go where. 192.168.0.1 is the IP address of our DSL gateway, you should use whatever the equivalent is.

2. The <localnet> table helps to make sure that local traffic that normally bounces off of the router doesn’t go out the DSL modem (and is right handy for queueing).

3. The $dsl_enable macro just lets is quickly and efficiently disable the DSL routing if there is a suspected problem.

4. The $dsl_ports macro lets us specify explicitly which outgoing connections should go through the DSL modem by destination port. Specifying which ones not to go through will only frustrate you (yes, that is the voice of experience).

5. The $dsl_net macro lets us specify which internal machines are supposed to go through the DSL. Might be one of them, might be all of them, it’s up to you.

6. The NAT line mangles the packets so they get routed/returned properly.

7. Pass all of the local traffic without mangling of any sort.

8. Route the DSL traffic appropriately. These are somewhat dense rules but I’ll attempt to convert them to english:

Pass in any traffic coming into the internally bridged interface and route it to the DSL modem (udp and tcp only) from machines that are in the DSLnet that are making non-local connections on the dsl_ports.
Pass out traffic on the original external interface and route it to the DSL modem if it’s from a machine in the DSLnet to a non-local dsl_port.

9. Modulate state just for fun. Never hurts to have a clean TCP stream.

As an addendum, you’ll notice that there are labels on the two lines that pass DSL traffic .. this is so that you can run pfctl -vvsl and see whether or not and by how much it’s working. For more information on what all those numbers mean, read the pf.conf man page.

Now it is time to enable pf:

# pfctl -e
pf enabled
#

For it to be enabled on startup, edit the appropriate line in /etc/rc.conf.

pf=YES                  # Packet filter / NAT

Load in the ruleset:

# pfctl -f /etc/pf.conf
#

At this point, if there are any errors in pf.conf, the pfctl should let you know where.

As an example, a line from pfctl -vvsl on my setup now looks like:

DSL-1 216.28.123.99:80 17 702 377264

Which means that the ip 216.28.123.99 has sent transferred 702 packets for a total of 377264 bytes. Now, that doesn’t mean much now, but after that snapshot I installed FreeBSD and used portsnap to download 41M at an average of 467K/s. For those of you who’ve saturated a T1 line, you know that’s about 2.5 times as fast as a T1. Yep, I’d say it’s working ;)

Conclusion

Having run this in a production environment for a small shop for a year and a half, it’s certainly stable and reliable even though it likely breaks more RFCs than you can shake a stick at. That said, let’s sum up what this gives us:

Lots of extra bandwidth with zero configuration other than swapping some cables and tweaking the bridge machine itself. This makes it really nice to put into a legacy network where making alterations to the router or replacing it entirely are out of the question.

Though fodder for another article, you can have more than one external connection and you can round-robin between them. You want 6Mbps*2? Get two DSL accounts. Or two cable modems. Or ten. You are really only limited here by the number of NICs you can force into the machine.

If you are running servers on your network, this will not disrupt any of those connections since they are coming from external sources.

Having an OpenBSD bridge in the network is a good thing anyway because you can queue traffic, passively OS fingerprint, log/graph, run intrusion detection, whatever you want to do (and yes, you can run two failover bridges if you are paranoid that this crazy blackbox might eat your whole network if it goes down, but that’s another article)

Also, if this saves you lots of money and time and effort, consider donating some of it back to OpenBSD, they’ve earned it.

"Stupid Bridge Tricks #1 - How to add more bandwidth" by Kelley Reynolds

Mon, 19 Jun 2006 01:15:31 -0400

To be honest with you, it might not actually break any RFCs but it really seems like it ought to. Mostly it’s just to let people know it’s a dirty hack and to be aware of that fact if they decide to put it into production (though it appears to be a reliable dirty hack given the years we’ve used it in production with no ill effect).

For an increasing number of situations we deal with, it’s a great solution to instantly increase bandwidth without any sort of more complicated rejiggering… very handy for clients who have outgrown capacity on one line but (decide not to|can’t) afford a complete network restructuring.

"Stupid Bridge Tricks #1 - How to add more bandwidth" by frank

Sun, 18 Jun 2006 00:17:20 -0400

> even though it likely breaks more RFCs than you can shake a stick at

I’m just curious (and maybe stupid): which ones?

btw: great article giving a “ready rolled” experience :-)

blog@insidesystems.net: Stupid Bridge Tricks #1 - How to add more bandwidth