blog@insidesystems.net: Category Networking

Kerberos and FreeBSD

Kevin Way — Thu, 11 Dec 2008 19:17:00 -0500

I hit a bit of a stumper today. I was configuring a new Kerberos realm on a bunch of servers (Solaris, OpenSolaris, FreeBSD and OpenBSD), and everything seemed to work perfectly. I could use GSSAPI to auth into any of the machines from my OS X laptop.

Then I discovered something odd, I couldn’t log into any of the OpenBSD machines from the FreeBSD machines. After checking all the normal potential problems (DNS, rDNS, NTP, krb5.conf and keytabs), it was still broken. So I ran sshd with it in debug mode, attached to the console, on a high port, and in the midst of the spew was the following message:

GSSAPI MIC check failed

After some research, it turns out that various versions of Heimdal do MIC checking in different and incompatible manners, and that FreeBSD 7.1 was still shipping with a quite old and incompatible default setting.

Fortunately the OpenBSD man pages gave me a hint, and the problem was remedied by adding:


[gssapi]
    correct_des3_mic = host/*@MYREALM

to krb5.conf on the FreeBSD machines, and now they all seem to be playing nicely once more.

Improving Network Reliability on FreeBSD

Kevin Way — Fri, 27 Jun 2008 16:05:00 -0400

If you didn’t notice it during the FreeBSD 7.0/6.3 release, FreeBSD got a great new tool from OpenBSD. The lagg device.

This device allows you to setup links with failover, or to combine them using LACP, and the operation is dead simple. Here’s an example rc.conf, that just does a basic link failover:


cloned_interfaces="lagg0" 
ifconfig_bge0="up" 
ifconfig_bge1="up" 
ifconfig_lagg0="laggproto failover laggport bge1 laggport bge0 192.168.1.5 netmask 255.255.255.0"

Or if you use 802.1q trunks


cloned_interfaces="lagg0 vlan0" 
ifconfig_em0="up" 
ifconfig_em1="up" 
ifconfig_lagg0="laggproto failover laggport em0 laggport em1" 
ifconfig_vlan0="vlan 22 vlandev lagg0 192.168.1.5 netmask 255.255.255.0"

The only downside of this, at all, is you need to write a quick nagios plugin to check for dead links, but fortunately, that’s easy enough to do as well.

Note: We don’t currently use LACP, because we’ve had some issues with it losing connectivity altogether, after alternating link failures.

Example ifconfig output from a successful lagg setup:


em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
    ether 00:15:17:73:61:f4
    media: Ethernet 100baseTX <full-duplex>
    status: active
    lagg: laggdev lagg0
em4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
    ether 00:15:17:73:61:f4
    media: Ethernet 100baseTX <full-duplex>
    status: active
    lagg: laggdev lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
    ether 00:15:17:73:61:f4
    inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect
    status: active
    laggproto failover
    laggport: em4 flags=0<>
    laggport: em0 flags=5<MASTER,ACTIVE>

Example ifconfig output from a successful vlan and lagg combination:


bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:11:0a:30:21:04
    media: Ethernet autoselect (1000baseTX <full-duplex>)
    status: active
    lagg: laggdev lagg0
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:11:0a:30:21:04
    media: Ethernet autoselect (1000baseTX <full-duplex>)
    status: active
    lagg: laggdev lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:11:0a:30:21:04
    media: Ethernet autoselect
    status: active
    laggproto failover
    laggport: bge0 flags=0<>
    laggport: bge1 flags=5<MASTER,ACTIVE>
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=3<RXCSUM,TXCSUM>
    ether 00:11:0a:30:21:04
    inet 192.168.1.145 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect
    status: active
    vlan: 22 parent interface: lagg0

Shutting Apache Up in a FreeBSD Jail

Kelley Reynolds — Mon, 12 Feb 2007 22:08:00 -0500

If you’ve run Apache in a FreeBSD jail, you’ve probably seen the following error messages in your error log:

     (61)Connection refused: connect to listener on 0.0.0.0:80
     (61)Connection refused: connect to listener on 0.0.0.0:80

It turns out that if you specify the Listen directive in one of the following ways:

     Listen 80
     Listen *:80
     Listen 0.0.0.0:80

it complains about connecting to the listener as previously mentioned. To make these errors disappear from the logs, all you have to do is specify the IP address of the jail like so:

     Listen jail.ip.address.here:80

and it goes away. No more error messages filling the logs.

Stupid Bridge Tricks #1 - How to add more bandwidth

Kelley Reynolds — Fri, 16 Jun 2006 15:15:00 -0400

Ever been in a situation where you had one, maybe two T1 lines and it just wasn’t quite enough bandwidth? Ever thought it would it be really nice if you could just get a cheap DSL or cable account (or two) to offload some of the unexciting web and mail traffic and not have to go through the hassle of restructuring the network or even rebooting a running machine? Through the magic of OpenBSD and PF, it’s pretty easy.

Tools

For this exercise, we will be using OpenBSD 3.7 and PF. You will also need a machine with a minimum of 3 NICs (2 for the slicing and 1 per extra connection you are adding)

Why OpenBSD 3.7? That’s really old man. Yah, I know, but that is what is already installed and in production for this example. As far as I know, everything is the same with 3.9 (the current release).
What is it with you and PF? PF is ridiculously powerful and flexible .. you have to use it for a while to really understand how magical it is compared to most other tools, but that’s part of what these articles are for.
What about (ipf|iptables|whatever)? This might be possible in those systems, I don’t know. I’ve never had to explore them as PF accomplished everything I’ve needed it to thus far.

Proof of Concept

Phase One – Slicing into the existing network

The goal here is to make a transparent bridge that slides in between your existing network and the internet. In our case, this was between a ProCurve 4000m and a Cisco 2600 series. For physical cabling you will want to be careful .. you may need a crossover cable depending on the NICs you’ve chosen and your particular equipment. Bridge setup is pretty straightforward and right out of the OpenBSD FAQ but I’ll reproduce the steps here just for fun.

1. The interface going to the Cisco is fxp0 and we want it to have an IP so we can get to it remotely so we execute:

echo "inet 216.123.123.123 255.255.252.0 NONE" > /etc/hostname.fxp0

(You will obviously want to change this to a valid IP/netmask on your local network.)

2. The interface going to the ProCurve is em1 and it needs to be up to be part of the bridge so we execute:

echo "up media auto" > /etc/hostname.em1

(Again, you should pick whichever interface is going to the other device, not the one in the example)

3. Set up the bridge device for the two by editing the file /etc/bridgename.bridge0 with the contents:

add em1
add fxp0
up

(The order of the first two don’t matter, but up must be last as the startup scripts execute each line in turn.)

4. Enable forwarding of packets by uncommenting the following line in /etc/sysctl.conf:

net.inet.ip.forwarding=1

(You should look through some of these just for fun, you can encrypt swap pages and everything)

5. At this point, you should reboot the machine (to make sure that the settings come up properly on reboot) and test it between a workstation and the switch to make sure that the bridging is in fact working as advertised (outgoing interface now goes to the switch and the bridged interface goes to the workstation). Once you are certain that it is functioning properly, the additional internet connection can be added.

Phase Two – Adding another internet connection

To add another connection, we basically repeat some of the steps above but for the new interface (in our case a DSL modem and em0)

1. Set the interface in /etc/rc.conf

echo "inet 192.168.1.64 255.255.0.0 NONE" > /etc/hostname.em0

(You will need to alter this line depending on what type of configuration comes from your extra connection. In our case it’s a broken Speedstream modem that can’t really issue DHCP so we pick would would’ve been discovered as a DHCP address as a static address.)

2. Reboot and try to ping the new device by IP. If you can’t, don’t bother moving on until you can otherwise you will simply destroy your network.

Phase Three – Configuring PF to NAT sometimes

This step can be a little tricky because not all protocols can really be sliced out this way. FTP is a great example of this type of annoyance, but many other common ones such as SSH, HTTP, SMTP, IMAP have no issues at all. For the sake of clarity, I will just copy an entire example pf.conf and explain it after the fact.

#### Configuration for basic functions
# Specify our interfaces
ext_if1="fxp0" 
ext_if2="em0" 
ext_gw2="192.168.0.1" 
int_if="em1" 

# This contains localnet traffic that does not go out a T (shouldn't be state tracked (for this application anyway))
table <localnet> persist { 216.28.123.0/24 216.28.124.0/24 216.28.125.0/24 216.29.126.0/24 }

# Make this a # sign to disable the dsl routing
dsl_enable = "#" 

# A list of all ports to send through the dsl connection
dslports="{ 22 80 110 143 443 }" 
# Table of ips to use the dsl modem for outgoing traffic
dslnet = "{ 216.28.123.34/32 216.28.123.175/32 216.28.123.176/32 216.28.123.99/32 216.28.123.10/32 }" 

# Network address translations
$dsl_enable nat on $ext_if2 from $dslnet to any port $dslports -> ($ext_if2)

# Pass quick all of our local traffic without further ado
pass quick on { $ext_if1 $int_if } from <localnet> to <localnet>

# Run our load-balancer here
# pass our dsl ports first
$dsl_enable pass in quick on $int_if route-to ($ext_if2 $ext_gw2) proto { tcp udp } from $dslnet to !<localnet>  port $dslports modulate state label "DSL-1 $srcaddr:$dstport" 
$dsl_enable pass out quick on $ext_if1 route-to ($ext_if2 $ext_gw2) proto { tcp udp } from $ext_if2 to any port $dslports modulate state label "DSL-2 $srcaddr:$dstport" 

# Modulate state on all connections
pass on $ext_if1 modulate state

You can of course add filtering and optimizing rules here but for the purposes of this article, we’ll stick to the salient points.

1. The definitions at the top define which interfaces and IPs go where. 192.168.0.1 is the IP address of our DSL gateway, you should use whatever the equivalent is.

2. The <localnet> table helps to make sure that local traffic that normally bounces off of the router doesn’t go out the DSL modem (and is right handy for queueing).

3. The $dsl_enable macro just lets is quickly and efficiently disable the DSL routing if there is a suspected problem.

4. The $dsl_ports macro lets us specify explicitly which outgoing connections should go through the DSL modem by destination port. Specifying which ones not to go through will only frustrate you (yes, that is the voice of experience).

5. The $dsl_net macro lets us specify which internal machines are supposed to go through the DSL. Might be one of them, might be all of them, it’s up to you.

6. The NAT line mangles the packets so they get routed/returned properly.

7. Pass all of the local traffic without mangling of any sort.

8. Route the DSL traffic appropriately. These are somewhat dense rules but I’ll attempt to convert them to english:

Pass in any traffic coming into the internally bridged interface and route it to the DSL modem (udp and tcp only) from machines that are in the DSLnet that are making non-local connections on the dsl_ports.
Pass out traffic on the original external interface and route it to the DSL modem if it’s from a machine in the DSLnet to a non-local dsl_port.

9. Modulate state just for fun. Never hurts to have a clean TCP stream.

As an addendum, you’ll notice that there are labels on the two lines that pass DSL traffic .. this is so that you can run pfctl -vvsl and see whether or not and by how much it’s working. For more information on what all those numbers mean, read the pf.conf man page.

Now it is time to enable pf:

# pfctl -e
pf enabled
#

For it to be enabled on startup, edit the appropriate line in /etc/rc.conf.

pf=YES                  # Packet filter / NAT

Load in the ruleset:

# pfctl -f /etc/pf.conf
#

At this point, if there are any errors in pf.conf, the pfctl should let you know where.

As an example, a line from pfctl -vvsl on my setup now looks like:

DSL-1 216.28.123.99:80 17 702 377264

Which means that the ip 216.28.123.99 has sent transferred 702 packets for a total of 377264 bytes. Now, that doesn’t mean much now, but after that snapshot I installed FreeBSD and used portsnap to download 41M at an average of 467K/s. For those of you who’ve saturated a T1 line, you know that’s about 2.5 times as fast as a T1. Yep, I’d say it’s working ;)

Conclusion

Having run this in a production environment for a small shop for a year and a half, it’s certainly stable and reliable even though it likely breaks more RFCs than you can shake a stick at. That said, let’s sum up what this gives us:

Lots of extra bandwidth with zero configuration other than swapping some cables and tweaking the bridge machine itself. This makes it really nice to put into a legacy network where making alterations to the router or replacing it entirely are out of the question.

Though fodder for another article, you can have more than one external connection and you can round-robin between them. You want 6Mbps*2? Get two DSL accounts. Or two cable modems. Or ten. You are really only limited here by the number of NICs you can force into the machine.

If you are running servers on your network, this will not disrupt any of those connections since they are coming from external sources.

Having an OpenBSD bridge in the network is a good thing anyway because you can queue traffic, passively OS fingerprint, log/graph, run intrusion detection, whatever you want to do (and yes, you can run two failover bridges if you are paranoid that this crazy blackbox might eat your whole network if it goes down, but that’s another article)

Also, if this saves you lots of money and time and effort, consider donating some of it back to OpenBSD, they’ve earned it.

Passively OS Fingerprinting Email with PF

Kelley Reynolds — Tue, 06 Jun 2006 20:32:00 -0400

Ever want more fodder for those Spam Assassin rules or Bayes statistics? The normal assumption is that our spam and viruses come from networks of infected zombie Windows machines, but is that really true? With passive OS fingerprinting, you can answer this question instantly and with minimal resource usage.

Tools

For this exercise, we will be using FreeBSD 5.4, PF, Exim, and some basic shell scripting.

Why FreeBSD and not OpenBSD? Jails, but that’s another article entirely.
What’s so special about PF? PF just happens to have integrated passive OS fingerprinting so a simple keep state rule for each fingerprint allows us to use pfctl to see which source IP matches which fingerprint.
What about (Postfix|Sendmail|Qmail|Exchange)? This will work with those too, but we happen to like Exim. Yes, you can do it with Exchange too, but again that’s another article.

Proof of Concept

Phase One – Get PF Fingerprinting

In order to use PF on FreeBSD 5.4, the pf kernel module must first be loaded and pf enabled:

  server# kldload pf
  server# pfctl -e
  No ALTQ support in kernel
  ALTQ related functions disabled
  pf enabled
  server#

You can disregard anything about ALTQ, that’s another article. Now that pf is loaded and enabled, we need some actual rules in order to do the matching. First let’s create a minimal /etc/pf.conf:

ext_if="bge0" 
set fingerprints "/etc/pf.os"

(Note: You’ll want to change bge0 to your actual ethernet interface found by running ifconfig -a.)

With a minimal pf.conf created, the rules used for fingerprinting are next. To accomplish this, we’ll run a command that gets the list of supported fingerprints, strips off the first two lines, removes extraneous spaces, reverses the input, and adds it all to pf.conf (it’s all one command so you can cut and paste the four lines at once into a shell and it should work):


/sbin/pfctl -qso | sed 's,[[:space:]], ,g' | \
egrep -v '^(-----|Class)' | \
sed 's,\(.*\)[[:space:]],pass in quick proto tcp from any os "\1" to \$ext_if keep state label "\1",' | \
tail -r >> /etc/pf.conf

Now that pf.conf has rules, they can be loaded with pfctl -f /etc/pf.conf and we can start to see some of the information we need (to reload pf.conf on reboot, consult the FreeBSD Handbook):

pfctl -vvsl shows the traffic and packet count per rule
pfctl -vvsr shows lists the actual rule and current state matches (contains the rule number which is part of the formula)
pfctl -vvss shows the current states tracked (the other part of the formula)

Phase Two – Fingerprint Fetching Script

Information is nice but we still need access to it from inside the MTA. For that we will create a shell script (called get_os.sh in my case, and don’t forget chmod +x) that returns the OS or ‘Unknown’:

#!/bin/sh
tempvar=`pfctl -qvss | grep -A 2 $1 | egrep -m 1 -o 'rule[[:space:]][[:digit:]]*' | sed 's,rule ,,'`
if [ -z $tempvar ]; then
        echo "Unknown";
        exit;
else
        pfctl -qvvsr | egrep -m 1 "^@$tempvar" | egrep -m 1 -o '"[^"]*"' | uniq | sed 's,",,g'
fi

Go on, try it out. SSH back into the server from somewhere (to reset the state in PF) and try it with your source IP (replacing the x’s with your IP of course):

/usr/local/etc/exim/get_os.sh xxx.xxx.xxx.xxx

Last thing to note, the permissions on /dev/pf will need to be changed from 600 to 644 because this script will run as mailnull:mail which has permission to do nearly nothing:

chmod 644 /dev/pf

(Note: Yes, this is an enormous security problem, but we’ll address this later. Remember, it’s a proof of concept not a space shuttle.)

Phase Three – Putting it in Exim

There are lots of different ways to put this in exim but to keep things simple, we’ll put this in the acl_smtp_data ACL in order to just add a header for later:

warn message = X-OS-Fingerprint: ${run {/usr/local/etc/exim/get_os.sh $sender_host_address}{$value}{Unknown}} ($sender_host_address)

(Note: Make sure that your get_os.sh script is in a place where the exim server can see it and has permission to execute it. This usually means in /usr/local/etc/exim and chown mailnull:mail)

All that’s left is a reload of exim with the new ACL line:

kill -HUP `cat /var/run/exim.pid`

Each message that goes through should now have the header specifying the OS that was scanned with PF.

Results

To determine some results, we tracked the accepted/rejected status of each e-mail in addition to it’s fingerprint for about one week. Combining all OS variants into single groups, the results were somewhat interesting:

Operating System	Accepted	Rejected	Ratio
Unknown	83052	300356	78.34%
AIX	2716	100601	97.37%
OpenBSD	4111	22719	84.68%
Windows	2827	2823	49.96%
Linux	2946	706	19.33%
FreeBSD	801	2056	71.96%
AOL	4	772	99.48%
NetApp	622	109	14.91%
PocketPC	84	565	87.06%
MacOS	80	135	62.79%
ULTRIX	89	28	23.93%
OpenVMS	3	91	96.81%
AXIS	0	85	100.00%
OS/400	1	43	97.73%
Alteon	0	37	100.00%
Tru64	2	27	93.10%
NewtonOS	19	2	9.52%
IRIX	7	14	66.67%
Clavister	7	1	12.50%
SCO	0	8	100.00%
BeOS	0	6	100.00%
Contiki	1	3	75.00%
HP-UX	0	3	100.00%
Dell	0	3	100.00%
BSD/OS	0	1	100.00%

Why isn’t Windows at the top? Why is AIX at the top? Contiki is an OS? Doesn’t Alteon run switches? These are all fine questions. The answer to many of them is that passive OS fingerprinting is not as accurate as active OS fingerprinting—the same passive fingerprint might actually hit several different kinds of Operating Systems. My PowerBook shows up as NetApp 5.2.1 even though it’s much too shiny to be one of those, for instance.

So what’s the point if it’s not accurate? Well, here is the fun part about statistics. It doesn’t actually matter how correct the information is as long as it’s consistent. Simply based on the fingerprint, correct or not, the data shows that we can be >97% certain that an e-mail is spam if it comes from something passively identified as AIX (assuming the rest of the anti-spam system is accurate and the test sample is large enough.)

And since I know you are all wondering, let’s run a breakdown for Windows:

Operating System	Accepted	Rejected	Ratio
Windows 2000 RFC1323	1134	584	33.99%
Windows 98 noSACK	1684	2113	55.65%
Windows 2000 SP3	16	91	85.05%
Windows NT	0	41	100.00%

It’s much less exciting than it should be, I know.[1]

Conclusion

Is it massively useful? Is it utterly pointless? Who’d have thought SCO only sent spam? Since when does AOL only send legitimate mail? Such questions are not for me, but for e-mail administrators and armchair statisticians. What I do know is that when it comes to classification of e-mail, the more tools the merrier.

Scalability

You might be saying to yourself, “This gets executed for every single e-mail that comes through? You don’t seriously expect me to bog down my already overloaded server with more slow shell scripts do you? You said minimal resource usage in the abstract!” Well, that’s true, I did. So just how long does it take to run that thing (on a xeon 2.8)?

server# time ./fingerprinter.sh 69.2.123.45
OpenBSD 3.4 opera
0.014u 0.033s 0:00.05 80.0%     178+263k 0+0io 0pf+0w
server#

That’s plenty fast for most servers compared to the usual gauntlet of scanning and classification. There are even some optimizations left in the shell script for the shell-programming savvy. Of course, it’s not great for enormously large volumes of email, but that’s why it’s only a proof of concept. Run it as a daemon using UNIX sockets and cache the lookups for the interval that makes you and your load average the happiest – the fingerprint is unlikely to change that often given an IP.

Security

We all know that changing the permissions on /dev/pf is a cardinal sin, so let’s address that issue. This particular problem can be tackled in a number of ways, the most common is to use “sudo”http://www.courtesan.com/sudo/ to allow the mail user to execute it. If you don’t like sudo, you could use a client/server type setup as either a UNIX socket or TCP server. Details on both of these are fodder for another article though.

Difficulty

Setting up the proof of concept isn’t immensely difficult, though some minimal script programming knowledge is required as well as some minimal Exim and FreeBSD knowledge. For that, I give it a 5/10 on the difficulty scale. The difficulty level rises when addressing some of the security concerns or further optimizations.

For Crazy People

You could set up a dnsrbl that uses occasional active OS fingerprinting and timeouts to distribute aggregate IP/fingerprint mappings to a wider audience for analysis, or you could write a plugin for your favorite MTA that communicates directly with the kernel to retrieve pf information to absolutely minimize latency, or ignoring the fingerprinting aspect altogether you could use PF tables to dynamically limit the number of TCP states a spamming server is able to make or prevent it from connecting entirely thereby reducing the load on the MTA. And no, those are not recommendations (though that last one might make a decent article.)

Enjoy!

¹ Okay, so the numbers don’t actually add up to the above report. This information was taken from a live SQL database at slightly different times.