blog@insidesystems.net

After some experimentation, it became clear that the least bad solution would be to create a statically linked 32-bit binary, that could be run an i386 compatible kernel.

As such, first we built a new amd64 kernel with the following added to the kernel configuration:

options       COMPAT_IA32

Then on an FreeBSD/i386 machine, we went about building a binary package with a statically linked binary, by making a few edits to the darcs ports Makefile. Here is the diff:

--- Makefile.orig       Sat Feb 10 14:02:34 2007
+++ Makefile    Sat Feb 10 14:18:44 2007
@@ -14,10 +14,10 @@
 MAINTAINER=    haskell@FreeBSD.org
 COMMENT=       Yet another replacement for CVS, written in Haskell

-BUILD_DEPENDS= ghc:${PORTSDIR}/lang/ghc
-LIB_DEPENDS=   curl:${PORTSDIR}/ftp/curl \
-               gmp.7:${PORTSDIR}/math/libgmp4 \
-               readline.5:${PORTSDIR}/devel/readline
+BUILD_DEPENDS= ghc:${PORTSDIR}/lang/ghc \
+               curl:${PORTSDIR}/ftp/curl \
+               /usr/local/lib/libgmp.so.7:${PORTSDIR}/math/libgmp4 \
+               /usr/local/lib/libreadline.so.5:${PORTSDIR}/devel/readline

 OPTIONS=       SERVER "install server" off
 USE_AUTOTOOLS= autoconf:259
@@ -25,6 +25,7 @@
 CONFIGURE_ENV= CPPFLAGS="-I${LOCALBASE}/include ${PTHREAD_CFLAGS}" \
                LDFLAGS="-L${LOCALBASE}/lib -L${PREFIX}/lib/ ${PTHREAD_LIBS}" \
                CFLAGS="" 
+CONFIGURE_ARGS= --with-static-libs
 USE_GMAKE=     yes
 MAKEFILE=      GNUmakefile
 ALL_TARGET=    darcs darcs.1

And then, still on the i386, in ports/devel/darcs, I ran make package to create a new darcs binary package, containing the statically linked binary.

This package was then able to be copied over to the amd64 machines, and installed via pkg_add, without issue.

Edit: This binary then worked fine on a test repository, but it dumped core on a large, production repository. As such we switched to a different tactic. Read about it in Part 2.

make delete-old
>>> Removing old files (only deletes safe to delete libs)
>>> Old files removed
>>> Removing old directories
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.

If you decide to run “make delete-old-libs”, you’ll receive this dire warning.

make delete-old-libs
>>> Removing old libraries
Please be sure no application still uses those libraries, else you
can not start such an application. Consult UPDATING for more
information regarding how to cope with the removal/revision bump
of a specific library.
remove /lib/libalias.so.4?

And now what? Do any of your apps require /lib/libalias.so.4? I sure don’t know, and I don’t want to deal with unexpected breakage, so I came up with the attached script.

It is written to function as a nagios plugin, but it is perfectly usable from the command line as well. The only thing to note is that you will want to add a “-v” or two, in order to get detail, if it does say that anything is wrong.

What it does is relatively simple:

1. It compiles a list of all the spots where executables and libraries are likely to live. It does this by grabbing a list of all the directories in your PATH, and in your ldconfig’s hints file.
2. It finds all of the dynamically linked ELF objects in those directories. (or to say that in English, it finds all your applications and libraries.)
3. It checks each of these for two things:
   1. That the library it depends on exists
   2. That the library it depends on is not listed in /usr/src/ObsoleteFiles.inc
4. Then it generates a report, that varies in length depending on whether you used the “-v” option 0, 1 or 2 times.

We then distributed this script across all of our servers, and setup Nagios to automatically alert us if we have any unexpected library dependency failures (or if any of the libraries we built our applications against have become obsolete.)

Hopefully this can save you some trouble, and allow you to run “make delete-old-libs” fearlessly, to keep your computing environment neat, tidy, safe and secure!

-----------------------------

#!/usr/bin/env ruby
#
# Checks a FreeBSD system for incorrectly linked libraries

require 'optparse'
require 'timeout'

Version='1.0.1'
OPTIONS = {
  :verbose => 0,
  :obsoletefiles => '/usr/src/ObsoleteFiles.inc',
  :warning => 0,
  :critical => 0,
  :timeout => 0,
  :obsolete => true
}

OptionParser.new do |opts|
  script_name = File.basename($0)
  opts.banner = "Usage: #{script_name} [options]" 

  opts.on("-o", "--[no-]obsolete", "Warn if system is using libs listed in /usr/src/ObsoleteFiles.inc.  Default: #{OPTIONS[:obsolete]}") do |o|
    OPTIONS[:obsolete] = o
  end

  opts.on("-w", "--warning N", Integer, "Warning threshold.  Default: #{OPTIONS[:warning]}") do |n|
    OPTIONS[:warning] = n
  end

  opts.on("-c", "--critical N", Integer, "Critical threshold.  Default: #{OPTIONS[:critical]}") do |n|
    OPTIONS[:critical] = n
  end

  opts.on("-t", "--timeout N", Integer, "Timeout (in seconds.  0 means infinite.).  Default: #{OPTIONS[:timeout]}") do |n|
    OPTIONS[:timeout] = n
  end

  opts.on("-v", "--[no-]verbose", "Run verbosely.  If specified more than once, get more verbose.") do |v|
    if v
      OPTIONS[:verbose] += 1
    else
      OPTIONS[:verbose] = 0
    end
  end

  opts.on_tail("-h", "--help", "Show this message") do
    puts opts
    exit
  end

  opts.on_tail("-V", "--version", "Show version") do
    puts Version
    exit
  end
end.parse!

#
# track status with these
sys_error=false
error_cnt=0
warn_cnt=0
error_files=''
error_long=''

# wrap this whole fucker in a timeout loop, since nagios plugins
# are supposed to have a timeout option, by spec
begin
  Timeout::timeout(OPTIONS[:timeout]) {

    #
    # Check the major executable paths...
    dirlist=`/usr/bin/printenv PATH`.split(':')

    # And now get all the major lib paths
    `/sbin/ldconfig -r`.each_line{ |ldconfout|
      if ldconfout =~ /search directories: (.*)/
    dirlist << $1.split(':')
    break
      end
    }

    # And now read in the ObsoleteFiles OLD_LIBS list, to
    # help find things that will go away after a make delete-old-libs

    obsolete = Array.new
    if OPTIONS[:obsolete] 
      begin
    File.open(OPTIONS[:obsoletefiles]).each_line { |line|
      next if line !~ /^OLD_LIBS\+=(.*)/
      obsolete << "/#{$1}" 
    }
      rescue
    sys_error=true
    error_files="#{OPTIONS[:obsoletefiles]} not found." 
    error_long="#{OPTIONS[:obsoletefiles]} not found. Unable to check for obsolete libs." 
      end
    end

    dirlist.each { |dir|
      # Make sure the directory exists
      next if ! File.exists?(dir.to_s)

      # Get every file type
      `/usr/bin/file -N #{dir}/*`.each_line { |fileout|
    error_flag=false  # flag var
    warn_flag=false
    error_libs=''
    warn_libs=''

    # If it isn't an ELF object, we aren't bothering to check it
    # same goes if it is statically linked
    name, type = fileout.split(': ')
    next if type !~ /^ELF/
    next if type =~ /statically linked/

    # check the ELFs
    `/usr/bin/ldd -a -f '%o:\t%p\n' #{name}`.each_line { |lddout|
      lib, loc = lddout.split(":\t")

      # If loc is nil, move on
      next if ! loc.kind_of?(String)

      # Strip the symbol addresses and trailing crap
      loc.strip!.gsub!(/ \(.*/,'')

      # spew an error if a lib is unfindable
      if loc =~ /not found/
        error_libs << "\tmissing object: #{lib}\n" 
        error_flag=true
      # check if it is obsolete, but present
      elsif obsolete.index(loc)
        error_libs << "\tobsolete object: #{lib}\n" 
        warn_flag=true
      end
    }

    # Something hit the fan.
    if error_flag
      error_cnt+=1
      error_files << "#{name} " 

      pinfo=`pkg_info -W #{name}`
      if pinfo =~ /was installed by package (.*)/
        error_long << "#{name} from package #{$1} depends on missing libs.\n" 
      else
        error_long << "#{name} from unknown package depends on missing libs.\n" 
      end
      error_long << error_libs
    end

    # Something will hit the fan if you run make delete-old-libs
    if warn_flag
      warn_cnt+=1
      error_files << "#{name} " 

      pinfo=`pkg_info -W #{name}`
      if pinfo =~ /was installed by package (.*)/
        error_long << "#{name} from package #{$1} depends on obsolete libs.\n" 
      else
        error_long << "#{name} from unknown package depends on obsolete libs.\n" 
      end
      error_long << error_libs
    end
      }
    }
  }
rescue Timeout::Error
  sys_error=true
  error_files="Search timed out after #{OPTIONS[:timeout]} seconds" 
  error_long="Search timed out after #{OPTIONS[:timeout]} seconds" 
end

# Tests are over, time to say what was wrong
# 
if sys_error
  error_status="Unknown: " 
  rc=3
else
  case error_cnt + warn_cnt
    when 0 
      error_status="OK: " 
      rc=0
    when 0...OPTIONS[:warning] 
      error_status="OK: " 
      rc=0
    when OPTIONS[:warning]..OPTIONS[:critical]  
      error_status="Warning: " 
      rc=1
    else
      error_status="Critical: " 
      rc=2
  end
end

# Nagios specifies the following verbosity levels
#   0 - Single line, minimal output
#   1 - single line, additional information
#   2 - multi-line, debug output
#   3 - lots of detail
# We will implement 0-2 for the moment.
case OPTIONS[:verbose]
  when 0 
    error_status << "#{error_cnt + warn_cnt} problems." 
  when 1 
    error_status << "#{error_cnt + warn_cnt} problems - #{error_files}" 
  else
    error_status << "#{error_cnt + warn_cnt} problems\n#{error_long}" 
end

puts error_status
exit rc

Tools

For this exercise, we will be using FreeBSD 5.4, PF, Exim, and some basic shell scripting.

• Why FreeBSD and not OpenBSD? Jails, but that’s another article entirely.
• What’s so special about PF? PF just happens to have integrated passive OS fingerprinting so a simple keep state rule for each fingerprint allows us to use pfctl to see which source IP matches which fingerprint.
• What about (Postfix|Sendmail|Qmail|Exchange)? This will work with those too, but we happen to like Exim. Yes, you can do it with Exchange too, but again that’s another article.

Proof of Concept

Phase One – Get PF Fingerprinting

In order to use PF on FreeBSD 5.4, the pf kernel module must first be loaded and pf enabled:

  server# kldload pf
  server# pfctl -e
  No ALTQ support in kernel
  ALTQ related functions disabled
  pf enabled
  server#

You can disregard anything about ALTQ, that’s another article. Now that pf is loaded and enabled, we need some actual rules in order to do the matching. First let’s create a minimal /etc/pf.conf:

ext_if="bge0" 
set fingerprints "/etc/pf.os" 

(Note: You’ll want to change bge0 to your actual ethernet interface found by running ifconfig -a.)

With a minimal pf.conf created, the rules used for fingerprinting are next. To accomplish this, we’ll run a command that gets the list of supported fingerprints, strips off the first two lines, removes extraneous spaces, reverses the input, and adds it all to pf.conf (it’s all one command so you can cut and paste the four lines at once into a shell and it should work):

/sbin/pfctl -qso | sed 's,[[:space:]], ,g' | \
egrep -v '^(-----|Class)' | \
sed 's,\(.*\)[[:space:]],pass in quick proto tcp from any os "\1" to \$ext_if keep state label "\1",' | \
tail -r >> /etc/pf.conf

Now that pf.conf has rules, they can be loaded with pfctl -f /etc/pf.conf and we can start to see some of the information we need (to reload pf.conf on reboot, consult the FreeBSD Handbook):

• pfctl -vvsl shows the traffic and packet count per rule
• pfctl -vvsr shows lists the actual rule and current state matches (contains the rule number which is part of the formula)
• pfctl -vvss shows the current states tracked (the other part of the formula)

Phase Two – Fingerprint Fetching Script

Information is nice but we still need access to it from inside the MTA. For that we will create a shell script (called get_os.sh in my case, and don’t forget chmod +x) that returns the OS or ‘Unknown’:

#!/bin/sh
tempvar=`pfctl -qvss | grep -A 2 $1 | egrep -m 1 -o 'rule[[:space:]][[:digit:]]*' | sed 's,rule ,,'`
if [ -z $tempvar ]; then
        echo "Unknown";
        exit;
else
        pfctl -qvvsr | egrep -m 1 "^@$tempvar" | egrep -m 1 -o '"[^"]*"' | uniq | sed 's,",,g'
fi

Go on, try it out. SSH back into the server from somewhere (to reset the state in PF) and try it with your source IP (replacing the x’s with your IP of course):

/usr/local/etc/exim/get_os.sh xxx.xxx.xxx.xxx

Last thing to note, the permissions on /dev/pf will need to be changed from 600 to 644 because this script will run as mailnull:mail which has permission to do nearly nothing:

chmod 644 /dev/pf

(Note: Yes, this is an enormous security problem, but we’ll address this later. Remember, it’s a proof of concept not a space shuttle.)

Phase Three – Putting it in Exim

There are lots of different ways to put this in exim but to keep things simple, we’ll put this in the acl_smtp_data ACL in order to just add a header for later:

warn message = X-OS-Fingerprint: ${run {/usr/local/etc/exim/get_os.sh $sender_host_address}{$value}{Unknown}} ($sender_host_address)

(Note: Make sure that your get_os.sh script is in a place where the exim server can see it and has permission to execute it. This usually means in /usr/local/etc/exim and chown mailnull:mail)

All that’s left is a reload of exim with the new ACL line:

kill -HUP `cat /var/run/exim.pid`

Each message that goes through should now have the header specifying the OS that was scanned with PF.

Results

To determine some results, we tracked the accepted/rejected status of each e-mail in addition to it’s fingerprint for about one week. Combining all OS variants into single groups, the results were somewhat interesting:

Operating System	Accepted	Rejected	Ratio
Unknown	83052	300356	78.34%
AIX	2716	100601	97.37%
OpenBSD	4111	22719	84.68%
Windows	2827	2823	49.96%
Linux	2946	706	19.33%
FreeBSD	801	2056	71.96%
AOL	4	772	99.48%
NetApp	622	109	14.91%
PocketPC	84	565	87.06%
MacOS	80	135	62.79%
ULTRIX	89	28	23.93%
OpenVMS	3	91	96.81%
AXIS	0	85	100.00%
OS/400	1	43	97.73%
Alteon	0	37	100.00%
Tru64	2	27	93.10%
NewtonOS	19	2	9.52%
IRIX	7	14	66.67%
Clavister	7	1	12.50%
SCO	0	8	100.00%
BeOS	0	6	100.00%
Contiki	1	3	75.00%
HP-UX	0	3	100.00%
Dell	0	3	100.00%
BSD/OS	0	1	100.00%

Why isn’t Windows at the top? Why is AIX at the top? Contiki is an OS? Doesn’t Alteon run switches? These are all fine questions. The answer to many of them is that passive OS fingerprinting is not as accurate as active OS fingerprinting—the same passive fingerprint might actually hit several different kinds of Operating Systems. My PowerBook shows up as NetApp 5.2.1 even though it’s much too shiny to be one of those, for instance.

So what’s the point if it’s not accurate? Well, here is the fun part about statistics. It doesn’t actually matter how correct the information is as long as it’s consistent. Simply based on the fingerprint, correct or not, the data shows that we can be >97% certain that an e-mail is spam if it comes from something passively identified as AIX (assuming the rest of the anti-spam system is accurate and the test sample is large enough.)

And since I know you are all wondering, let’s run a breakdown for Windows:

Operating System	Accepted	Rejected	Ratio
Windows 2000 RFC1323	1134	584	33.99%
Windows 98 noSACK	1684	2113	55.65%
Windows 2000 SP3	16	91	85.05%
Windows NT	0	41	100.00%

It’s much less exciting than it should be, I know.[1]

Conclusion

Is it massively useful? Is it utterly pointless? Who’d have thought SCO only sent spam? Since when does AOL only send legitimate mail? Such questions are not for me, but for e-mail administrators and armchair statisticians. What I do know is that when it comes to classification of e-mail, the more tools the merrier.

Scalability

You might be saying to yourself, “This gets executed for every single e-mail that comes through? You don’t seriously expect me to bog down my already overloaded server with more slow shell scripts do you? You said minimal resource usage in the abstract!” Well, that’s true, I did. So just how long does it take to run that thing (on a xeon 2.8)?

server# time ./fingerprinter.sh 69.2.123.45
OpenBSD 3.4 opera
0.014u 0.033s 0:00.05 80.0%     178+263k 0+0io 0pf+0w
server# 

That’s plenty fast for most servers compared to the usual gauntlet of scanning and classification. There are even some optimizations left in the shell script for the shell-programming savvy. Of course, it’s not great for enormously large volumes of email, but that’s why it’s only a proof of concept. Run it as a daemon using UNIX sockets and cache the lookups for the interval that makes you and your load average the happiest – the fingerprint is unlikely to change that often given an IP.

Security

We all know that changing the permissions on /dev/pf is a cardinal sin, so let’s address that issue. This particular problem can be tackled in a number of ways, the most common is to use “sudo”http://www.courtesan.com/sudo/ to allow the mail user to execute it. If you don’t like sudo, you could use a client/server type setup as either a UNIX socket or TCP server. Details on both of these are fodder for another article though.

Difficulty

Setting up the proof of concept isn’t immensely difficult, though some minimal script programming knowledge is required as well as some minimal Exim and FreeBSD knowledge. For that, I give it a 5/10 on the difficulty scale. The difficulty level rises when addressing some of the security concerns or further optimizations.

For Crazy People

You could set up a dnsrbl that uses occasional active OS fingerprinting and timeouts to distribute aggregate IP/fingerprint mappings to a wider audience for analysis, or you could write a plugin for your favorite MTA that communicates directly with the kernel to retrieve pf information to absolutely minimize latency, or ignoring the fingerprinting aspect altogether you could use PF tables to dynamically limit the number of TCP states a spamming server is able to make or prevent it from connecting entirely thereby reducing the load on the MTA. And no, those are not recommendations (though that last one might make a decent article.)

Enjoy!

¹ Okay, so the numbers don’t actually add up to the above report. This information was taken from a live SQL database at slightly different times.