blog@insidesystems.net

Improving Network Reliability on FreeBSD

Kevin Way — Fri, 27 Jun 2008 16:05:00 -0400

If you didn’t notice it during the FreeBSD 7.0/6.3 release, FreeBSD got a great new tool from OpenBSD. The lagg device.

This device allows you to setup links with failover, or to combine them using LACP, and the operation is dead simple. Here’s an example rc.conf, that just does a basic link failover:


cloned_interfaces="lagg0" 
ifconfig_bge0="up" 
ifconfig_bge1="up" 
ifconfig_lagg0="laggproto failover laggport bge1 laggport bge0 192.168.1.5 netmask 255.255.255.0"

Or if you use 802.1q trunks


cloned_interfaces="lagg0 vlan0" 
ifconfig_em0="up" 
ifconfig_em1="up" 
ifconfig_lagg0="laggproto failover laggport em0 laggport em1" 
ifconfig_vlan0="vlan 22 vlandev lagg0 192.168.1.5 netmask 255.255.255.0"

The only downside of this, at all, is you need to write a quick nagios plugin to check for dead links, but fortunately, that’s easy enough to do as well.

Note: We don’t currently use LACP, because we’ve had some issues with it losing connectivity altogether, after alternating link failures.

Example ifconfig output from a successful lagg setup:


em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
    ether 00:15:17:73:61:f4
    media: Ethernet 100baseTX <full-duplex>
    status: active
    lagg: laggdev lagg0
em4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
    ether 00:15:17:73:61:f4
    media: Ethernet 100baseTX <full-duplex>
    status: active
    lagg: laggdev lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
    ether 00:15:17:73:61:f4
    inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect
    status: active
    laggproto failover
    laggport: em4 flags=0<>
    laggport: em0 flags=5<MASTER,ACTIVE>

Example ifconfig output from a successful vlan and lagg combination:


bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:11:0a:30:21:04
    media: Ethernet autoselect (1000baseTX <full-duplex>)
    status: active
    lagg: laggdev lagg0
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:11:0a:30:21:04
    media: Ethernet autoselect (1000baseTX <full-duplex>)
    status: active
    lagg: laggdev lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:11:0a:30:21:04
    media: Ethernet autoselect
    status: active
    laggproto failover
    laggport: bge0 flags=0<>
    laggport: bge1 flags=5<MASTER,ACTIVE>
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=3<RXCSUM,TXCSUM>
    ether 00:11:0a:30:21:04
    inet 192.168.1.145 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect
    status: active
    vlan: 22 parent interface: lagg0

Monitoring iLO2 with Nagios

Kevin Way — Wed, 05 Sep 2007 08:21:00 -0400

We have a whole bunch of HP servers, purchased in large part for their excellent Lights-Out management software. We wanted to have Nagios monitor some basic things, like the status of the fans, internal temperatures, power supplies, and VRMs.

Fortunately, that turned out to be pretty easy.

The first step was to download the HP Lights-Out XML Perl Scripting Sample for Linux. I did this even though I don’t use Linux as a platform. The resulting file contains a bunch of sample XML scripts for accomplishing various goals, and a perl script (locfg.pl) that submits them to an iLO2 processor.

I then used locfg.pl to submit the following XML to each of our servers, in order to create a user with no substantial privileges.


<RIBCL VERSION="2.0">
  <LOGIN USER_LOGIN="adminuser" PASSWORD="adminpass">
  <USER_INFO MODE="write">
    <ADD_USER 
      USER_NAME="Nagios Monitor" 
      USER_LOGIN="nagiosuser" 
      PASSWORD="nagiospass">
      <ADMIN_PRIV value ="N"/>
      <REMOTE_CONS_PRIV value ="N"/>
      <RESET_SERVER_PRIV value ="N"/>
      <VIRTUAL_MEDIA_PRIV value ="N"/>
      <CONFIG_ILO_PRIV value="N"/>
    </ADD_USER>
  </USER_INFO>
  </LOGIN>
</RIBCL>

This script allowed me to quickly create unprivileged users on each of the iLO2 consoles.

Armed with an unprivileged user, I set about writing the actual plugin, and ended up with this:


#!/usr/bin/env ruby
require 'optparse'
require 'socket'
require 'openssl'
require 'rexml/document'

# Command Line Options
options = {
  :server => nil
}

opts = OptionParser.new do |opt|
  opt.banner = "Usage: #{$0}  [options]" 
  opt.on('-s', '--server HOSTNAME', String, "Hostname or IP of the server to query") { |i| options[:server] = i }
end
opts.parse!(ARGV)

if not options[:server]
  $stderr.puts "Server must be specified" 
  exit!
end

# iLO XML
xml_start = <<EOF
<RIBCL VERSION="2.22">
  <LOGIN USER_LOGIN="nagiosuser" PASSWORD="nagiospass">
EOF

xml_end = <<EOF
  </LOGIN>
</RIBCL>
EOF

xml_emhealth = <<EOF
<SERVER_INFO MODE="read">
  <GET_EMBEDDED_HEALTH />
</SERVER_INFO>
EOF

error_cnt = 0
error_msg = ''
error_summary = ''

s = TCPsocket.open(options[:server], 443)
ssl = OpenSSL::SSL::SSLSocket.new(s, OpenSSL::SSL::SSLContext.new)
ssl.sync
ssl.connect
ssl.write("<?xml version=\"1.0\"?>\r\n")
ssl.write(xml_start)
ssl.write(xml_emhealth)
ssl.write(xml_end)
ssl.flush
res = ssl.readlines

ssl.close
s.close

doc = REXML::Document.new(res.to_s.match(/<GET_EMBEDDED_HEALTH_DATA>.*<\/GET_EMBEDDED_HEALTH_DATA>/m).to_s)

if ! doc.elements["GET_EMBEDDED_HEALTH_DATA"]
  error_cnt += 1
  error_msg += "Unable to fetch embedded health data\n" 
end

doc.root.elements["FANS"].each_element('//FAN') { |mod|
  if mod.elements["STATUS"].attributes['VALUE'] != 'Ok'
    error_cnt += 1
    error_msg += "#{mod.elements['LABEL'].attributes['VALUE']} - #{mod.elements['ZONE'].attributes['VALUE']} - #{mod.elements['STATUS'].attributes['VALUE']}\n" 
    error_summary += "#{mod.elements['LABEL'].attributes['VALUE']}." 
  end
}

doc.root.elements["TEMPERATURE"].each_element('//TEMP') { |mod|
  if mod.elements["STATUS"].attributes['VALUE'] != 'Ok' and mod.elements["STATUS"].attributes['VALUE'] != 'n/a'
    error_cnt += 1
    error_msg += "#{mod.elements['LABEL'].attributes['VALUE']} - #{mod.elements['LOCATION'].attributes['VALUE']} - #{mod.elements['STATUS'].attributes['VALUE']} - #{mod.elements['CURRENTREADING'].attributes['VALUE']} #{mod.elements['CURRENTREADING'].attributes['UNIT']} (Caution/Critical: #{mod.elements['CAUTION'].attributes['VALUE']}/#{mod.elements['CRITICAL'].attributes['VALUE']})\n" 
    error_summary += "#{mod.elements['LABEL'].attributes['VALUE']}." 
  end
}

doc.root.elements["VRM"].each_element('//MODULE') { |mod|
  if mod.elements["STATUS"].attributes['VALUE'] != 'Ok'
    error_cnt += 1
    error_msg += "#{mod.elements['LABEL'].attributes['VALUE']} - #{mod.elements['STATUS'].attributes['VALUE']}\n" 
    error_summary += "#{mod.elements['LABEL'].attributes['VALUE']}." 
  end
}

doc.root.elements["POWER_SUPPLIES"].each_element('//SUPPLY') { |mod|
  if mod.elements["STATUS"].attributes['VALUE'] != 'Ok'
    error_cnt += 1
    error_msg += "#{mod.elements['LABEL'].attributes['VALUE']} - #{mod.elements['STATUS'].attributes['VALUE']}\n" 
    error_summary += "#{mod.elements['LABEL'].attributes['VALUE']}." 
  end
}

if error_cnt == 0
  puts "OK: 0 problems" 
  rc=0
else
  puts "Critical: #{error_cnt} problems. #{error_summary}" 
  puts error_msg
  rc=2
end

exit rc

Now having that in place, I simply added all of the iLO2 hosts to a hostgroup, and added a service to check that group using the script, and all my fans, power supplies and such are now monitored without any operating-system level overhead.

QS, AppleScript and moving windows

Kevin Way — Mon, 20 Aug 2007 02:53:00 -0400

A 15” laptop has a relatively small display, so I find myself routinely wanting to put a window all the way to one edge or the other, to help maximize the use of this.

I decided that this should be simple, something along the lines of ^⌥⌘← to slam a window to the left edge of my screen and so on.

There were a few hitches, but I did it as follows:

First I made some scripts:

move_left


tell application "Finder" 
    set {disp_x1, disp_y1, disp_x2, disp_y2} to bounds of window of desktop
end tell

tell application "System Events" 
    set frontmostApplication to name of the first process whose frontmost is true

    tell process frontmostApplication
        tell window 1
            set {x1, y1} to position
            set position to {disp_x1, y1}
        end tell
    end tell
end tell

move_right


tell application "Finder" 
    set {disp_x1, disp_y1, disp_x2, disp_y2} to bounds of window of desktop
end tell

tell application "System Events" 
    set frontmostApplication to name of the first process whose frontmost is true

    tell process frontmostApplication
        tell window 1
            set {x1, y1} to position
            set {x_off, y_off} to size

            set position to {disp_x2 - x_off, y1}
        end tell
    end tell
end tell

move up


tell application "Finder" 
    set {disp_x1, disp_y1, disp_x2, disp_y2} to bounds of window of desktop
end tell

tell application "System Events" 
    set frontmostApplication to name of the first process whose frontmost is true

    set disp_y1 to disp_y1 + 22

    tell process frontmostApplication
        tell window 1
            set {x1, y1} to position

            set position to {x1, disp_y1}
        end tell
    end tell
end tell

move down


tell application "Finder" 
    set {disp_x1, disp_y1, disp_x2, disp_y2} to bounds of window of desktop
end tell

tell application "System Events" 
    set frontmostApplication to name of the first process whose frontmost is true

    tell process frontmostApplication
        tell window 1
            set {x1, y1} to position
            set {x_off, y_off} to size

            set position to {x1, disp_y2 - y_off}
        end tell
    end tell
end tell

And at the suggestion of Alex L.

move center


tell application "Finder" 
    set {disp_x1, disp_y1, disp_x2, disp_y2} to bounds of window of desktop
end tell

tell application "System Events" 
    set frontmostApplication to name of the first process whose frontmost is true

    set disp_y1 to disp_y1 + 22

    tell process frontmostApplication
        tell window 1
            set {x1, y1} to position
            set {x_off, y_off} to size

            set position to {disp_x1 + (disp_x2 - x_off) / 2, disp_y1 + (disp_y2 - y_off) / 2}
        end tell
    end tell
end tell

I added these to QS hotkeys, and used the same technique to make windows do incremental moves as well.

Credit to the MacNN Forums, who helped me find a useful answer in this thread.

Follow-up: Zachary Cohen wrote a correction: Seems like a cool little article, but you need to make sure that Access for assistive devices is turned on. You do this by going to “System Preferences” then selecting the “Universal Access” from the options. Then click the click the checkbox at the bottom “Enable access for assistive devices” then these scripts should work.

Growl, Mail.app and AppleScript

Kevin Way — Fri, 17 Aug 2007 14:55:00 -0400

I wanted a simple system that would notify me whenever I received an e-mail from somebody “important”, but not for every new message that hit my inbox.

I tried GrowlMail and Mail.Appetizer, but found them both unsatisfactory. I finally decided that what I really wanted was a growl message to appear for messages from selected recipients. Something with the name and the subject of the e-mail, like this:

Fortunately, it turned out to be reasonably simple.

First, I opened up Address Book, and made sure that the important people were all listed in appropriate groups and correct e-mail addresses.

Having done that, I opened Mail.app, and added a rule:

Description: Growl

If any of the following conditions are met:
Sender is member of Group: Clients
Sender is member of Group: Family
Sender is member of Group: InsideSystems

Perform the following actions:
Run AppleScript: ~/Library/Scripts/Applications/Mail/MailScript.scpt

but ho, you say, that script does not exist! Never fear, then I loaded up Script Editor and created MailScript.scpt as follows:


on perform_mail_action(info)

    tell application "Mail" 

        set selectedMessages to |SelectedMessages| of info
        set theRule to |Rule| of info

        repeat with eachMessage in selectedMessages
            set theSubject to subject of eachMessage
            set theSender to sender of eachMessage

            tell application "GrowlHelperApp" 

                set the allNotificationsList to ¬
                    {"New Mail"}

                set the enabledNotificationsList to ¬
                    {"New Mail"}

                register as application ¬
                    "MailScript" all notifications allNotificationsList ¬
                    default notifications enabledNotificationsList ¬
                    icon of application "Mail" 

                notify with name ¬
                    "New Mail" title ¬
                    "New Mail from " & theSender description ¬
                    theSubject application name "MailScript" 

            end tell

        end repeat
    end tell
end perform_mail_action

Now, this was the first time I ever did anything in AppleScript, so it’s quite possible there is a much better way to do all of this. As it stands, I simply glued together some sample code from Growl and some sample code from this hint on the excellent MacOSXHints.com, and it appears to work as desired.

Finally, I went into the growl prefpane, and set the priority and stickiness, so that these messages would stay on my screen even if I happened to be otherwise occupied when they came in.

Voila! Nothing spectacularly complex, but it’s a nice example of what I like about Apple. Everything’s easy to use out of the box, but when I dreamed up a random feature, I was able to implement it in a short period of time, despite having never even used their glue language before.

Hopefully somebody else would like this feature too!

PostgreSQL scaling on 6.2 and 7.0

Kevin Way — Wed, 11 Apr 2007 18:40:00 -0400

My previous posts have documented the utterly insane lack of scalability of PostgreSQL (at least according to that particular metric) on FreeBSD 6.2.

I ran the same test, on the same machine, using 7.0 with the 4BSD and the ULE schedulers. Postgres was tuned as in the previous posts, and WITNESS, INVARIANTS and all malloc debugging options were off.

Results:

Hopefully nobody will extrapolate this to mean more than it really does (it’s just one test, but it’s one that happens to matter to me.)

More is better?

Kevin Way — Wed, 11 Apr 2007 03:55:00 -0400

Not always. I did some more tests, and confirmed that this particular workload actually gets slower as you add more than 2 cores.


sysbench --num-threads=${i} --test=oltp --pgsql-user=bench --pgsql-db=bench --db-driver=pgsql --max-time=60 --max-requests=0 --oltp-read-only=on run

Here’s the results, with the tests run on absolutely identical hardware. (HP DL360, 2.00 GHz E5335 Xeon, 10K RPM SAS drives, 10 GB of RAM)

I know 6.2 isn’t the greatest for SMP, but I was still surprised to learn that we could get significantly lower overall performance by adding more processors.

I tried to test 7.0, but the 7.0 installer didn’t seem to recognize the iLO virtual keyboard tonight. Maybe tomorrow will go better.

What did I do wrong?

Kevin Way — Mon, 09 Apr 2007 21:09:00 -0400

I’m hoping that somebody will look at this and say “hey, dummy, you did this all wrong.” I’m hoping that, because the results are shocking.

There have been a buzz about FreeBSD’s SMP performance compared to Linux’s, particularly a particular 8-core SMP test, using FreeBSD 7.0, sysbench, and mysql. A number of people mentioned that there were known problems with this test and PostgreSQL, so I decided to see how bad the problems were.

The results are so shocking, that my first thought was that I must have done something wrong.

The hardware:

Machine 1:
HP DL360 G5
1 5140 (dual core) 2.33 GHz (1333 FSB) Xeon
4 GB RAM
2 15K RPM SAS hard drives, running as mirrors through an e400i drive array

Machine 2:
HP DL360 G5
2 E5335 (quad core) 2.33 GHz (1333 FSB) Xeons
10 GB RAM
2 10K RPM SAS hard drives, running as mirrors through an e400i drive array

Operating System: Both machines are running FreeBSD 6.2-p3. After all, I wanted to see what performance levels to expect with production-quality software, not the 7.0 sweetness.

Kernel: Both are running identically configured SMP-enabled amd64 kernels.

sysctl.conf: both have the following


kern.ipc.shmmax=2147483647
kern.ipc.shmall=524288
kern.ipc.semmsl=512
kern.ipc.semmap=256
kern.ipc.somaxconn=2048
kern.maxfiles=65536
vfs.read_max=32

loader.conf: both have the following


kern.ipc.semmni=256
kern.ipc.semmns=2048

postgresql.conf: both have the following settings changed


shared_buffers = 1GB                 
work_mem = 64MB
maintenance_work_mem = 32MB
max_fsm_pages = 204800   
random_page_cost = 3.0
effective_cache_size = 512MB
update_process_title = off

Both drive arrays are partitioned identically, and postgres was installed as the only running application in each case.

The test was run as follows:


sysbench --num-threads=${i} --test=oltp --pgsql-user=bench --pgsql-db=bench --db-driver=pgsql --max-time=120 --max-requests=0 --oltp-read-only=on run

And yet here are the results:

Yes, the 8-core, 10GB machine is massively slower than the 2-core 4GB machine.

This can’t be right, can it?

Followup: I ran this single-cpu test:


openssl speed rsa

1×2-Core 2.33GHz 5140 Xeon


                  sign    verify    sign/s verify/s
rsa  512 bits   0.0003s   0.0000s   3905.9  45028.8
rsa 1024 bits   0.0009s   0.0001s   1164.6  18787.2
rsa 2048 bits   0.0049s   0.0002s    205.2   6332.2
rsa 4096 bits   0.0320s   0.0005s     31.2   1917.9

2×4-Core 2.33 GHz E5335 Xeon


rsa  512 bits   0.0003s   0.0000s   3381.1  38972.0
rsa 1024 bits   0.0010s   0.0001s   1003.5  15991.7
rsa 2048 bits   0.0057s   0.0002s    176.7   5481.7
rsa 4096 bits   0.0373s   0.0006s     26.8   1638.6

I wanted to see if this was simply one algorithm, or if other algorithms showed the same sort of slowdown, so I ran the following:


openssl speed md5 sha1 aes-256-cbc

The 2-core machine:


The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              14172.75k    45816.42k   114118.73k   182293.03k   220175.36k
sha1             14901.45k    42795.69k    97387.27k   143547.69k   166264.67k
aes-256 cbc      91518.54k    94159.65k    93817.92k    92696.25k    93907.24k

The 8-core machine:


type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              12305.58k    39149.96k    97745.88k   155771.63k   188252.02k
sha1             12740.06k    36648.90k    83318.76k   122733.05k   142119.67k
aes-256 cbc      78109.56k    80589.95k    81222.08k    81406.47k    81401.94k

The 8-core is a little slower than the 2-core again, but not by the enormous levels we see on the pgsql sysbench.

When running RSA across multiple threads, the results are closer to what one would expect. The command I used to do this:


openssl speed rsa -multi N

And now I’m left with the question: Did I screw something up, or is this what Kris Kennaway was talking about when he wrote this message?

I must admit, that while I expected the performance to fall off, it never occurred to me that an 8-core system could be slower than a 2-core system right out of the gate.

Unscientific 15K v 10K SAS Drive Comparison

Kevin Way — Mon, 09 Apr 2007 16:50:00 -0400

We’ve been buying a lot of new machines today, and almost all of them come with SAS drives. We bought 10K SAS drives for obviously performance-neutral applications, and 15K SAS drives for performance-critical applications, but what exactly did we get for an extra $200/drive?

I decided to do some wildly unscientific tests, using two HP DL360s, one with mirrored 10K RPM drives, the other with mirrored 15K RPM drives, both running FreeBSD 6.2/AMD64. Both machines are equipped with P400i drive controllers.

10K SAS drives


# diskinfo -t /dev/da0
/dev/da0
        512             # sectorsize
        73372631040     # mediasize in bytes (68G)
        143305920       # mediasize in sectors
        17562           # Cylinders according to firmware.
        255             # Heads according to firmware.
        32              # Sectors according to firmware.

Seek times:
        Full stroke:      250 iter in   1.016654 sec =    4.067 msec
        Half stroke:      250 iter in   0.955639 sec =    3.823 msec
        Quarter stroke:   500 iter in   1.960625 sec =    3.921 msec
        Short forward:    400 iter in   1.444117 sec =    3.610 msec
        Short backward:   400 iter in   1.457452 sec =    3.644 msec
        Seq outer:       2048 iter in   2.067837 sec =    1.010 msec
        Seq inner:       2048 iter in   2.069040 sec =    1.010 msec
Transfer rates:
        outside:       102400 kbytes in   2.078970 sec =    49255 kbytes/sec
        middle:        102400 kbytes in   2.598712 sec =    39404 kbytes/sec
        inside:        102400 kbytes in   3.115841 sec =    32864 kbytes/sec

15K SAS Drives


#  diskinfo -t /dev/da0
/dev/da0
        512             # sectorsize
        73372631040     # mediasize in bytes (68G)
        143305920       # mediasize in sectors
        17562           # Cylinders according to firmware.
        255             # Heads according to firmware.
        32              # Sectors according to firmware.

Seek times:
        Full stroke:      250 iter in   0.723955 sec =    2.896 msec
        Half stroke:      250 iter in   0.756001 sec =    3.024 msec
        Quarter stroke:   500 iter in   1.255116 sec =    2.510 msec
        Short forward:    400 iter in   0.680525 sec =    1.701 msec
        Short backward:   400 iter in   0.878597 sec =    2.196 msec
        Seq outer:       2048 iter in   2.055020 sec =    1.003 msec
        Seq inner:       2048 iter in   2.062749 sec =    1.007 msec
Transfer rates:
        outside:       102400 kbytes in   1.331560 sec =    76902 kbytes/sec
        middle:        102400 kbytes in   1.848040 sec =    55410 kbytes/sec
        inside:        102400 kbytes in   1.652754 sec =    61957 kbytes/sec

So far, so good. As expected, the results are faster for all the tasks that require seeking, and sequential transfer rates are considerably higher as well. I think I like these tiny lil drives.

Now let’s try some bonnie, with some big files. Let’s try bonnie++ -d /usr/home -u root -s 16g -n 256:65536:65536:16

10K SAS mirror


Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
anemone.insides 16G   379  99 49303  13 14526   9   621  98 30751   7 423.1  94
Latency             22409us     394ms    1414ms   37000us     142ms     121ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
anemone.insidesyste -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 256:65536:65536/16   540  14   661  47 32500  63   588  15   117   8  8358  64
Latency              3045ms     175ms     506ms    2631ms    1841ms     946ms
1.93c,1.93c,anemone.insidesystems.net,1,1176110507,16G,,379,99,49303,13,14526,9,621,98,30751,7,423.1,94,256,65536,65536,,16,540,14,661,47,32500,63,588,15,117,8,8358,64,22409us,394ms,1414ms,37000us,142ms,121ms,3045ms,175ms,506ms,2631ms,1841ms,946ms

15k SAS mirror


Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
noname.insidesy 16G   444  99 77857  19 22512   6   613  94 85113  15 490.1  15
Latency             20163us    2910ms     296ms     274ms     114ms   51627us
Version 1.93c       ------Sequential Create------ --------Random Create--------
noname.insidesystem -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 256:65536:65536/16  1013  19   845  13 44078  86   959  18   103   3 13215  94
Latency              2763ms     800ms   57543us    2798ms    1805ms     259ms
1.93c,1.93c,noname.insidesystems.net,1,1176153578,16G,,444,99,77857,19,22512,6,613,94,85113,15,490.1,15,256,65536,65536,,16,1013,19,845,13,44078,86,959,18,103,3,13215,94,20163us,2910ms,296ms,274ms,114ms,51627us,2763ms,800ms,57543us,2798ms,1805ms,259ms

And with that, I was reasonably satisfied that we weren’t wasting money when we bought the 15K SAS drives.

DELETE FROM important WHERE overly broad condition;

Kevin Way — Tue, 13 Mar 2007 23:20:00 -0400

A client had an interesting problem. They had a critical PostgreSQL database, and they accidentally ran an overly broad delete on a table.

And their newest backup was two weeks old.

Could we help? Of course we could help.

The procedure used was as follows:

1) Stop the database.

2) Backup the database files to a secure location.

3) Make a second copy of the database files, to a new, empty postgresql instance.

4) Now if you have a dump prior to the error, and xlogs for the entire time period, you can do a Point-In-Time Recovery, with a properly configured recovery.conf. Unfortunately we didn’t have that. We had a two-week old dump and 6 hours of xlogs.

5) Try to determine an approximate transaction ID of the mistake. We did this by looking at timestamps of files on the newest pg_clog file, finding one that happened before the mistake, and then multiplying it’s name by 1048576.

6) Then we used that xid with pg_resetxlog, on the secondary database that we had created. It worked, and we had a view of the data. We then played a bit, to find an xid that was acceptably close to the point of failure.

7) From there, we just did a straight SQL COPY, to get the data from the db server we had just setup onto the original database, and to get it loaded.

It wasn’t the most elegant solution, but it worked beautifully, and the client was pleased.

Here’s to hoping you never need this information!

FreeBSD, PHP and Pecl

Kevin Way — Tue, 27 Feb 2007 21:55:00 -0500

I was helping move a legacy PHP app to a new server, and found it needed a PECL extension (dbx).

After installing PEAR, I got the following error:


[root@machine php]# pecl install something

Fatal error: Call to undefined function preg_match() in /usr/local/share/pear/PEAR/Frontend/CLI.php on line 70

It turns out that the problem is that within FreeBSD’s installation of PECL, it calls PHP from the command line with the -n flag. a.k.a. —no-php-ini

This prevents /usr/local/etc/php/extensions.ini from getting loaded, and thus there is no PCRE support.

You can work around it by removing “-n” from inside the pecl script.