Christian's blog

Latest SLES10/ESX-3.5 RPMS

Christian — Sun, 04 Jan 2009 22:14:11 +0000

Disclaimer: I don’t take any responsibility for faults within the software, I just provide the RPM’s! Feel free to ask me about stuff concerning these RPM’s, but I ain’t accountable if your stuff goes kaboom …

Well, even though Jan said that I don’t need to be doing this, at least some RPMS I need to build myself, since

The openSUSE build service isn’t always as fast as you’d wish
some of my RPMS contain fixes/patches from other sources

So, here we go (again):

VMware ESX 3.5 (binary compatible to CentOS 3.9/RHEL 3.9):

htop-0.8.1-1.0 (i386, SRPM)
nagios-plugins-1.4.13-1.0 (i386, SRPM)
- nagios-plugins-setuid-1.4.13-1.0 (i386)
perl-Config-Tiny-2.12-1.0 (noarch, SRPM)
perl-Math-Calc-Units-1.06-1.0 (noarch, SRPM)
perl-Nagios-Plugin-0.31-1.0 (noarch, SRPM)

SUSE Linux Enterprise Server 10:

dstat-0.6.8-1.0 (SRPM)
iotop-0.2.1-1.0 (noarch, SRPM)
nagios-plugins-check_diskio-2.2.0-1 (noarch, SRPM)
nagios-plugins-check_lm_sensors-3.1.0-1.0 (noarch, SRPM)
nagios-plugins-check_nagios_latency-0.9.3-1.0 (noarch, SRPM)
perl-Config-Tiny-2.12-1.0 (noarch, SRPM)
perl-Math-Calc-Units-1.06-1.0 (noarch, SRPM)
perl-Nagios-Plugin-0.31-1.0 (noarch, SRPM)

If you are looking for older versions, try the respective directory on distributions.barfoo.org (for example i386 for ESX 3.5 or x86_64 for SUSE Linux Enterprise Server 10).

New Theme

Christian — Sun, 04 Jan 2009 17:47:48 +0000

Well, since it’s yet another new year (oh, Happy new year! by the way ..), I decided to drop the old crybook and switch over to iNove, which I think looks quite good with a few additions.

If anyone finds something that ain’t working or looks just wrong, tell me please.

More MD weirdness

Christian — Mon, 29 Dec 2008 17:55:17 +0000

Well, at last I’m getting somewhere with my troubles. This only seems to be happening when creating an RAID5 multiple device with four disks, this doesn’t happen with three.

Now, the next thing I tried was to create a three disk array, and then adding the fourth disk as spare and then extending the array with that fourth disk. After that, all these errors seem to appear again *yuck* So I either possess rather faulty disks, or something else is fishy, since I’m having another four disk RAID5 array with the old disks …

MD (Multiple Devices) weirdness

Christian — Mon, 29 Dec 2008 17:15:20 +0000

Well, I don’t think my problem has anything to do with the DawiControl card anymore. I did a little experiment today. I created a 1TiB EXT3 file system on a single drive (one of the new 1TiB drives obviously) and started syncing data over to it (roughly 800MiB).

Now, then I unmounted the drive(s), ran fsck -C -f /dev/sd${deviceletter}1 and it went through without any trouble. Then I removed the partition and created a 1GiB partition on each drive, which I then used to build a new device mapper RAID5 array (with EXT3 on top …).

And guess what happened after I copied the data over, unmounted the file system and ran fsck ? Sure, same thing as yesterday. Now, this means either it’s a mdadm bug, while creating the array or really MD’s fault (which I can rule out, since the same happens on 2.6.25 as well as on 2.6.2 … *shrug*

SIL 3114 barfing

Christian — Sun, 28 Dec 2008 10:16:24 +0000

Well, after I had so much trouble with the USB converter (which isn’t really suited for Linux), I went ahead and bought a DawiControl DC-154 (which is using a SIL3114) controller to migrate my stuff.

After fucking up the new RAID array with the 1TB disks on the old controller (luckily I had the old hard disks still lying around, which still contained the RAID array), I plugged the 1TB disks onto the new controller and started building the array. So after 760 minutes (that’s nearly 13 hours) of synchronizing the newly created array, I was finally able to create the file system — that should be without trouble, right ?

Well, yeah … it was … So I started putting the data on the newly created array (using rsync). Only problem: something seems to be corrupting data (as in EXT3 is barfing up a lot of file system errors).

Dec 28 08:47:21 epimetheus [67092.652866] EXT3-fs: mounted filesystem with ordered data mode.
Dec 28 09:53:20 epimetheus [71058.253027] EXT3-fs error (device md2): ext3_add_entry: bad entry
 in directory #23371810: directory entry across blocks - offset=260, inode=18964552,
 rec_len=26988, name_len=115
Dec 28 09:53:20 epimetheus [71058.305558] EXT3-fs error (device md2): ext3_add_entry: bad entry
 in directory #23371810: directory entry across blocks - offset=260, inode=18964552,
 rec_len=26988, name_len=115

(fsck.ext3 is returning much, much more ..)

After putting the blame on EXT3, I tried out reiserfs (yeah, yeah I know .. baaaad idea). Well, at first it didn’t put out any errors, but running fsck.reiserfs turned up errors that looked a lot like the ones fsck.ext3 returned.

Then, I started looking at the array size (since I was curious), and it said the new array on four 1TB disks is ~760GB. Now according to my improper math, using 4* 1000GB drives the total usable amout of disk space should be something like 2793.96GB, and not ~760GB. *shrug*

I’m out of idea’s right now, and I’m gonna wait till January till I do anything else.

USB weirdness

Christian — Tue, 23 Dec 2008 20:50:43 +0000

Well, I was at work for a brief moment, where I grabbed me one of our SATA->USB bridges, since I need to migrate some (~750GB) data of the old raid-array and onto a new one. The troublesome about that is simply, that the current RAID controller only supports four attached devices, that’s why I do have to use something like this … Sure I could have bought a new RAID controller, but why spend 45+ EUR on something, that you can solve differently ?

Well, after figuring that I need to change my kernel config yet again (didn’t have USB support till Tue Dec 23 ~16:45:00 CET 200 I attached the adapter to two adjacent USB ports. And shortly after copying 4-10MB, the transfer would result in a read-only EXT3 file system with something like this in the syslog:

Dec 23 17:23:57 epimetheus usb 1-2: new high speed USB device using ehci_hcd and address 2
Dec 23 17:23:57 epimetheus usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
Dec 23 17:23:57 epimetheus usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Dec 23 17:23:57 epimetheus usb usb3: Product: OHCI Host Controller
Dec 23 17:23:57 epimetheus usb usb3: Manufacturer: Linux 2.6.25-hardened-r11 ohci_hcd
Dec 23 17:23:57 epimetheus usb usb3: SerialNumber: 0000:00:02.1
Dec 23 17:23:57 epimetheus Initializing USB Mass Storage driver...
Dec 23 17:23:57 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:57 epimetheus usb 1-2: device descriptor read/64, error -71
Dec 23 17:23:58 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:58 epimetheus usb 1-2: device descriptor read/64, error -71
Dec 23 17:23:58 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:58 epimetheus usb 1-2: device not accepting address 2, error -71
Dec 23 17:23:59 epimetheus usb 1-2: reset high speed USB device using ehci_hcd and address 2
Dec 23 17:23:59 epimetheus usb 1-2: device not accepting address 2, error -71
Dec 23 17:23:59 epimetheus usb 1-2: USB disconnect, address 2
Dec 23 17:23:59 epimetheus sd 4:0:0:0: Device offlined - not ready after error recovery
Dec 23 17:23:59 epimetheus sd 4:0:0:0: [sde] Result: hostbyte=0x01 driverbyte=0x00
Dec 23 17:23:59 epimetheus end_request: I/O error, dev sde, sector 1229380295
Dec 23 17:23:59 epimetheus end_request: I/O error, dev sde, sector 1229380535
Dec 23 17:23:59 epimetheus Aborting journal on device sde1.
Dec 23 17:23:59 epimetheus ext3_abort called.
Dec 23 17:23:59 epimetheus EXT3-fs error (device sde1): ext3_journal_start_sb: Detected aborted journal
Dec 23 17:23:59 epimetheus Remounting filesystem read-only
Dec 23 17:23:59 epimetheus EXT3-fs error (device sde1) in ext3_ordered_writepage: IO failure
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_frozen_data
Dec 23 17:23:59 epimetheus __journal_remove_journal_head: freeing b_committed_data

Well, now what ? I googled a for a bit, apparently this happens when EHCI tries to write to the device and gets a timeout, cause the device is rather slow — or whatever (or the device drops down to USB 1.1). So, after disabling EHCI, the transfer has been running for about three hours now, and roughly only 1/12 of the data transferred to the external disk. Only trouble with that is, that even USB 1.1 is kinda slow to transfer 750GiB

Followup: Well, due to USB 1.1 being slow as a snail, I went surfing for alternatives using Windows (since I know that the bridge does full USB 2.0 with Windows without any troubles). And guess what I found ?
There’s an EXT2/3 device driver for Windows XP, yay! So I’m copying with full 100Mbit speed right now *shrug*

Short vacation

Christian — Tue, 02 Dec 2008 22:07:06 +0000

Well, Arne recently (not really recently though .. ) complained about my blog being waaay to technical, so I ended up writing this lil’ anecdote.

I’m finally on my long awaited, the remaining year lasting vacation. Last week was interrupted by a short job interview in Nuremberg, and also by the flu (not “again“, I still got it in me, haven’t been able to shake it now for about three months).

Today I’m spending the night on Spiekeroog, which is a smallish island in the middle of the North Sea. It’s a real neat island, and if I would have the choice of deciding whether or not to move here for the same salary, I’d probably do it. Spiekeroog certainly does have a certain amount of flair, which I ain’t gonna deny. But it also does have it’s drawbacks.

One thing “neat” about Spiekeroog is, that it’s completely isolated from the CO² pollution. There is not a single car on the island (well, besides emergency services like police and fire/rescue) - only electric cars.

It’s located in the North Sea, so it’s exposed to the raw Atlantic weather of which I got a good taste today. When I was planning the trip a few weeks back, I allotted about 8 hours for the 400km’ish trip to Neuharlingersiel. Due to my driving “skills” ( others would say I do have a rather heavy foot - 170km/h or 105.6331 mph for those not able to deal with the metric system ), I was there about two hours early. So I went eating some crab soup (which was really delicious) and then went down to the harbour, where the hotel boat was gonna pick me up. And then it started raining (as in pouring), with only about 2°C air temperature. So basically I was freezing my bollocks off.

Anyway, after eating dinner worth fifty bucks in the restaurant (well, don’t forget the expensive rose wine - which was a bit sour actually, as well as the three bucks of delicious tea) I’m now all cosy within my bed and all tucked in.

Cheerio for now!

IBM TS7530 and DNS

Christian — Sat, 22 Nov 2008 16:31:50 +0000

Well, we had our TS7530 delivered in late September, the day after the IBM service guys came by to prep the VTL for our needs (IBM sells the thing as black box). Now, since that day; they fought with the Call Home functionality. The trouble was simply, that the Call Home Service running on the Virtualization Engines just didn’t start.

After about 6 weeks of trial and error (and the IBM service guys popping in every second week), they finally found the cause of the Call Home Service not being able to start. Domain Name Resolution. Neither the IP addresses of the VE’s nor the VE console were registered in our DNS/or local host files.

After I walked over to the networking department and had them register them IP addresses, everything is honky donkey.

Nagios and check_ram yet again

Christian — Fri, 14 Nov 2008 17:08:30 +0000

As some people know, I previously “created” (mostly modified the check_swap plug-in to print RAM usage) check_ram in C. Now one of my problems for the past few months was putting the C plug-in as well as “supported” environment under the same hat. Today I had another look at the amount of available plug-ins in NagiosExchange. There are quite a few plug-ins available, but as I do have some experience with Python, I used the one written in Python.

It was rather easy hacking in support for performance data into it, as the below shows. Someone else already posted a non-unified diff for performance data support, but that ain’t quite right according to the Nagios plug-in development guidelines.

--- check_ram.py.orig
+++ check_ram.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 #
 #   Copyright Hari Sekhon 2007
+#   Copyright Christian Heim 2008
 #
 #   This program is free software; you can redistribute it and/or modify
 #   it under the terms of the GNU General Public License as published by
@@ -64,27 +65,30 @@
     total_used_megs = float(memtotal-memfree) / 1024
     total_free_megs = float(total_free) / 1024
     memtotal_megs   = float(memtotal) / 1024
+    
+    total_warning_threshold_megs = round(float(memtotal) * float(warning_threshold) / 100 / 1024)
+    total_critical_threshold_megs = round(float(memtotal) * float(critical_threshold) / 100 / 1024)
 
     if percent == True:
         percentage_free = int( float(total_free) / float(memtotal) * 100 )
         if percentage_free < critical_threshold:
-            print "RAM CRITICAL: %d%% ram free (%d/%d MB used)" % (percentage_free,total_used_megs,memtotal_megs)
+            print "RAM CRITICAL - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return CRITICAL
         elif percentage_free < warning_threshold:
-            print "RAM WARNING: %d%% ram free (%d/%d MB used)" % (percentage_free,total_used_megs,memtotal_megs)
+            print "RAM WARNING - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return WARNING
         else:
-            print "RAM OK: %d%% ram free" % percentage_free
+            print "RAM OK - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return OK
     else:
         if total_free < critical_threshold:
-            print "RAM CRITICAL: %dMB ram free (%d/%d MB used)" % (total_free_megs,total_used_megs,memtotal_megs)
+            print "RAM CRITICAL - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return CRITICAL
         if total_free < warning_threshold:
-            print "RAM WARNING: %dMB ram free (%d/%d MB used)" % (total_free_megs,total_used_megs,memtotal_megs)
+            print "RAM WARNING - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return WARNING
         else:
-            print "RAM OK: %dMB ram free" % (total_free_megs)
+            print "RAM OK - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
 
 
 
             return OK

Now, with that simple plug-in I can go ahead and package the thing separately, without the need to figure out a way to get them nagios-plugins to install some useful includes …

IBM TS7530 engine failover and HBA mode

Christian — Tue, 28 Oct 2008 14:21:06 +0000

Well, when they delivered the VTL about four weeks ago, nobody figured this thing would be such a mess. Apparently IBM hasn’t set up that much VTL’s with engine failover.

Point being, the VE’s have eight HBA ports (four inside, four outside the black box). Now, as they configured the VTL, the ports were all in initiator mode. And we needed the fourth port in target mode as well, as it’s better to have 4 independent paths to the VTL. The only problem was, the VE console didn’t think so.

There is no way in hell you can switch the darn HBA port to the target mode. — Well, IBM just called and told us the solution.

Disolve the Failover group, reconfigure the HBA port and then recreate the Failover group. Tada …..

IBM TS7530 zoning

Christian — Mon, 27 Oct 2008 14:41:59 +0000

At first, as we prepped the zoning for the VTL, we did it WWN-based. Now the trouble with the HBA’s of the VTL is simply that it has different WWPN’s on the same WWN. And WWN-based zoning simply doesn’t allow access to that.

So off we went and switched to Switchport-based zoning, and see. It just works *shrug*

MessPC Ethernetbox 2 and Nagios

Christian — Fri, 24 Oct 2008 19:12:39 +0000

As I talked to Tobi yesterday, we came to talk about our Ethernet Box thermometer. It’s a neat device, which works pretty much out of the box. Integrating it with Nagios is a bit of a bummer.

Ethernet box 2

That’s what the ~300 EUR box looks like. It’s basically a small black box with a RJ45 jack, and four RJ11 jacks for attached external devices. The box itself only functions as a “management station” and doesn’t come with a sensor.
Normally, you can attach up till four RJ11 sensors to it. But, MessPC also has RJ11 port splitters, which enables you to attach up to eight RJ11 sensors to the MessPC.

Thermometer RJ45 jacks

As you can see, the box has a RJ45 jack on the other side, which you basically hook up to your network and then configure an IP address (or if you fancy DHCP for those things, it’s possible too).

Thermometer RJ11 jacks

On the opposite site, are the RJ11 jacks for the sensors. As you can see, we currently do have 4 splitters attachted to the box, enabling up till 8 sensors to be measured.
Once you have it up and running, you can look at the web interface and you’ll be able to see the state of the sensors right on the first page.

Now, if you have a Nagios installation (like I do), you might want to integrate the Ethernetbox with Nagios.
You do have two choices: either do the checking by means of SNMP but forego graphing by Nagios, or use check_pcmeasure and use the full features of your Nagios installation (read: graphing done by pnp4nagios).
As I recently updated our Nagios installation to 3.0.4, I ran into trouble with the pcmeasure plugin. It simply didn’t run with the embedded perl (which it should, but apparently became a bit stricter between 2.x and 3.x). So I went ahead and changed it, so it would run with embedded perl, as I was in the mood.
After that was done, I simply dumped it into /usr/lib/nagios/plugins on my nagios box,
created a /etc/nagios/objects/commands/etherbox.cfg containing the necessary command definition:

# 'check_ethernetbox' command definition
define command{
        command_name    check_ethernetbox
        command_line    /usr/lib/nagios/plugins/check_pcmeasure.pl -H $HOSTADDRESS$ -S $ARG1$ -W $ARG2$ -C $ARG3$
}

Nagios Service Detail

Now we can go ahead and *really* integrate it into Nagios, by simply creating a host utilizing this command definition.

define host {
  use                   generic-network
  host_name             ethernetbox
  alias                 ethernetbox.home.barfoo.org
  address               10.0.0.9
  parents               home-0
}
 
define service {
  use                   generic-service
  host_name             ethernetbox
  service_description   1.1: Aussentemperatur
  check_command         check_ethernetbox!com1.1!29.0!35.0!
  action_url            /pnp/index.php?host=$HOSTNAME$&amp;srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}
 
define service {
  use                   generic-service
  host_name             ethernetbox
  service_description   1.2: Druckfussboden
  check_command         check_ethernetbox!com1.2!16.0!19.0!
  action_url            /pnp/index.php?host=$HOSTNAME$&amp;srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

Adapter bonding on Linux

Christian — Thu, 16 Oct 2008 16:47:23 +0000

Well, today I had a rather weird error. I was testing the adapter bonding on one of the boxen designated as Tivoli Storage Manager Server, when I noticed that the bonding wasn’t working as expected when simulating an error (that is unplugging one of the TP cables for the bond).

Now, the bond had “mode=6 miimon=100″ as options. After running “linux bond debug” through Google (which turned up nothing useful, besides one document on the Oracle Wiki about IOS/Linux adapter teaming), I figured “Hey, just lets test switching the arguments.” And guess what ?

Afterwards, it just works when you unplug one of the cables of the bond, while it didn’t work before … *shrug*

Windows Server 2003 Terminal services

Christian — Wed, 15 Oct 2008 12:11:18 +0000

Well, once you thought you don’t have any more problems, another one just pops up. I’m currently bashing my head against the wall, why the hell the forwarded (or is it redirected ?) drives are not shown in the in the “My Computer” explorer view. I pretty sure have an idea why (basically, HKEY_CURRENT_USERS\Software\Classes isn’t writeable, but that’s where Windows, or rather the Terminal Services — or whatever is creating the associations), just don’t know a clever way around/by it.

It’s basically a dead end. The user has no access to that particular subkey, and I can’t change the permissions by changing it in ntuser.dat apparently. Neither do the inherited permissions apply, so I’m basically stuck.

Linux-HA and Tivoli Storage Manager (Finito!)

Christian — Sun, 05 Oct 2008 15:40:42 +0000

As I previously said, I was writing my own OCF resource agent for IBM’s Tivoli Storage Manager Server. And I just finished it yesterday evening (it took me about two hours to write this post).

Trac revision log (shortened)

Only took me about four work days (that is roughly four hours each, which weren’t recorded in that subversion repository) plus most of this week at home (which is 10 hours a day) and about one hundred subversion revisions. The good part about it is, that it actually just works (I was amazed on how good actually). Now you’re gonna say, “but Christian, why didn’t you use the included Init-Script and just fix it up, so it is actually compilant to the LSB Standard ?”

The answer is rather simple: Yeah I could have done that, but you also know that wouldn’t have been fun. Life is all about learning, and learn something I did (even if I hit the head against the wall from time to time during those few days) … There’s still one or two things I might want to add/change in the future (that is maybe next week), like

adding support for monitor depth by querying the dsmserv instance via dsmadmc (if you read through the resource agent, I already use it for the shutdown/pre-shutdown stuff)
I still have to properly test it (like Alan Robertson mentioned in his one hour thirty talk on Linux-HA 2.0 and on his slides, Page 100-102) in a pre-production environment
I’m probably configure the IBM RSA to act as a stonith device (shoot the other node in the head) - just for the case one of them ever gets stuck in a case, where the box is still up, but doesn’t react to any requests anymore

So here’s a small (well the picture is rather big duuh) screen capture of a grouped resource (basically, IP + Filesystem mounts + dsmserv instance) migrating to a different physical host (both share some shared storage in this example - I used two VM’s).

Showing the VM configuration

First the VM configuration. It looks a bit messy, as I needed two database volumes for Tivoli, two recovery log volumes for Tivoli, one Active Data storage pool for Tivoli and one volume for the configurations of the Tivoli Storage Manager server itself. I could have done it with only 5 disks each. But I wanted to test how Linux-HA handles the migration with resources, as well as I wanted to test how Tivoli is handling it.

Now, the first hard disk is where the system (and all the other stuff, like heartbeat …) is installed. It got two network cards, one for cluster link and the other for the serviceing to the public network. It also got a second controller, simply due to the nature VMware handles disk sharing. This controller is set to SCSI Bus Sharing = Physical (or scsi?.sharedBus = “physical” if you rather want the vmx term), to allow it to share hard disks with other virtual machines.

I then started installing stuff (needed heartbeat-2.1.3, heartbeat-pils-2.1.3 and heartbeat-stonith-2.1.3 - which sadly pulls in GNOME due to “hb_gui“), imported the configurations (they are in my earlier post about Setting up Linux-HA) like this:

cibadmin -o resources -C -x resources.xml
cibadmin -o constraints -C -x constraints.xml

Showing the initial state of the cluster resources

And that’s basically it. The cluster is up and running (even if I set the dsmserv to stopped for the time being since I wasn’t sure whether or not the resource agent would work). Now, after we’re done with setting up the cluster, it basically looks as showed on the picture.

The good part about the constraints is though, they are the “good kind” of resource constraints. They basically say “run the resource group tsm1_group on the host tsm1, but if you really have to, starting it on the host tsm2 is also ok”. If you looked closely at my resources.xml, you might have noticed that I specified name=”target_role” value=”stopped” for each dsmserv resource.

Well I did this on purpose, since I didn’t knew whether or not the resource agent is actually gonna work. After I knew (I played with it for a while, remember ?), I started the resource, by issuing this:

cibadmin -o resources -M -X ''
cibadmin -o resources -M -X ''

Now guess, what happens if you start rebooting one box ? Exactly. Linux-HA figures after 30 seconds that the box is dead (which it is, at least from the service’s point of view) and simply starts it on the still “living” cluster node. Now, lets play with those resource groups. By simply issuing

crm_resource -M -r tsm1_group -H tsm2

Failing over a resource is pretty simple

we’re gonna tell Linux-HA to migrate the resource group to the second host, and keep it pinned there. As you can see on the left, it actually takes a bit until Linux-HA is starting to enforce the policy, which crm_resource just created for us.

Linux-HA really does shut down the resource group in order

After a brief moment (that is about thirty seconds - due to the ocf resource agent sleeping for 2*5s), the resources are shutting down on the first host.

Now, after thinking about it, I could have done this differently. As you can see, Linux-HA is first shutting down the whole resource group on one host, before it starts bringing any resources (because it treats the resource group as a single resource itself). This implies a longer outage than if I would have done it with before/after constraints. I still might do that, but for now the resource group is ok.

After a short break in service availibility the resource is back online

As you can see, about fifty seconds after I ordered Linux-HA to migrate the resource group tsm1_group to the second host, it finally enforced policy and the resources are up and running again. Now, sadly Linux-HA doesn’t do something like VMotion does (copying the memory map and stuff), which would safe us the trouble of doing the shutdown in the first place. I know openVZ does support that kind of stuff, but I guess that’s just day-dreaming for now. Would make one hell of a solution

Oh yeah, and if you’re as much annoyed as me of heartbeat (or rather logd) polluting your messages file, and you are lucky beggar and are running syslog-ng, simple put this into your /etc/syslog-ng/syslog-ng.conf (or rather /etc/syslog-ng/syslog-ng.conf.in on SLES10 and running SuSEconfig –module syslog-ng):

# The main filter for all Heartbeat related programs
filter f_ha         { program("^Filesystem$")
                      or program("^IPaddr2$")
                      or program("^dsmserv$")
                      or program("^attrd$")
                      or program("^attrd_updater$")
                      or program("^ccm$")
                      or program("^ccm_tool$")
                      or program("^cib$")
                      or program("^cibadmin$")
                      or program("^cluster$")
                      or program("^cl_status$")
                      or program("^crmadmin$")
                      or program("^crmd$")
                      or program("^crm_attribute$")
                      or program("^crm_diff$")
                      or program("^crm_failcount$")
                      or program("^crm_master$")
                      or program("^crm_resource$")
                      or program("^crm_standby$")
                      or program("^crm_uuid$")
                      or program("^crm_verify$")
                      or program("^haclient$")
                      or program("^heartbeat$")
                      or program("^ipfail$")
                      or program("^logd$")
                      or program("^lrmd$")
                      or program("^mgmtd$")
                      or program("^pengine$")
                      or program("^stonithd$")
                      or program("^tengine$");
                    };
 
filter f_ha_debug   { level(debug);     };
filter f_ha_main    { not level(debug); };
 
#
# heartbeat messages in separate file:
#
destination ha_main   { file("/var/log/heartbeat/ha-log"   owner(root) group(root)
                        create_dirs(yes) dir_group(root) dir_owner(root) dir_perm(0700)); };
 
destination ha_debug  { file("/var/log/heartbeat/ha-debug" owner(root) group(root)
                        create_dirs(yes) dir_group(root) dir_owner(root) dir_perm(0700)); };
 
log { source(src); filter(f_ha); filter(f_ha_main); destination(ha_main); };
log { source(src); filter(f_ha); filter(f_ha_debug); destination(ha_debug); };

That should get you a directory in /var/log named heartbeat with two logfiles, the main log “ha-log” which holds the important stuff and the debug log “ha-debug“. You also should put a logrotate rule somewhere, as those two files grow big rather fast.