Additionally, the OID’s for those three failure elements seem to be dependent on the firmware version, as with 6.3.x they appear as different OIDs. So I may need to introduce another command-line switch, which selects the firmware version and depending on that, the OID.

Even despite those problems I just described, I ended up using the plugin to watch our SAN infrastructure. I even wrote a simple pnp4nagios template, so all the data would show up in a single graph and not a graph per data source.

Graph: 4 Hours

Only trouble I had with that script, is that it crams every single port into one result. As I wanted something, that a) could watch a single port and b) return performance data, I went ahead an used the script to do a basic rewrite. But after a short while, I grew antsy and started writing a script from scratch, using the OIDs I got from that script and a Cacti template.

So far, I got a good plugin, but it’s still lacking a few things:

1. Support for warning/critical thresholds for each error category
2. Sadly the important errors (er_link_fail, er_loss_sync and er_loss_sig) are kept in a separate table structure (swEndDeviceRlsEntry), which I can’t seem to access right now; even though the entries are mandatory and according to the MIB should be at least read-only.
3. The plugin isn’t doing a proper $session->close(); . After moving the snmp stuff into a subroutine, Perl refuses to do the session closing. Don’t know why right now.

Right now, the plugin supports two modes. The first just checks if the port is operational and in sync and the second checks the port status, but also returns the performance data.

Only do a basic check if the Port is in operational status

./check_snmp_brocade_fcport.pl -H 10.0.0.50 -C public -P 2 -N
SNMP_BROCADE_FCPORT OK - FC port 0/2's swFCPortPhyState is inSync

Check the port status, but also return performance data

./check_snmp_brocade_fcport.pl -H 10.0.0.50 -C public -P 2
SNMP_BROCADE_FCPORT OK - FC port 0/2's swFCPortPhyState is inSync|stat_wtx=577976968;0;0;0;0 stat_wrx=4069984468;0;0;0;0 stat_ftx=422378205;0;0;0;0 stat_frx=123789748;0;0;0;0 er_enc_in=0;0;0;0;0 er_crc=0;0;0;0;0 er_trunc=0;0;0;0;0 er_toolong=0;0;0;0;0 er_bad_eof=0;0;0;0;0 er_enc_out=0;0;0;0;0 er_c3_timeout=0;0;0;0;0

That might look like much, but Nagios is gonna pass everything after “|” to your performance data command.

List of OIDs, which hold the various information:

   swFCPort
   PhyState: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.3.
   OpStatus: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.4.
  AdmStatus: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.5.
  LinkState: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.6.
    TxWords: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.11. (stat_wtx)
    RxWords: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.12. (stat_wrx)
   TxFrames: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.13. (stat_ftx)
   RxFrames: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.14. (stat_frx)
 RxEncInFrs: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.21. (er_enc_in)
     RxCrcs: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.22. (er_crc)
   RxTruncs: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.23. (er_trunc)
 RxTooLongs: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.24. (er_toolong)
  RxBadEofs: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.25. (er_bad_eof)
RxEncOutFrs: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.26. (er_enc_out)
 C3Discards: .1.3.6.1.4.1.1588.2.1.1.1.6.2.1.28. (er_c3_timeout)

swEndDevice
LinkFailure: .1.3.6.1.4.1.1588.2.1.1.1.21.1.1.4. (er_link_fail)
   SyncLoss: .1.3.6.1.4.1.1588.2.1.1.1.21.1.1.5. (er_loss_sync)
    SigLoss: .1.3.6.1.4.1.1588.2.1.1.1.21.1.1.6. (er_loss_sig)

The last three OIDs, as well as the ones in FCMGMT-MIB (as I mentioned in the TODO), sadly don’t exist (or I’m doing something wrong ? — no clue right now), so I can’t incorporate them into the script at this time.

However, I found something in a separate OID-tree (also the FCMGMT-MIB), which seems to be exactly what I’m looking for.

connUnitPortStatCount
   LossofSynchronization: .1.3.6.1.3.94.4.5.1.44.16.0.0.5.30.52.240.185.0.0.0.0.0.0.0.0.
     LossofSignal: .1.3.6.1.3.94.4.5.1.43.16.0.0.5.30.52.240.185.0.0.0.0.0.0.0.0.
   LinkFailures: .1.3.6.1.3.94.4.5.1.44.16.0.0.5.30.52.240.185.0.0.0.0.0.0.0.0.

Only trouble with those OIDs is, that they are OCTET STRING’s, which right now just return crap (either nothing or just a new-line) with my script. Gonna have to work on that.

If you’re interested in the Perl script (for now, lacking some options, performance data, $session->close();), you’ll find it here.

nobody ALL = NOPASSWD: /usr/bin/zypper sl, /usr/bin/zypper --non-interactive --no-gpg-checks --terse list-updates

(Keep in mind this needs to be a single line …)

Praxisbuch Nagios

He actually mentions it in the credits at the beginning of the book. Turns out it is an animal, a sea pen or sea feather (I’m guessing at Pennatula aculeata).

Now as for the content of the book itself, I do have to admit that I haven’t read the whole book. I just picked a few topics (SNMP-Traps with Nagios, notifications) which I did find rather well written. My (soon ex-) trainee, Michel, however already bugged Tobias about some errors in the book itself, or rather some changes which happened after 3.0.3 (that’s the Nagios version the book is based on).

All in all, I guess I can congratulate Tobias on yet another good written book!

Usually I ended up increasing the cscript timeout, or the nsclient++ socket timeout, but it still kept showing up. Since I rely heavily on my surveillance tools, I have the demand, that as few as possible false positives show up. So I ended up chasing down this error today, and after that I have to say it was quite simple.

In my case, it wasn’t cscript (that timeout is set to 300 seconds), neither nsclient++ (socket timeout is set to 300 seconds too), nor the nrpe plugin itself (that has 300 seconds as well).

As it turns out, Nagios got an additional setting controlling these things, called service_check_timeout which defaults to 60 seconds. Sadly the plugin, or rather Windows needs longer than those 60 seconds to figure out whether or not it needs updating, thus Nagios is killing the plugin and returning a CRITICAL message.

After increasing the value of service_check_timeout that’ll be fixed hopefully.

For Nagios, I dismissed the fans, since the fan speed is only passed on in percent values. So I only added this:

define hostgroup{
  hostgroup_name                  rsa-snmp
  alias                           Remote Supervisor Adapter (allowing SNMP connections)
}

define service{
  use                             generic-perfdata

  check_command                   check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.1.1!45!60!°C!Temperature CPU0!
  hostgroup_name                  rsa-snmp
  service_description             TEMP CPU0
}

define service{
  use                             generic-perfdata

  check_command                   check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.2.2.1!45!60!°C!Temperature CPU1!
  hostgroup_name                  rsa-snmp
  service_description             TEMP CPU1
}

define service{
  use                             generic-perfdata

  check_command                   check_rsa_snmpv1_public!.1.3.6.1.4.1.2.3.51.1.2.1.5.1.0!29!35!°C!Temperature Ambient!
  hostgroup_name                  rsa-snmp
  service_description             TEMP AMBIENT
}

Oh, and if anyone else is curious like me, here’s the list with the OID’s, courtesy of Gerhard Gschlad and Leonardo Calamai.

For the fans:

Fan1: .1.3.6.1.4.1.2.3.51.1.2.3.1.0
Fan2: .1.3.6.1.4.1.2.3.51.1.2.3.2.0
Fan3: .1.3.6.1.4.1.2.3.51.1.2.3.3.0
Fan4: .1.3.6.1.4.1.2.3.51.1.2.3.4.0
Fan5: .1.3.6.1.4.1.2.3.51.1.2.3.5.0
Fan6: .1.3.6.1.4.1.2.3.51.1.2.3.6.0
Fan7: .1.3.6.1.4.1.2.3.51.1.2.3.7.0
Fan8: .1.3.6.1.4.1.2.3.51.1.2.3.8.0
Fan9: .1.3.6.1.4.1.2.3.51.1.2.3.9.0
Fan10: .1.3.6.1.4.1.2.3.51.1.2.3.10.0
Fan11: .1.3.6.1.4.1.2.3.51.1.2.3.11.0
Fan12: .1.3.6.1.4.1.2.3.51.1.2.3.12.0

And for the temperatures:

CPU1: .1.3.6.1.4.1.2.3.51.1.2.1.2.1.1
CPU2: .1.3.6.1.4.1.2.3.51.1.2.1.2.2.1
CPU3: .1.3.6.1.4.1.2.3.51.1.2.1.2.3.1
CPU4: .1.3.6.1.4.1.2.3.51.1.2.1.2.4.1
Ambient: .1.3.6.1.4.1.2.3.51.1.2.1.5.1.0

I just found a proper list of OID’s for the IBM RSA adapter. That’s rather nice, since I really was looking for the OID’s for the VRM failure OID and other warning/critical events.

As you know, all the other plugins accept WARNING and CRITICAL-thresholds based on the calculation, if the return integer is above this threshold it reached WARNING/CRITICAL state. But check_snmp doesn’t play that way.

It expects only ranges, which are NOT gonna result in warning or critical events. Which is kinda stupid, since you gotta rethink twice about the thresholds

define service {
  use                   generic-service
  host_name             ibm-bc1-mgmt
  service_description   Chassis Cooling - Bay 1
  check_command         check_snmpv1_public!.1.3.6.1.4.1.2.3.51.2.2.3.20.0!\
                                            1900:8000!1900:0,10000:8000!\
                                            RPM!Chassis Cooling - Bay 1
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

All in all, another lesson learned

Now, my first tries failed horribly. I brought down a complete cluster group, resulting in a major annoyance. Now, today I went at it a bit smarter I cloned myself two VM’s off my Windows Server 2003 Enterprise R2 template, created a new cluster.

After that, I tried it on the test cluster again, same result. The resource is successfully created, but once I try to take it online, it breaks and moves the whole cluster group to the other node (as cyclic moving between the cluster nodes with no end).

After that, I figured something has to be wrong with the command I’m trying to use, the one as instructed by the NSClient++ wiki. I then tried the command on the command line, but as soon as hitting <TAB> (oooold bash habit ), it completed the path, but put quotes around it … Don’t ask me.

If I try the path without the quotes, no-joy at all. Once you put quotes around it, everything becomes honky-dory and the resource comes online without the slightest trouble!

Hint to self: When creating a NSClient++ cluster resource (or any application resource using a command that needs switches for that matter), use a quoted command line along the lines of this:

"Q:\_nsclient\nsclient.exe" /test

The first mistake I made, was to trust Cisco with their guide about how to get the utilization from the device using SNMP. They stated some OID’s, which I tried with snmpwalk and got a result from.

snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.9.5.1.1.8
.1.3.6.1.4.1.9.5.1.1.8.0 = INTEGER: 0

Now, as I tried retrieving the SNMP data by means of the check_snmp plugin, I got some flaky results:

/usr/lib/nagios/plugins/check_snmp -H 10.0.0.35 -C public \
                                   .1.3.6.1.4.1.9.5.1.1.8
SNMP problem - No data received from host
CMD: /usr/bin/snmpget -t 1 -r 5 -m '' -v 1 [authpriv] 10.0.0.35:161

Those of you, who read the excerpts carefully will notice the difference between snmpwalk and the OID I passed on to check_snmp.

The point being, the OID’s Cisco gave in their Design tech notes are either old, or just not accurate at all. After passing on the .0 to each value given by Cisco, the check_snmp is all honky dory and integrated into Nagios.

As usual, the Nagios definitions are further down, for those interested.

define host {
  use                   generic-network
  host_name             c3012-1
  alias                 c3012-1.home.barfoo.org
  address               10.0.0.35
  hostgroups            network
}

define service {
  use                   generic-service
  host_name             c3012-1
  service_description   SYS: Backplane utilization
  check_command         check_snmpv1_public!.1.3.6.1.4.1.9.5.1.1.8.0!60!80!\
                                            %!"Backplane utilization:"
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             c3012-1
  service_description   SYS: CPU utilization - 5sec
  check_command         check_snmpv1_public!.1.3.6.1.4.1.9.2.1.56.0!130!160!\
                                            %!"CPU utilization (5sec):"
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             c3012-1
  service_description   SYS: CPU utilization - 1min
  check_command         check_snmpv1_public!.1.3.6.1.4.1.9.2.1.57.0!80!95!\
                                            %!"CPU utilization (1min):"
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             c3012-1
  service_description   SYS: CPU utilization - 5min
  check_command         check_snmpv1_public!.1.3.6.1.4.1.9.2.1.58.0!35!60!\
                                            %!"CPU utilization (5min):"
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

For now, the pnp4nagios graphing for the back plane utilization isn’t working (don’t ask me why). Also, it might be a good idea, to combine the CPU utilization commands into one, so you’d get a single graph out of it.

• Fan speeds for Chassis Cooling/Power Module Cooling Bay(s)
• Temperature
• Power Domain utilization

It wasn’t *that* hard to implement. Only trouble(s) I ran into, were (1) IBM did a real shitty job with the MIB’s. If you look closely into the mmblade.mib, you’re gonna notice, that not a single OID is specified for the events. (2) As the MIB’s weren’t documented anywhere, I had to look them up via snmpwalk (which I had never used before). So as a reminder (to myself), here’s how it is done:

snmpwalk -v1 -c public -O n 10.0.0.35 .1.3.6.1.4.1.2.3.51.2.2

This will get you a list, with a lot of output (5154 lines to be exact). Lucky me, the web interface of the management module/ssh interface is rather verbose, so all you need to do is compare those values with what you are looking for.

So for myself (and anyone interested) read ahead for the list of checks we are currently running on the management module.

define command {
  command_name      check_snmpv1
  command_line      /usr/lib/nagios/plugins/check_snmp -C public \
                             -H $HOSTADDRESS$ -o $ARG1$ -w $ARG2$ \
                             -c $ARG3$ -l $ARG5$ -u $ARG4$
}

define host {
  use                   generic-network
  host_name             bc-mgmt1
  alias                 bc-mgmt1.home.barfoo.org
  address               10.0.0.35
  parents               uni-greif-05
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Temperature
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.1.5.1.0!\
                                  33!38!C!Input temperature
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Chassis Cooling - Bay 1
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.3.20.0!\
                                  1600:1200,2100:2600!1200:0,2600:3000!RPM!\
                                  Chassis Cooling - Bay 1
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Chassis Cooling - Bay 2
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.3.21.0!\
                                  1600:1200,2100:2600!1200:0,2600:3000!RPM!\
                                  Chassis Cooling - Bay 2
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 1
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.6.1!\
                                  6200:5400,6700:7000!5300:0,7000:7500!RPM!\
                                  Power Module Cooling - Bay 1
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 1 Fans
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.4.1!\
                                  2:1!1:0!Fans present!\
                                  Power Module Cooling - Bay 1
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 2
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.6.2!\
                                  6200:5400,6700:7000!5300:0,7000:7500!RPM!\
                                  Power Module Cooling - Bay 2
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 2 Fans
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.4.2!\
                                  2:1!1:0!Fans present!\
                                  Power Module Cooling - Bay 2
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 3
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.6.3!\
                                  6200:5400,6700:7000!5300:0,7000:7500!RPM!\
                                  Power Module Cooling - Bay 3
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 3 Fans
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.4.3!\
                                  2:1!1:0!Fans present!\
                                  Power Module Cooling - Bay 3
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 4
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.6.2!\
                                  6200:5400,6700:7000!5300:0,7000:7500!RPM!\
                                  Power Module Cooling - Bay 4
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Module Cooling - Bay 4 Fans
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.6.1.1.4.2!\
                                  2:1!1:0!Fans present!\
                                  Power Module Cooling - Bay 4
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Domain 1: Utilization
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.10.1.1.1.10.1!\
                                  2600:2400!2880:2600!W!\
                                  Power Domain 1: Utilization
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             bc-mgmt1
  service_description   Power Domain 2: Utilization
  check_command         check_snmpv1!.1.3.6.1.4.1.2.3.51.2.2.10.1.1.1.10.2!\
                                  2600:2400!2880:2600!W!\
                                  Power Domain 2: Utilization
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

Opsview has the advantage that it at least lets you use the package manager. But, it also needs *a lot* of handy work (just like Centreon, which I really dislike since it’s real error prone).

I started doing the setup, but gave up halfway through … Dude, and they expect people to pay money for training ?!?

I mean, come on .. you can do a better job at making this thing fit a bit better into the system, and even make the install a bit more straight forward.

It was rather easy hacking in support for performance data into it, as the below shows. Someone else already posted a non-unified diff for performance data support, but that ain’t quite right according to the Nagios plug-in development guidelines.

--- check_ram.py.orig
+++ check_ram.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 #
 #   Copyright Hari Sekhon 2007
+#   Copyright Christian Heim 2008
 #
 #   This program is free software; you can redistribute it and/or modify
 #   it under the terms of the GNU General Public License as published by
@@ -64,27 +65,30 @@
     total_used_megs = float(memtotal-memfree) / 1024
     total_free_megs = float(total_free) / 1024
     memtotal_megs   = float(memtotal) / 1024
+
+    total_warning_threshold_megs = round(float(memtotal) * float(warning_threshold) / 100 / 1024)
+    total_critical_threshold_megs = round(float(memtotal) * float(critical_threshold) / 100 / 1024)

     if percent == True:
         percentage_free = int( float(total_free) / float(memtotal) * 100 )
         if percentage_free < critical_threshold:
-            print "RAM CRITICAL: %d%% ram free (%d/%d MB used)" % (percentage_free,total_used_megs,memtotal_megs)
+            print "RAM CRITICAL - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return CRITICAL
         elif percentage_free < warning_threshold:
-            print "RAM WARNING: %d%% ram free (%d/%d MB used)" % (percentage_free,total_used_megs,memtotal_megs)
+            print "RAM WARNING - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return WARNING
         else:
-            print "RAM OK: %d%% ram free" % percentage_free
+            print "RAM OK - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return OK
     else:
         if total_free < critical_threshold:
-            print "RAM CRITICAL: %dMB ram free (%d/%d MB used)" % (total_free_megs,total_used_megs,memtotal_megs)
+            print "RAM CRITICAL - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return CRITICAL
         if total_free < warning_threshold:
-            print "RAM WARNING: %dMB ram free (%d/%d MB used)" % (total_free_megs,total_used_megs,memtotal_megs)
+            print "RAM WARNING - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)
             return WARNING
         else:
-            print "RAM OK: %dMB ram free" % (total_free_megs)
+            print "RAM OK - %d%% free (%d MB out of %d MB) |ram=%dMB;%d;%d;0;%d" % (percentage_free,total_free_megs,memtotal_megs,total_free_megs,total_warning_threshold_megs,total_critical_threshold_megs,memtotal_megs)

             return OK

Now, with that simple plug-in I can go ahead and package the thing separately, without the need to figure out a way to get them nagios-plugins to install some useful includes ...

Ethernet box 2

That’s what the ~300 EUR box looks like. It’s basically a small black box with a RJ45 jack, and four RJ11 jacks for attached external devices. The box itself only functions as a “management station” and doesn’t come with a sensor.
Normally, you can attach up till four RJ11 sensors to it. But, MessPC also has RJ11 port splitters, which enables you to attach up to eight RJ11 sensors to the MessPC.

Thermometer RJ45 jacks

As you can see, the box has a RJ45 jack on the other side, which you basically hook up to your network and then configure an IP address (or if you fancy DHCP for those things, it’s possible too).

Thermometer RJ11 jacks

On the opposite site, are the RJ11 jacks for the sensors. As you can see, we currently do have 4 splitters attachted to the box, enabling up till 8 sensors to be measured.
Once you have it up and running, you can look at the web interface and you’ll be able to see the state of the sensors right on the first page.

Now, if you have a Nagios installation (like I do), you might want to integrate the Ethernetbox with Nagios.
You do have two choices: either do the checking by means of SNMP but forego graphing by Nagios, or use check_pcmeasure and use the full features of your Nagios installation (read: graphing done by pnp4nagios).
As I recently updated our Nagios installation to 3.0.4, I ran into trouble with the pcmeasure plugin. It simply didn’t run with the embedded perl (which it should, but apparently became a bit stricter between 2.x and 3.x). So I went ahead and changed it, so it would run with embedded perl, as I was in the mood.
After that was done, I simply dumped it into /usr/lib/nagios/plugins on my nagios box,
created a /etc/nagios/objects/commands/etherbox.cfg containing the necessary command definition:

# 'check_ethernetbox' command definition
define command{
        command_name    check_ethernetbox
        command_line    /usr/lib/nagios/plugins/check_pcmeasure.pl -H $HOSTADDRESS$ -S $ARG1$ -W $ARG2$ -C $ARG3$
}

Nagios Service Detail

Now we can go ahead and *really* integrate it into Nagios, by simply creating a host utilizing this command definition.

define host {
  use                   generic-network
  host_name             ethernetbox
  alias                 ethernetbox.home.barfoo.org
  address               10.0.0.9
  parents               home-0
}

define service {
  use                   generic-service
  host_name             ethernetbox
  service_description   1.1: Aussentemperatur
  check_command         check_ethernetbox!com1.1!29.0!35.0!
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

define service {
  use                   generic-service
  host_name             ethernetbox
  service_description   1.2: Druckfussboden
  check_command         check_ethernetbox!com1.2!16.0!19.0!
  action_url            /pnp/index.php?host=$HOSTNAME$&srv=$SERVICEDESC$
  notes                 View PNP RRD grap
}

Here’s a snippet from my nrpe.cfg:

dont_blame_nrpe=1
command[check_disk]=/usr/lib/nagios/plugins/check_disk -E -w $ARG1$ -c $ARG2$ -p $ARG3$

Now, if you’d check the free space for the root, it ain’t gonna show any inode percentage (that one isn’t what I’m talking about). But if you have to use bind mounts like I do (Tivoli needs a separate “domain” — that is a separate mount point for each domain), you might wanna check the free space on the *real* device, rather than the free space on the bind mount (which is gonna show you the free space of the parent file system – in my case the root fs).

Let’s take a look at what I’m talking about. If you use the check_disk locally like this:

# ./check_disk -w 20% -c 10% -p /apache/
DISK OK - free space: /apache 11090 MB (36% inode=36%);| /apache=19629MB;24575;27647;0;30719

Means, everything is okay, you have to pass the extra trailing slash to the –partition argument, as otherwise it would pick up the bind mount at /backup.

Now, if we do the above by means of NRPE, that’s gonna get you a different result. As I showed above, I have the check_disk command in my nrpe.cfg, I also specifically enabled command arguments during compile time.

# ./check_nrpe -H nagios.home.barfoo.org -c check_disk -a 20% 5% /apache/
DISK CRITICAL: /apache/ not found

Now, why the hell isn’t it picking up the *original* mount point of the file system ? Guess why … Because I added -E to the command, because it didn’t use the original mount point but rather the bind mount in /backup. Removing the -E and it picks up the *original* mount point without any trouble *shrug*.

define hostgroup {
        hostgroup_name      a-parent-hostgroup
        alias               Our toplevel parent hostgroup
}

define service {
       use                  generic-service
       check_command        check_dummy!0!
       service_description  SSH
       hostgroup_name       a-parent-hostgroup
}

define hostgroup {
        hostgroup_name      a-child-hostgroup
        hostgroup_members   a-parent-hostgroup
        alias               Our child hostgroup
}

define service {
       use                  generic-service
       check_command        check_dummy!0!
       service_description  LOAD
       hostgroup_name       a-child-hostgroup
}

As you can cleary see on line 14, I thought you define the relation between two hostgroups in the child hostgroup. The problem with it was basically (as I said in the earlier posts), that all the services defined for the child hostgroups are handed on upwards to the parent hostgroup(s).

But after talking to Tobi, I quickly found out, that the relation is in fact defined within the parent hostgroup. So if you simply put hostgroup_members within the parent hostgroup and define all child hostgroups which should inherit from the parent one, you should be just fine.

Gonna have to try to define the dependency backwards, maybe that’ll help. But that’s a topic for Monday. Guess I’ll finish viewing Ghost in the Shell – Stand Alone Complex first.

As you can imagine, it gets quite confusing with that amount of checks (~150). So the last two days I spent on reorganizing (with Visio), on which object/hostgroup placing a check would make sense. Now, this is my first result of two days planning, reorganizing, reordering and moving hosts into different hostgroups.

Nagios Hostgroup Inheritance - Linux

Nagios Hostgroup Inheritance - Windows

Thanks to Josh (and Chris I think), realizing the above is gonna get quite easy. Gonna talk about the config layout itself about once I have it all wrapped up. Stay tuned!

Now as I went into work today and started my email client, I received 4 nagios warnings about a LOAD service reaching critical state. Looked at the nagios box itself, opened up the VM console, looked into the syslog. Nothing.

Yet over 3/4 of the services were flapping, some ping checks were critical (for whatever reason). So I opened the nagios webinterface again, and noticed it dropping the connection over and over again (had to reauthentificate me again and again).

So I opened up Putty, which established the connection without a single problem, but dropped me like a stone after a short amount of time. I restarted the session and got a security warning from Putty (due to different than the saved sshd public key). That raised my suspicion. So I took a look at the hostname, and lookie there.

Somehow my old nagios box (which is a physical box), got turned online again, thus having the same IP address as my virtualized one. So the virtualized nagios wasn’t really dropping my connection, but I was being directed to the old nagios.

Walked over into the data center, turned of the old box (well, I kept the power button pressed for a short time), and away went my troubles.

The bad thing about that is, that stuff becomes dependant on the Active Directory (we do have four domain controllers, so that should be fine).

Now, here’s the ssl-(only) apache2 configuration file for my vhost:

  ## mod_core
  DocumentRoot "/usr/share/nagios"

  ServerName nagios.barfoo.org
  ServerAlias nagios3.barfoo.org
  ServerAdmin nagiosadmin@barfoo.org

  ## mod_rewrite
  RewriteEngine On
  RewriteRule ^/(.*)         https://nagios.barfoo.org/$1 [L,R]





  ## mod_core
  DocumentRoot "/usr/share/nagios"

  ServerName nagios.barfoo.org
  ServerAdmin nagiosadmin@barfoo.org

  ScriptAlias /nagios/cgi-bin /usr/lib/nagios/cgi
  Alias /nagios /usr/share/nagios
  Alias /pnp /usr/share/nagios/html/pnp4nagios

  
    AllowOverride None
    Order deny,allow
    Deny from all
    Allow from 10.0.0.
    Options None

    # Authorization
    AuthType Basic
    AuthName "Nagios Barfoo"

    # The authentification provider is mod_ldap
    AuthBasicProvider ldap

    # mod_ldap is our *only* authentification provider for this!
    AuthzLDAPAuthoritative on

    # Redirect the userfile requests to /dev/null
    AuthUserFile /dev/null

    # AD requires an authentication DN to access any records
    AuthLDAPBindDN "BARFOO\\ldap_nagios"
    AuthLDAPBindPassword "somethingrandom"

    # The URL to search in
    AuthLDAPURL "ldap://dc0.barfoo.org dc1.barfoo.org dc2.barfoo.org dc3.barfoo.org/OU=Users,dc=barfoo,dc=org?sAMAccountName?sub?(objectClass=*)"

    # Search the group membership in the specified group, otherwise it's gonna
    # get searched at the binding DN's location
    AuthLDAPGroupAttributeIsDN on
    Require ldap-group CN=gr_nagios,OU=Groups,DC=barfoo,DC=org
  

  ## mod_log
  ErrorLog /var/log/apache2/nagios.barfoo.org.error_log
  TransferLog /var/log/apache2/nagios.barfoo.org.access_log
  CustomLog /var/log/apache2/nagios.barfoo.org.ssl_request_log   ssl_combined

  ## mod_ssl
  SSLEngine on
  SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
  SSLCertificateFile /etc/apache2/ssl.crt/nagios.barfoo.org.crt
  SSLCertificateKeyFile /etc/apache2/ssl.key/nagios.barfoo.org.key

  
    SSLOptions +StdEnvVars
  

  SetEnvIf User-Agent ".*MSIE.*" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0

As you can see, AuthLDAPUrl holds the four LDAP servers separated by spaces (that’s what the Apache2 documentation says about that), and that actually works.

The only additional thing I had to change from the nagios part is in /etc/nagios/cgi.cfg to allow everyone to issue system commands. Also, if you ever stumble upon extranous chars in the check_nrpe output, update to a newer NRPE version, that fixed it for me (that is on the receiver side – as in the box running the NRPE agent).

Since I wanted a manageable setup (as in unified code base across all distributions), I wanted it to work without installing too much. So I took the swap plugin and basically removed what wasn’t necessary and voila!

Here we go, yay ME!

Only thing I need to finish sometime soon, is getting the NSClient++ work on my Windows boxen (which I do have quite a few, the domain controllers, nas-cluster, …)