evilrouters.net

1. If you come to me to ask technical questions, please don’t argue when you don’t like my answer. If you think you know more about what you’re asking than I do, then why even ask? On that same note, if I am arguing with you, it’s because I’m certain that I am correct; otherwise I’d just tell you “I don’t know” or perhaps point you somewhere that you could look it up. We don’t argue just for the sake of arguing.

2. When you start a conversation by insulting yourself (e.g. “I’m such an idiot”), you will not make me laugh or feel sorry for you; all you will succeed in doing is reminding me that yes, you are, indeed, an idiot, and that I’m going to hate having to talk to you. Trust me, you don’t want to start out this way.

3. We’re okay with you making mistakes; fixing them is part of our job. We are NOT, however, okay with you lying to us about a mistake that you made. It just makes it that much harder to resolve and thus makes our job more difficult. Be honest and we’ll get the problem fixed and both of us can continue on with our business. Lying to us and, therefore, costing us twice as much of our time will not win you any brownie points with IT.

4. There is no magic “Fix it” button. Everything takes some amount of work to fix, and not everything is worth fixing or — gasp! — even possible to fix. If I tell you that you’re going to have to re-do a document that you accidentally deleted two months ago, please don’t get mad at ME. I’m not ignoring your problem and it’s not that I don’t like you, we just can’t always fix everything.

5. Not everything you ask us to do is “urgent”. In fact, by marking things as “urgent” every time, you’ll almost certainly ensure that we treat none of it as a priority.

6. You are not the only one who needs help, and you usually don’t have the most urgent issue. Give us some time to get to your problem; it will get fixed.

7. E-mailing us several times about the same issue is not only unnecessary, it’s highly annoying as well. We record issues in a database so that we don’t lose track of them (remember how we ask that you create a ticket? That’s why.) We will typically respond as soon as we have a useful update to make. If your problem is urgent, please do let us know (but see number five).

8. Yes, we prefer e-mail over phone calls. It has nothing to do with being friendly or anti-social, it’s about efficiency. It is much faster and easier for us to list out a set of questions that we need answers to than it is for us to call and ask you them one by one. You can find the answers at your leisure and, while we’re waiting, we can work on other problems.

9. We may, at times, seem blunt and rude. It’s not that we mean to, we just don’t have the time to sugar coat things for you. We assume that we are both adults and can handle the reality of a problem. If you did something wrong, don’t be surprised when we tell you. We don’t care that it was a mistake because, honestly, it makes no difference to us. Please don’t take it personal, we just don’t want it to happen again.

10. Finally, yes, I can read your e-mail, yes, I can see what web pages you look at while you’re at work, yes, I can access every file on your work computer, and yes, I can tell if you are chatting with people on instant messenger (and can read what you’re typing, as well). But no, we don’t do it. It’s highly unethical and, perhaps more importantly, in all reality you really aren’t that interesting. Unless I am instructed to specifically monitor or investigate your actions, I don’t do it. There really are much more interesting things on the Internet than you.

I hope this didn’t come off the wrong way because, even as much as us IT guys refer to “users” as “lusers”, we do like (most of) you. Just like you, we’re here to do a job and we try to do it the best that we can. It’s easiest to do that if we all work together, stop pointing fingers, and give other people the space that we would like to get as well. If we can do that more often than not, things will go well and work out for all of us.

P.S. IT guys are easily bribed with food and/or beer (personally, I prefer the latter). That’s a sure way to get your problems moved to the top of the list. *wink*

If you were to e-mail me over the holidays, you’d get a response like this. Unfortunately, I had to “tone it down” a bit (couldn’t make it as funny as I would have liked). =)

---------- Original message ----------
Date: Tue, 23 Dec 2008 17:35:03 -0500
From: Jeremy L. Gaddis
Subject: Out of Office AutoReply:

[This message was generated by an automated system.]

Greetings and salutations!

I am out of the office and my current whereabouts are unknown.  Even I do not know
where I am.  I am not *really* on vacation (as in, I'm not really going anywhere far away),
but as far as responding to email is concerned, I appear to be well outside the solar system.

Other than December 29-30, I will not be around until January 5th.

Boiler plate:  If you have any special needs, please contact the Help Desk.  They will have
the tools and skills needed to get you back on track.

If you are the Help Desk, then please contact Kevin or Ben.

If you *ARE* Kevin or Ben:  stop, take a deep breath, and reassess things.  If you are still
hyperventilating, then go ahead and break the glass (a.k.a. call my Blackberry)!

Happy holidays!

I found myself recently setting up new HP ProCurve 5400 switches in production. Because I’m a network guy, I like to keep an eye on them (interface counters, traps, etc.), thus setting up SNMPv3 was necessary. In addition, these devices come (”out of the box”) with a default read-write community string set to — you guessed it — “public”, open to anywhere. That had to be taken care of first.

Setting up SNMPv3:

First, let’s set some basic information so we can track this device amongst all the others:

SWITCH1# conf
SWITCH1(config)# snmp-server location S123
SWITCH1(config)# snmp-server contact jlgaddis

Next, we’ll enable SNMPv3 which, on these 5400s, also has the effect of creating an “initial” user:

SWITCH1(config)# snmpv3 enable
SNMPv3 Initialization process.
Creating user 'initial'
Authentication Protocol: MD5
Enter authentication password: ******
Privacy protocol is DES
Enter privacy password: ******

User 'initial' is created
Would you like to create a user that uses SHA? n

User creation is done.  SNMPv3 is now functional.
Would you like to restrict SNMPv1 and SNMPv2c messages to have read only
access (you can set this later by the command 'snmp restrict-access'): y

What happened here is that an SNMPv3 user (with username “initial”) was automatically created for us. We were prompted for the authentication password and privacy password (note that the protocols were automatically chosen). At this point, I just entered “123456″ as I have plans to delete that user anyway. I went ahead and answered “y” to the last question, but I’ll be turning off SNMPv1 and SNMPv2 in a bit moment regardless.

Let’s configure our switch to only run SNMPv3 and go ahead a create a new SNMPv3 user as well:

SWITCH1(config)# snmpv3 only
SWITCH1(config)# snmpv3 restricted-access
SWITCH1(config)# snmpv3 user cacti auth sha AUTHPASS priv aes PRIVPASS

Here I was setting up a user so that my “graphing application” of choice, cacti, can communicate with the switch to retrieve interface statistics. Substitute your own authentication password and privacy passwords above (”AUTHPASS” and “PRIVPASS”). You can change the protocols as well, if you’d like, to MD5 and DES, respectively. I prefer to go the “high security” route whenever possible, however, so that’s what I opted for here. Be sure your management software is compatible with these settings!

Now, we need to assign our “cacti” user to a group that’s appropriate for the level of access we want it to have. I won’t describe all of the ones available (see Chapter 14 of the Management and Configuration Guide for that), but the one I want (in this case) is “operatorauth”. This group provides for “operator” level access (a.k.a. “unprivileged”) and requires authentication. We’ll also specify “sec-model ver3″ as an SNMPv3 access group should only use the ver3 security model:

SWITCH1(config)# snmpv3 group operator auth user cacti sec-model ver3

Okay, almost there! Now we just need to allow SNMP access to the switch from the host that cacti is running on. In my case, it’s 172.30.144.17:

SWITCH1(config)# ip authorized-managers 172.30.144.17 255.255.255.255 access operator access-method snmp

You can change that, of course, to your own IP address (or whole networks — be sure to change the netmask, however).

At this point, we should be good to go. We could add the device into cacti’s web interface and within a few polling cycles we’ll start to see interface traffic statistics, such as this (from another device):

Finally, there’s one more step that might be necessary, depending upon your switch’s configuration. Because my switch has a loopback address assigned to it, that’s the IP address I want to tell cacti to poll. This method will still allow the switch to be reachable if one (or more) of it’s interfaces go down (there are multiple routes to it). By default, the ProCurve 5400 will respond to SNMP requests with a source IP address of the interface that the requests were received on, and NOT a source IP matching the original destination of the requests:

SWITCH1(config)# snmp-server response-source dst-ip-of-request

…and that’s it! We can now “speak” SNMPv3 (and ONLY SNMPv3) to our switch. In addition, only the “cacti” user can access it, and only from 172.30.144.17.

That’s a helluva lot better than the default read-write “public” community string that’s accessible from anywhere, huh!?

UPDATE: I forgot the part where I deleted the “initial” user that was created automatically for us. Here’s how that’s done:

SWITCH1(config)# no snmpv3 user initial

Easy enough!

-----Original Message-----
From: PCC-Americas
Sent: Friday, December 19, 2008 5:22 PM
To: Jeremy L. Gaddis
Subject:RE: en-us: Possible bug in K.13.45 (5400zl series)?

Dear Jeremy,

Thank you for contacting HP ProCurve Networking.

It seems that you have discovered an anomaly.  We would like to
investigate this for you.  At your convenience, would you mind
collecting the textual output of the command, "show tech all" as
issued within the CLI of the switch?  Please follow-up the text
capture by again issuing the "show ip igmp config" command.

We will work with our engineers to reproduce this issue, and
identify its root cause.

Thank you very much for contacting HP ProCurve Networking Support.
We hope to hear form you soon.

Sincerely,
Linda
HP ProCurve Networking

Here’s what I was seeing (serial numbers of my installed GBICs “sanitized”). This was on a HP ProCurve 5406zl:

SWITCH# show ip igmp config

 IGMP Service

                       IGMP     Forward with   Querier  Querier
  VLAN ID VLAN Name    Enabled  High Priority  Allowed  Interval
  ------- ------------ -------- -------------- -------- ---------
  1       DEFAULT_VLAN No       No             Yes      125
  2       VLAN2        No       No             Yes      125
  14      VLAN14       No       No             Yes      125
  16      VLAN16       No       No             Yes      125
  20      VLAN20       No       No             Yes      125
  30      VLAN30       No       No             Yes      125
  31      VLAN31       No       No             Yes      125
  32      VLAN32       No       No             Yes      125
  36      VLAN36       No       No             Yes      125
  38      VLAN38       No       No             Yes      125
  41      VLAN41       No       No             Yes      125
  42      VLAN42       No       No             Yes      125
  43      VLAN43       No       No             Yes      125
  64      VLAN64       No       No             Yes      125
        GBIC 1 (  Port A1): J4858C               XXXX2EK3W9
        GBIC 2 (  Port A2): J4858C               XXXX2EK3X4
        GBIC 3 (  Port A3): J4858C               XXXX2EK1Z2
        GBIC 4 (  Port A5): J4858C               XXXX2EK1RT
        GBIC 5 (  Port A7): J4858C               XXXX2EK2G4
        GBIC 6 (  Port A9): J4858C               XXXX2EK3FM
        GBIC 7 ( Port A11): J4858C               XXXX2EK3WD
        GBIC 8 ( Port A13): J4858C               XXXX2EK2NF
        GBIC 9 ( Port A14): J4858C               XXXX2EK4YD
        GBIC 10 ( Port A15): J4858C               XXXX2EK1HG
        GBIC 11 ( Port A16): J4858C               XXXX2EK5HA
        GBIC 12 ( Port A17): J4858C               XXXX2EK2CG
        GBIC 13 ( Port A18): J4858C               XXXX2EK2GH
        GBIC 14 ( Port A20): J4858C               XXXX2EK1RP
        GBIC 15 ( Port A21): J4859C               XXXX0EL04Y
        GBIC 16 ( Port A22): J4859C               XXXX0EL06W
        GBIC 17 ( Port A23): J4859C               XXXX4EL053
        GBIC 18 ( Port A24): J4859C               XXXX4EL02X
  78      VLAN78       No       No             Yes      125
  79      VLAN79       No       No             Yes      125
  80      VLAN80       No       No             Yes      125
  94      VLAN94       No       No             Yes      125
  96      VLAN96       No       No             Yes      125
  101     VLAN101      No       No             Yes      125
  110     VLAN110      No       No             Yes      125
  112     VLAN112      No       No             Yes      125
  128     VLAN128      No       No             Yes      125
  172     VLAN172      No       No             Yes      125
  192     VLAN192      No       No             Yes      125
  202     VLAN202      No       No             Yes      125
  4011    VLAN4011     No       No             Yes      125
  4012    VLAN4012     No       No             Yes      125
  4030    VLAN4030     No       No             Yes      125
  4040    VLAN4040     No       No             Yes      125
  4050    VLAN4050     No       No             Yes      125
  4060    VLAN4060     No       No             Yes      125
  4070    VLAN4070     No       No             Yes      125

Geez, an “anomaly”? Ya think? =)

It’s been a looooong time since I posted any networking stuff that wasn’t Cisco-centric, but I’m sitting here at home configuring an HP ProCurve 5406zl so I thought I’d take the opportunity.

The ProCurve 5400zl series have a USB port on them that you can use to transfer files, in addition to TFTP and SCP/SFTP. Since I had a few of these to upgrade and they were in a lab environment (e.g. not connected to any “real” networks), I didn’t want to bother with setting up a TFTP server. The upgrade process is pretty straightforward and is similar to doing an upgrade via TFTP.

We can find the latest software for our ProCurve switches on the “Software for switches” page. Software (”firmware”) updates do not require that you have a valid login or service contract, unlike Cisco. I grabbed the latest version (at the time of writing), which is K.13.45 (be sure to read the Release Notes that accompany each release as well, prior to performing an upgrade). Save the .downloaded file to your USB flash drive and plug the flash drive into the switch.

To check what version of the software is currently running, issue the “show version” command:

SW1# show version
Image stamp:    /sw/code/build/btm(t3a)
                Aug  4 2008 15:08:24
                K.13.25
                93
Boot Image:     Primary

We can see that we’re running version K.13.25 and that we booted from the primary flash. We can see the current contents of flash, as well as our USB drive:

SW1# show flash
Image           Size(Bytes)   Date   Version
-----           ----------  -------- -------
Primary Image   : 7442476   08/04/08 K.13.25 
Secondary Image : 6782942   12/07/07 K.12.57 
Boot Rom Version: K.12.12
Default Boot    : Primary

SW1# dir

Listing Directory /ufa0:
-rwxrwxAwx  1 0       0          7442476 Nov  3  2008 K_13_25.SWI 
-rwxrwxAwx  1 0       0          7494786 Oct 30  2008 K_13_45.SWI 
SW1#

Because I’ve been running K.13.25 and it’s been stable, I’m going to copy it to secondary flash and then overwrite the primary with the new software. We’ll then reboot the switch with the new software (keeping the previous version in secondary as a “backup” in case anything goes wrong).

SW1# copy flash flash secondary

This command isn’t real intuitive (and it takes a while as well), but here we’re basically copying from flash, to flash, with the secondary as our destination. In this case, the contents of the primary flash will be copied to the secondary. “copy flash flash primary” would copy the contents of the secondary into the primary. Let’s verify what we have now:

SW1# show flash
Image           Size(Bytes)   Date   Version
-----           ----------  -------- -------
Primary Image   : 7442476   08/04/08 K.13.25 
Secondary Image : 7442476   08/04/08 K.13.25 
Boot Rom Version: K.12.12
Default Boot    : Primary

We can see that the contents of the primary have now been copied to the secondary as well. Let’s copy the K1345.SWI image from the USB drive to primary flash:

SW1# copy usb flash K_13_45.SWI primary
The Primary OS Image will be deleted, continue [y/n]?  y

After a moment, we’ll see this message:

Validating and Writing System Software to the Filesystem ...

When the copy has completed, we need to reload the switch with the new software:

SW1# boot system flash primary
System will be rebooted from primary image. Do you want to continue [y/n]?  y

The switch will take a minute to reboot (I won’t bother pasting the complete bootup process) and then we can, again, use “show version” to verify that we’re now running the latest software:

SW1# show version
Image stamp:    /sw/code/build/btm(t3a)
                Oct 17 2008 20:03:02
                K.13.45
                706
Boot Image:     Primary

See, wasn’t that easy!? We’ve successfully upgraded the firmware, and we’ve also kept a backup copy of the previous software in case things go badly. If that happens, just issue the “boot system flash secondary” command to reload the switch with the previous software.

This “upgrade via USB” method can come in handy at times, e.g. when the switch is in a lab and you don’t have a server handy to load the files from. For the switches in my production network, I would use SFTP to ugprade them instead of having to visit each switch individually to plug in and remove the USB drive. Yes, you can SFTP to the switch and upload a new version of firmware. It rocks. =)

This morning, I received an e-mail from a cronjob on one of my production RHEL 5.2 servers:

From: Cron Daemon
To: Jeremy L. Gaddis
Cc:
Subject: Cron <root@SERVERNAME> run-parts /etc/cron.weekly

/etc/cron.weekly/makewhatis.cron:

rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: b4a0
error: db4 error(22) from db->close: Invalid argument
error: cannot open Pubkeys index using db3 - Cannot allocate memory (12)

...

There were probably a couple hundred errors in that e-mail. In addition, I also received an e-mail from our RHN Satellite Server letting me know that this particular server had failed to check in. Logging in, I saw that, indeed, it had not been checking in with the our satellite.

So, what to do? Google, of course! Fortunately, major over at Racker Hacker encountered this same issue about a year and a half ago and has already provided the fix for us:

[root@SERVERNAME ~]# tar cvzf rpmdb-backup.tar.gz /var/lib/rpm
[root@SERVERNAME ~]# rm /var/lib/rpm/__db.00*
[root@SERVERNAME ~]# rpm --rebuilddb
[root@SERVERNAME ~]# rpm -qa | sort # to make sure everything's okay

I wanted to verify that the cronjob would now successfully execute, so I invoked it manually:

[root@SERVERNAME ~]# sh /etc/cron.weekly/makewhatis.cron
[root@SERVERNAME ~]#

Success! It also seemed like a good time to go ahead and install the updates that were missing so I took care of those using yum.

Many thanks to major at Racker Hacker for the fix!

From an e-mail I received Tuesday:

Hello Jeremy,

Thank you for your interest in VMware beta programs. Our upcoming release of VMware Converter 4.0 Standalone product includes many exciting enhancements that our customers have been requesting including P2V support for Linux and Win 2K8 sources, hot cloning enhancements as well as workflow automation enhancements. We are certain you will find participation in this beta program a valuable experience. We are looking forward to working closely with you during this beta program.

As part of this beta, we request you to extensively test several areas of feature enhancements including P2V support for Linux and Win 2K8 sources, hot cloning enhancements as well as workflow automation enhancements. Your active participation in this beta program is critical. We appreciate and value your efforts to install upon downloading the software and actively provide us with your valuable product feedback.

I’m just about (in the next week or so) to attempt to P2V a Red Hat Enterprise Linux 4 host over to ESX, so maybe I’ll give the new 4.0 beta a shot. Anyone used it yet, especially to P2V Linux hosts? I am, of course, interested in hearing feedback on it.

Coming in to work to see this is never good. It wasn’t bad, it just wasn’t good. RAID FTW!