evilrouters.net

This morning, I received an e-mail from a cronjob on one of my production RHEL 5.2 servers:

From: Cron Daemon
To: Jeremy L. Gaddis
Cc:
Subject: Cron <root@SERVERNAME> run-parts /etc/cron.weekly

/etc/cron.weekly/makewhatis.cron:

rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: b4a0
error: db4 error(22) from db->close: Invalid argument
error: cannot open Pubkeys index using db3 - Cannot allocate memory (12)

...

There were probably a couple hundred errors in that e-mail. In addition, I also received an e-mail from our RHN Satellite Server letting me know that this particular server had failed to check in. Logging in, I saw that, indeed, it had not been checking in with the our satellite.

So, what to do? Google, of course! Fortunately, major over at Racker Hacker encountered this same issue about a year and a half ago and has already provided the fix for us:

[root@SERVERNAME ~]# tar cvzf rpmdb-backup.tar.gz /var/lib/rpm
[root@SERVERNAME ~]# rm /var/lib/rpm/__db.00*
[root@SERVERNAME ~]# rpm --rebuilddb
[root@SERVERNAME ~]# rpm -qa | sort # to make sure everything's okay

I wanted to verify that the cronjob would now successfully execute, so I invoked it manually:

[root@SERVERNAME ~]# sh /etc/cron.weekly/makewhatis.cron
[root@SERVERNAME ~]#

Success! It also seemed like a good time to go ahead and install the updates that were missing so I took care of those using yum.

Many thanks to major at Racker Hacker for the fix!

From an e-mail I received Tuesday:

Hello Jeremy,

Thank you for your interest in VMware beta programs. Our upcoming release of VMware Converter 4.0 Standalone product includes many exciting enhancements that our customers have been requesting including P2V support for Linux and Win 2K8 sources, hot cloning enhancements as well as workflow automation enhancements. We are certain you will find participation in this beta program a valuable experience. We are looking forward to working closely with you during this beta program.

As part of this beta, we request you to extensively test several areas of feature enhancements including P2V support for Linux and Win 2K8 sources, hot cloning enhancements as well as workflow automation enhancements. Your active participation in this beta program is critical. We appreciate and value your efforts to install upon downloading the software and actively provide us with your valuable product feedback.

I’m just about (in the next week or so) to attempt to P2V a Red Hat Enterprise Linux 4 host over to ESX, so maybe I’ll give the new 4.0 beta a shot. Anyone used it yet, especially to P2V Linux hosts? I am, of course, interested in hearing feedback on it.

Coming in to work to see this is never good. It wasn’t bad, it just wasn’t good. RAID FTW!