evilrouters.net

This morning, I received an e-mail from a cronjob on one of my production RHEL 5.2 servers:

From: Cron Daemon
To: Jeremy L. Gaddis
Cc:
Subject: Cron <root@SERVERNAME> run-parts /etc/cron.weekly

/etc/cron.weekly/makewhatis.cron:

rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: b4a0
error: db4 error(22) from db->close: Invalid argument
error: cannot open Pubkeys index using db3 - Cannot allocate memory (12)

...

There were probably a couple hundred errors in that e-mail. In addition, I also received an e-mail from our RHN Satellite Server letting me know that this particular server had failed to check in. Logging in, I saw that, indeed, it had not been checking in with the our satellite.

So, what to do? Google, of course! Fortunately, major over at Racker Hacker encountered this same issue about a year and a half ago and has already provided the fix for us:

[root@SERVERNAME ~]# tar cvzf rpmdb-backup.tar.gz /var/lib/rpm
[root@SERVERNAME ~]# rm /var/lib/rpm/__db.00*
[root@SERVERNAME ~]# rpm --rebuilddb
[root@SERVERNAME ~]# rpm -qa | sort # to make sure everything's okay

I wanted to verify that the cronjob would now successfully execute, so I invoked it manually:

[root@SERVERNAME ~]# sh /etc/cron.weekly/makewhatis.cron
[root@SERVERNAME ~]#

Success! It also seemed like a good time to go ahead and install the updates that were missing so I took care of those using yum.

Many thanks to major at Racker Hacker for the fix!