evilrouters.net

Here’s a fine example of that wonderful HP ProCurve engineering and quality control I’ve grown intimately familiar with:

The following problems were resolved in release K.13.71.

…

IP Communication (PR_0000044004) – Switches running software versions K.13.65 - K.13.68 may experience a resource leak in ICMP that eventually causes loss of IP communication with the following symptoms.

• The switch will stop routing traffic for hosts for which it provides gateway services.
• All dynamic routing protocols stop functioning. The switch will stop sending routing protocol packets, and will not process incoming routing protocol packets sent from its neighbors.
• In-band network management stops functioning. The switch will become inaccessible via Telnet, SSH, WEB, and TFTP.
• Console management may become very sluggish and may appear to be non-responsive.
• Output from the CLI command show ip route will be corrupted.
• A reboot is required to clear up the symptoms.

BUT AT LEAST WE SAVED SOME MONEY!

(Note: Yes, I have been bit by this bug.)

Apparently I need to stop “following” a couple folks on Twitter, namely @procurve. Because of my past experiences, I tend to not care much for HP ProCurve equipment and so sometimes the things posted by @procurve irk me, just a little.

One example was on August 28th, when @procurve “re-tweeted” this:

Now, in terms of functionality, the ProCurve 5406zl switch does not even come anywhere near a Cisco 7206VXR. They are in two completely different categories, as far as I am concerned. I made the remark that “they couldn’t have been doing much with that 7206vxr if they were able to replace it w/ a 5406zl”. Anyone who is familiar with both devices would, I’m positive, agree with me. Nonetheless, nice try at patting yourself on the back, HP.

So it’s little instances like this that annoy the shit out of me, considering the not-so-great experiences I’ve had with ProCurve in the last year or so. Today, @procurve posted this:

I, of course, wanted to hear what they had to say, so I clicked. Read the blog post, “Cost effective networking — The choice is yours”, if you’d like. I won’t post the whole thing here, but reading it will help you to understand my response.

If you did click that link, you won’t see my comments posted. As far as I can tell, my posted was never allowed to be displayed on that page. I’m glad I kept a copy before I submitted it. Here’s what I wrote:

Actually, the choice isn’t mine, unfortunately. I’m a “technical” guy, i.e. one who has to implement, configure, and maintain ProCurve equipment. You guys do well at catering to the beancounters, though, I’ll give you that.

If the choice were mine, I wouldn’t have had a couple truckloads of 5400s come through the loading dock. It would have been Cisco or Juniper gear instead — you know, equipment that actually has some of the features I need.

Your switches are okay at layer 2, they work great there. But for those of us who need some of these advanced features, you’re just not there. First chance I get, I’m ripping out every piece of ProCurve gear I have that’s running at layer 3 and they’ll be replaced with an “incredibly inflexible and limiting solution”.

I don’t think I said anything that was untrue. I got a bunch of new 5400zl switches (22 of them, if memory serves) and, to be quite honest, I’m not impressed. I did think it was ironic that a nearly eight-year-old “HP” ProCurve 9300 switch (which wasn’t actually an “HP” at all) had features I needed that the brand spankin’ new 5400s don’t have (route maps and policy based routing, for example). When a 9300 was yanked and replaced with a 5406zl, it was then necessary to move that functionality off to — you guessed it — a Cisco device.

To HP’s credit, as I mentioned above, their switches are fine — as long as you stick to layer 2. Unfortunately, I’m using them at layer 3 as well, and am amazed by some of the issues that this gear has. There have been numerous, repeated, and documented issues with OSPF ECMP. There was a bug where scp’ing a config file over (roughly) 4200 bytes in size would cause the switch to reset itself to factory defaults and, of course, a case I myself opened last December that still hasn’t been resolved (more on that in a minute).

Anyway, my comments never got to see the light of day on HP’s blog. Even so, “imtechchris” was nice enough to reply to me. He chose not to argue my points, though, and instead decided to attack my technical skills. Here’s his reply, in its entirety (just in case it disappears from the blog):

Now with regard to this statement I made:

First chance I get, I’m ripping out every piece of ProCurve gear I have that’s running at layer 3 and they’ll be replaced with an “incredibly inflexible and limiting solution”.

I was actually quoting their article and pointing to the fact that, if or when I have the chance, I’ll yank out that ProCurve gear and replace it with either Cisco or Juniper, which their original article alluded to as “incredibly inflexible and limiting solutions”. Apparently Chris couldn’t grok that. He chose to end his reply with “If anything is clear it’s that you don’t really know how to configure these switches and you don’t want to bother learning because it’s not Cisco or Juniper.”

I’ll argue each of those points with you. After about six years of dealing with this garbage (I’m referring to the ProCurves, Chris), I think I’ve got a little bit of an understanding of how they work. I’ve managed to keep a multi-site network with hundreds and hundreds of nodes running for the last several years. I took one HP certification test, the “Accredited Integration Specialist” exam (without studing, of course), which was a complete joke (it took all of 20 minutes to complete, and I scored 100% on it — I have the score report somewhere around here if anyone thinks I’m bullshitting). I never bothered to pursue any further HP certifications. Regardless, I’d argue that I do, in fact, “know how to configure these switches”.

Now, as far as me not wanting to bother learning because it’s Cisco or Juniper, well, you’re almost right. More specifically, I “don’t want to bother” because I’ve had repeated bad experiences. It is my job to manage this gear, however, and regardless of vendor, I will, to the best of my ability, keep our network up and running — that’s what I’m paid to do. Would I be a helluva lot happier about it if it were Cisco or Juniper gear? FUCK YES.

Last point: I was going to mention and provide details (with a screenshot, of course) of the case I opened w/ ProCurve support last December (22-Dec-2008, if memory serves), but the “Support Case Manager – Enterprise edition” is apparently having some issues. When I enter in the case ID to view the details, it takes forever and finally gives me a “Could not access case” error message. The issue in this case, by the way, was replicated by at least two others who read this web site and e-mailed me privately. Apparently HP told them two different things, including telling one that the issue he was experiencing (CPU utilization at 99%) was “by design”. I still have not heard of a fix, even after an HP guy came and spent the day with me documenting everything we could find related to the issue (thanks for lunch, Richard).

So, HP, fix my issues, quit shipping stupid bugs in what is supposed to be production-level code, and get some support and maybe then my opinion of you might improve.

Oh, and please, no calls or e-mails from my sales guys this time.

UPDATE: I finally got to the case details:

Looks like they closed the case on me on August 7th, even though the last message I got from an HP engineer reads:

Just a quick status update on this issue. Our lab has been very busy with some hot issues recently and hasn’t yet got around to reproduction testing yet. I hope to have a more definitive answer for you by the end of next week.

That was on July 10th, and was the last thing I heard about this issue. There is no “solution” or “fix” mentioned in the case details, either. Thanks again, HP, way to suck.

And people wonder why I hate HP…

I hate spammers.

That bears repeating, and emphasis: I *HATE* spammers. A lot.

I received an e-mail from Mark Czarnowski of Meru Networks (I’m intentionally not linking to them). Actually, I can be more specific than that. I received an unsolicited commercial e-mail from Mark Czarnowski of Meru Networks, a company I have never heard of nor had any business dealings with. Here is Mark’s e-mail:

Jeremy,

    Per my message, I was trying to touch base with you in order to discuss your wireless initiatives
you and the college have.  Meru Networks develops and markets wireless infrastructure solutions
that enable the All-Wireless Enterprise.  Our customer base range from healthcare, education (both
K-12 and higher), and the private sector.  I will like to take a few minutes of your time to discuss
your iniatives.

    If you can please inform me when a good time it will be to follow up I will make sure to do so
accordingly.  However, if you are not the appropriate person to speak to regarding wireless can you
please refer me to the person I should contact?  Your help will be much appreciated.  I will follow
up with you in a couple of days if I do nothear from you.  I look forward to you response.  Thank
you and have a great day!

Best Regards,
Mark

Mark Czarnowski
Meru Networks
(781) 702-6972

www.merunetworks.com

I’m not sure what “message” he’s referring to, unless he also cold called me on my office phone. That’s entirely possible, but I usually don’t answer my office phone unless I know from the caller ID who is calling. Also, my voice mail message asks that you do not leave me a voice mail (“Voicemail is dead.”).

Regardless, I sent Mark a nice little reply. I hope he “gets” it:

Hey Mark,

We actually just recently completed an upgrade of our wireless gear, so I think we're all set on that front.

Do you happen to have anything to help prevent unsolicited commercial e-mail, though?  I really hate spam!

--
Jeremy L. Gaddis

Okay, maybe not so funny, but I’ve been sick all week and it’s been the highlight of my day so far. =)

UPDATE: Well, it seems that my buddy Mark Czarnowski didn’t “get it”. I posted this late on a Friday afternoon, and had a reply from him Monday morning, basically spouting more marketing shit at me (along with a 1 MB PDF I didn’t bother to read). Stupid sales people. Oh well, I have another company name to add to my “don’t ever buy shit from these people” list.

I am unfortunate enough to have to maintain 30 or so HP ProCurve switches (with another 10 or so almost ready to be installed). I hate them, with a passion. It is impossible for me to accurately convey how much I hate them using mere words alone.

People often wonder why.

The reasons vary. I’ve had “support” respond to me that they cannot reproduce issues in the lab because they don’t have enough GBICs available (well, go fucking get them!). Their “solution” in this case was to offer me a fresh-off-the-compiler “beta” version of the switch firmware. Because I deal with production equipment (you know, stuff I want to work), I always pass. I am not HP’s QA department, nor will I act as such. Here’s why.

Today I was reading through the K.13.63 Release Notes and noticed this “known issue”:

“Transferring a switch configuration of 4,201 bytes or larger to a switch’s /cfg/startup-config directory via SCP will result in the switch coming up on factory defaults or with the new configuration only partly installed after reboot”

Are you fuckin’ kidding me, HP? How does shit like this make it past QA?

Looking at one of my 5400s, I see its startup config is 18,370 bytes. I have a lot of stuff to add to the config on that one, so it will likely easily pass 30k before it’s “done”. Glad I didn’t “upgrade” to this firmware earlier — I would’ve hated to have been the guy to find that issue.

There’s more that I’d share, but I’ll let you browse through the “Fixes” and “Known Issues” sections of the Release Notes for yourself. Look out for phrases like “hang”, “may reboot unexpectedly”, and the like.

Tomorrow, I’ll be receiving one of the new HP ProCurve Threat Management Services zl Modules. My HP sales rep has warned me in advance that “this module is not very intuitive, like many things you may have seen before”. Why am I not surprised?