evilrouters.net

I recently started using the free monitoring service available from Pingdom to monitor (via simple ICMP echo requests) the VPS that this website runs on. Here’s the “monthly report” that I received from them via e-mail for the month of February:

For the month of February, there were a total of two outages detected for a total “downtime” of exactly three minutes.

The first outage was for two minutes from 01:20:38 to 01:22:38 on February 13th. This was observed from Pingdom monitoring servers in London, UK, and Tampa, Florida.

The second was a few moments later, for one minute, from 01:25:38 to 01:26:38. It was observed from Pingdom monitoring servers in London, UK, and New York, New York.

The response times seem to be averaged across all checks from all monitoring servers. Monitoring servers that are close, network-wise, obviously see much lower response times, e.g. 1 millisecond from a monitoring server in Los Angeles, California. The VPS physically is in the Wilshire Annex facility at 900 N Alameda in Los Angeles, so that makes sense.

I can’t really comment on the “unavailability” of the web site as compared to its previous home (a 1&1 shared server) as no monitoring was done then, but I’m quite happy with only three minutes of downtime in a month.

Side note: a few months signing up with ARP Networks, I’m still quite pleased. I know of a number of others who have since signed up that are as well. If you’re interested in a VPS (768MB RAM, 20GB HDD, 200GB bandwidth for $20 USD/mo.), definitely check out their deals. If you sign up, mention my name (I’ll score a free month if enough people do). For us network geeks, they also have native IPv6 to every VPS.

Someone found these online but they apparently disappeared. He found them in his cache, though. =)

I’m not sure what order they were originally in, so they’re in no particular order here.

Here’s a fine example of that wonderful HP ProCurve engineering and quality control I’ve grown intimately familiar with:

The following problems were resolved in release K.13.71.

…

IP Communication (PR_0000044004) – Switches running software versions K.13.65 - K.13.68 may experience a resource leak in ICMP that eventually causes loss of IP communication with the following symptoms.

• The switch will stop routing traffic for hosts for which it provides gateway services.
• All dynamic routing protocols stop functioning. The switch will stop sending routing protocol packets, and will not process incoming routing protocol packets sent from its neighbors.
• In-band network management stops functioning. The switch will become inaccessible via Telnet, SSH, WEB, and TFTP.
• Console management may become very sluggish and may appear to be non-responsive.
• Output from the CLI command show ip route will be corrupted.
• A reboot is required to clear up the symptoms.

BUT AT LEAST WE SAVED SOME MONEY!

(Note: Yes, I have been bit by this bug.)

UPDATE: Be sure to see the (more recent) post, “New Cisco ISR G2 routers (slides)”, as well.

Hmm…

“In Cisco IOS Release 15.0(1)M, this feature was introduced on the Cisco 1941, 2900sm, 2901, and 3900 routers.”

Well now, this certainly looks interesting:

Router# show version

Cisco IOS Software, C3900 Software (C3900-UNIVERSALK9-M), Experimental Version 
12.4(20090904:044027) [guraman-revpi12 577]
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 04-Sep-09 09:22 by guraman

ROM: System Bootstrap, Version 12.4(20090303:092436) 
[BLD-xformers_dev.XFR_20090303-20090303_0101-53 102], DEVELOPMENT SOFTWARE

C3900-2 uptime is 8 hours, 41 minutes
System returned to ROM by reload at 08:40:40 UTC Tue May 21 1901! 
System image file is "flash0:c3900-universalk9-mz.SPA"
Last reload reason: Reload Command



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco CICSCO3945-MIDPLN (revision 1.0) with CISCO3900-MPE140 with 987136K/61440K bytes of 
memory.
Processor board ID FHH1231P02Y
3 Gigabit Ethernet interfaces
1 terminal line
1 Virtual Private Network (VPN) Module
1 cisco Integrated Service Engine(s)
DRAM configuration is 72 bits wide with parity enabled.
255K bytes of non-volatile configuration memory.
1020584K bytes of USB Flash usbflash0 (Read/Write)
1020584K bytes of USB Flash usbflash1 (Read/Write)
500472K bytes of ATA System CompactFlash 0 (Read/Write)

License Info:

License UDI:

-------------------------------------------------
Device#   PID                   SN
-------------------------------------------------
*0        CISCO3900-MPE140      FHH1215008L     



Technology Package License Information for Module:'c3900' 

----------------------------------------------------------------
Technology    Technology-package          Technology-package
              Current       Type          Next reboot  
-----------------------------------------------------------------
ipbase        ipbasek9      Permanent     ipbasek9
security      securityk9    Evaluation    securityk9
uc            None          None          None
data          None          None          None

Configuration register is 0x0

Taken from “Digitally Signed Cisco Software” (cisco.com).

Thoughts?

Apparently I need to stop “following” a couple folks on Twitter, namely @procurve. Because of my past experiences, I tend to not care much for HP ProCurve equipment and so sometimes the things posted by @procurve irk me, just a little.

One example was on August 28th, when @procurve “re-tweeted” this:

Now, in terms of functionality, the ProCurve 5406zl switch does not even come anywhere near a Cisco 7206VXR. They are in two completely different categories, as far as I am concerned. I made the remark that “they couldn’t have been doing much with that 7206vxr if they were able to replace it w/ a 5406zl”. Anyone who is familiar with both devices would, I’m positive, agree with me. Nonetheless, nice try at patting yourself on the back, HP.

So it’s little instances like this that annoy the shit out of me, considering the not-so-great experiences I’ve had with ProCurve in the last year or so. Today, @procurve posted this:

I, of course, wanted to hear what they had to say, so I clicked. Read the blog post, “Cost effective networking — The choice is yours”, if you’d like. I won’t post the whole thing here, but reading it will help you to understand my response.

If you did click that link, you won’t see my comments posted. As far as I can tell, my posted was never allowed to be displayed on that page. I’m glad I kept a copy before I submitted it. Here’s what I wrote:

Actually, the choice isn’t mine, unfortunately. I’m a “technical” guy, i.e. one who has to implement, configure, and maintain ProCurve equipment. You guys do well at catering to the beancounters, though, I’ll give you that.

If the choice were mine, I wouldn’t have had a couple truckloads of 5400s come through the loading dock. It would have been Cisco or Juniper gear instead — you know, equipment that actually has some of the features I need.

Your switches are okay at layer 2, they work great there. But for those of us who need some of these advanced features, you’re just not there. First chance I get, I’m ripping out every piece of ProCurve gear I have that’s running at layer 3 and they’ll be replaced with an “incredibly inflexible and limiting solution”.

I don’t think I said anything that was untrue. I got a bunch of new 5400zl switches (22 of them, if memory serves) and, to be quite honest, I’m not impressed. I did think it was ironic that a nearly eight-year-old “HP” ProCurve 9300 switch (which wasn’t actually an “HP” at all) had features I needed that the brand spankin’ new 5400s don’t have (route maps and policy based routing, for example). When a 9300 was yanked and replaced with a 5406zl, it was then necessary to move that functionality off to — you guessed it — a Cisco device.

To HP’s credit, as I mentioned above, their switches are fine — as long as you stick to layer 2. Unfortunately, I’m using them at layer 3 as well, and am amazed by some of the issues that this gear has. There have been numerous, repeated, and documented issues with OSPF ECMP. There was a bug where scp’ing a config file over (roughly) 4200 bytes in size would cause the switch to reset itself to factory defaults and, of course, a case I myself opened last December that still hasn’t been resolved (more on that in a minute).

Anyway, my comments never got to see the light of day on HP’s blog. Even so, “imtechchris” was nice enough to reply to me. He chose not to argue my points, though, and instead decided to attack my technical skills. Here’s his reply, in its entirety (just in case it disappears from the blog):

Now with regard to this statement I made:

First chance I get, I’m ripping out every piece of ProCurve gear I have that’s running at layer 3 and they’ll be replaced with an “incredibly inflexible and limiting solution”.

I was actually quoting their article and pointing to the fact that, if or when I have the chance, I’ll yank out that ProCurve gear and replace it with either Cisco or Juniper, which their original article alluded to as “incredibly inflexible and limiting solutions”. Apparently Chris couldn’t grok that. He chose to end his reply with “If anything is clear it’s that you don’t really know how to configure these switches and you don’t want to bother learning because it’s not Cisco or Juniper.”

I’ll argue each of those points with you. After about six years of dealing with this garbage (I’m referring to the ProCurves, Chris), I think I’ve got a little bit of an understanding of how they work. I’ve managed to keep a multi-site network with hundreds and hundreds of nodes running for the last several years. I took one HP certification test, the “Accredited Integration Specialist” exam (without studing, of course), which was a complete joke (it took all of 20 minutes to complete, and I scored 100% on it — I have the score report somewhere around here if anyone thinks I’m bullshitting). I never bothered to pursue any further HP certifications. Regardless, I’d argue that I do, in fact, “know how to configure these switches”.

Now, as far as me not wanting to bother learning because it’s Cisco or Juniper, well, you’re almost right. More specifically, I “don’t want to bother” because I’ve had repeated bad experiences. It is my job to manage this gear, however, and regardless of vendor, I will, to the best of my ability, keep our network up and running — that’s what I’m paid to do. Would I be a helluva lot happier about it if it were Cisco or Juniper gear? FUCK YES.

Last point: I was going to mention and provide details (with a screenshot, of course) of the case I opened w/ ProCurve support last December (22-Dec-2008, if memory serves), but the “Support Case Manager – Enterprise edition” is apparently having some issues. When I enter in the case ID to view the details, it takes forever and finally gives me a “Could not access case” error message. The issue in this case, by the way, was replicated by at least two others who read this web site and e-mailed me privately. Apparently HP told them two different things, including telling one that the issue he was experiencing (CPU utilization at 99%) was “by design”. I still have not heard of a fix, even after an HP guy came and spent the day with me documenting everything we could find related to the issue (thanks for lunch, Richard).

So, HP, fix my issues, quit shipping stupid bugs in what is supposed to be production-level code, and get some support and maybe then my opinion of you might improve.

Oh, and please, no calls or e-mails from my sales guys this time.

UPDATE: I finally got to the case details:

Looks like they closed the case on me on August 7th, even though the last message I got from an HP engineer reads:

Just a quick status update on this issue. Our lab has been very busy with some hot issues recently and hasn’t yet got around to reproduction testing yet. I hope to have a more definitive answer for you by the end of next week.

That was on July 10th, and was the last thing I heard about this issue. There is no “solution” or “fix” mentioned in the case details, either. Thanks again, HP, way to suck.

And people wonder why I hate HP…